Visual information extraction method based on multi-modal large model

Abstract

A visual information extraction method based on a multi-modal large model comprises the following steps: (1) performing fine adjustment on the multi-modal large model by using three types of special data, namely text positioning data, visual reference data and chain thinking data; (2) a user uploads a target picture and inputs a character prompt, and a template library is triggered to retrieve and add context information according to conditions; (3) the multi-modal large model generates a preliminary reply result and generates a final reply result in combination with an OCR character recognition result; (4) the dialogue history and the meta-information are integrated and stored in a template library; according to the method, accurate output of the result textbox is achieved, error correction of the output result of the large model is completed in cooperation with the special OCR model, and the accuracy and credibility of visual information extraction are improved.

Description

Technical Field

[0001] The invention relates to a visual information extraction method based on a multimodal large model, and belongs to the field of information technology, in particular to the field of visual information extraction technology. Background Art

[0002] With the rapid development of Internet technology and the growing demand for information exchange, a large number of documents are now digitized, stored, and distributed in the form of images. In a variety of application scenarios, such as receipt understanding, card recognition, automatic paper scoring, and document matching, key information needs to be extracted from document images. This process is called visual information extraction (VIE), which involves mining, analyzing, and extracting information from visually rich documents. For example, in receipt understanding, VIE algorithms can extract information such as store name, product details, and price from an image of a receipt. Unlike extracting information from plain text, the results of VIE are not determined by the text content alone. They are also affected by other visual factors such as document layout, font style, block color, charts, graphics, etc.

[0003] Analyzing and processing visually rich documents is a challenging task. The accuracy and credibility of document OCR by existing visual information extraction technologies need to be improved, and they lack ease of use, poor generalization ability, high cost, and cumbersome processes.

[0004] How to effectively utilize large model technology, fully explore the potential thinking chain of large models, and solve the above-mentioned problems existing in existing visual information extraction is a technical problem that urgently needs to be solved in the field of visual information extraction technology. Summary of the invention

[0005] In view of this, the purpose of the present invention is to invent a visual information extraction method based on a multimodal large model, which can achieve technical goals such as high result credibility, strong interpretability, strong versatility, and easy use.

[0006] In order to achieve the above object, the present invention proposes a visual information extraction method based on a multimodal large model, the method comprising the following steps:

[0007] (1) Fine-tune the multimodal large model using three types of specialized data, namely text location data, visual reference data, and chain thinking data, to ensure that it can accurately understand the content of document images and extract key information from them;

[0008] (2) After completing fine-tuning of the multimodal large model, the user uploads the target image to the multimodal large model and enters a text prompt; if the entered text prompt does not contain the trigger word "answer according to the template library", it directly goes to step (3); otherwise, the multimodal large model retrieves the most relevant template from the template library and adds the relevant context information to the current conversation, and then goes to step (3);

[0009] (3) The target image is first subjected to optical character recognition (OCR) to perform text recognition on the entire image, thereby obtaining the text content and the coordinates of its location; combined with relevant context information, a complete graphic prompt is formed, which is then submitted to the multimodal large model for processing to generate a preliminary response result; the preliminary response result includes not only the text content but also the coordinate information of these texts; the preliminary response result is then combined with the aforementioned OCR text recognition result and subjected to judgment and error correction processing to generate a final response result;

[0010] (4) When the user enters "make template" in the next conversation, the multimodal large model will give a detailed description of the image entered by the user, including the document type, title, main information, etc., and save these descriptions together with the user's question as the meta-information of the template, so that accurate retrieval and matching can be performed in step (2); then, the user's conversation history will be saved; the multimodal large model will request the user to confirm whether the content of the model's reply is accurate; if the user finds that the reply is incorrect, he or she can make manual corrections; finally, the manually corrected conversation history and the meta-information summarized by the model will be integrated and saved in the template library.

[0011] The purpose of the text positioning data is to enable the multimodal large model to output accurate text coordinates, so as to facilitate subsequent error correction and improve the overall credibility; the text positioning data comes from pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data. After these data are processed by PaddleOCR recognition, the text content and its corresponding text box coordinates are obtained; in order to make the multimodal large model adapt to diverse instructions, fixed question prompts are not used during fine-tuning. A pre-set question is expressed as " \nPlease give the bounding box coordinates of the area described by this sentence: XXX", and the answer of the multimodal large model is " <ref>XXX< / ref> <box> [[x1,y1,x2,y2]]< / box> ”;

[0012] The purpose of the visual reference data is to enable the multimodal model to strengthen its understanding of the marked box, so that the multimodal model can focus on the text marked by the eye-catching color box, so that the user can guide the multimodal model by simply selecting or marking the target text area on the picture without providing the coordinates of the target text area; the visual reference data comes from pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data. After PaddleOCR recognizes the text content and text box, opencv-python is used to mark these text boxes in the picture with eye-catching colors; a pre-set problem is expressed as " \nPlease identify the content marked by the eye-catching color box in the picture", there is no fixed prompt; the answer of the multimodal large model is " <ref> XXX< / ref> <box> [[x1,y1,x2,y2]]< / box> ”;

[0013] The purpose of the chain thinking data is to improve the performance of the multimodal large model in complex document scenarios through Chain-of-Thought reasoning; the thought chain used is the layout thinking chain, and the chain thinking process of the layout thinking chain includes three steps: problem analysis, relevant area division and answer determination; the chain thinking data is based on pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data, and is formed by Zhipu Qingyan ChatGLM in combination with manual annotation.

[0014] The multimodal large model adopts InternVL2, whose architecture is a combination of a visual encoder ViT module + a projection connection layer MLP module + a large language model LLM module;

[0015] A two-stage fine-tuning strategy is used to fine-tune the multimodal large model;

[0016] In the first stage, the ViT module and the LLM module are frozen first, and the MLP module is only fine-tuned using the text location data and visual reference data to enhance the multimodal model's understanding of text location and parsing capabilities of visual references, which helps ensure that the multimodal model can accurately locate text elements in the document and understand the marked content;

[0017] In the second stage, the ViT module and the LLM module are kept frozen, and the chain thinking data is used to adopt the low-rank adaptation technology to fine-tune the parameters of the small adapter layer attached to the ViT and LLM modules of the multimodal large model. This not only reduces the number of additional parameters, but also effectively avoids the risk of overfitting, while keeping the core functions of the multimodal large model unchanged.

[0018] The content of the template library is composed of user's conversation history record instances and meta-information summarized by the multimodal large model; each conversation history record instance contains a document image of a specific type, a user's question regarding specific key information and the model's reply, and the meta-information includes a description of the document image and the question; the model's reply has been manually corrected to ensure the accuracy of the result;

[0019] When "Answer according to template library" is triggered, the multimodal large model retrieves the most relevant template from the template library. The specific content is: the multimodal large model summarizes the information of the current target image and question content, mainly involving elements such as document type and title content, and then uses the BM25 algorithm to search and match the most relevant template instance in the meta-information part of the template library.

[0020] The specific content of the multimodal large model adding relevant context information to the current conversation is: after finding a matching instance, the multimodal large model will attach the conversation history record of the matching instance to the context of the current conversation, so as to perform In-Context Learning.

[0021] In step (3), the specific content of the judgment and error correction processing of the preliminary reply result is: using the coordinate information in the multimodal large model reply result to match the text box obtained previously by OCR, and selecting the matching result with the largest intersection over union (IoU) value and greater than a preset threshold.

[0022] The beneficial effects of the present invention are as follows: the present invention uses a multimodal large model to achieve accurate result text box output, and cooperates with a dedicated OCR model to complete error correction of the large model output results, which avoids the hallucination problem of the large model to a certain extent and improves the accuracy and credibility of the results; the present invention enables the multimodal large model to understand the visual reference marks on the picture, avoiding the problem that the model is difficult to understand with text descriptions alone in complex document contexts, and can easily make the model focus on the target area; the present invention adds historical conversations after manual error correction to the current conversation context, and utilizes the small sample learning characteristics of the large model itself to improve the ability to extract information from unknown document data, without the need to train multiple models for different types of documents, which is efficient, convenient and cost-saving; the present invention gives full play to the thinking chain ability of the multimodal large model, greatly improves the accuracy of model information extraction by introducing intermediate reasoning steps, and also enhances the interpretability of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of a visual information extraction method based on a multimodal large model proposed in the present invention.

[0024] Figure 2This is an example diagram of the set business image data used in the embodiment of the present invention.

[0025] Figure 3 is an example diagram of visual reference data used in an embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of using three types of special data to perform two-stage fine-tuning on a large model in an embodiment of the present invention.

[0027] Figure 5 It is a schematic diagram of the template library and template library retrieval in an embodiment of the present invention.

[0028] Figure 6 It is a schematic diagram of judging and correcting the initial reply result in step (3) of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0030] See also Figure 1 , introduces a visual information extraction method based on a multimodal large model proposed by the present invention, the method comprising the following steps:

[0031] (1) Fine-tune the multimodal large model using three types of specialized data, namely text location data, visual reference data, and chain thinking data, to ensure that it can accurately understand the content of document images and extract key information from them;

[0032] (2) After completing fine-tuning of the multimodal large model, the user uploads the target image to the multimodal large model and enters a text prompt; if the entered text prompt does not contain the trigger word "answer according to the template library", it directly goes to step (3); otherwise, the multimodal large model retrieves the most relevant template from the template library and adds the relevant context information to the current conversation, and then goes to step (3);

[0033] (3) The target image is first subjected to optical character recognition (OCR) to perform text recognition on the entire image, thereby obtaining the text content and the coordinates of its location; combined with relevant context information, a complete graphic prompt is formed, which is then submitted to the multimodal large model for processing to generate a preliminary response result; the preliminary response result includes not only the text content but also the coordinate information of these texts; the preliminary response result is then combined with the aforementioned OCR text recognition result and subjected to judgment and error correction processing to generate a final response result;

[0034] (4) When the user enters "make template" in the next conversation, the multimodal large model will give a detailed description of the image entered by the user, including the document type, title, main information, etc., and save these descriptions together with the user's question as the meta-information of the template, so that accurate retrieval and matching can be performed in step (2); then, the user's conversation history will be saved; the multimodal large model will request the user to confirm whether the content of the model's reply is accurate; if the user finds that the reply is incorrect, he or she can make manual corrections; finally, the manually corrected conversation history and the meta-information summarized by the model will be integrated and saved in the template library.

[0035] The purpose of the text positioning data is to enable the multimodal large model to output accurate text coordinates, so as to facilitate subsequent error correction and improve the overall credibility; the text positioning data comes from a pre-set data set including DocVQA-ZH (DocVQA-ZH is a public data set of an insurance text visual cognition question-answering competition, containing question-answer pairs for key information, the download link is https: / / bj.bcebos.com / paddlenlp / datasets / docvqa_ zh.tar.gz ), XFUND-ZH (XFUND-ZH is derived from the Chinese subset of the academic open source dataset XFUND, which contains entity-relationship pairs and their coordinate frames. The project link is https: / / github.com / doc-analysis / XFUND ) and set business image data. The data in the datasets DocVQA-ZH and XFUND-ZH are all Chinese document images with rich visual information such as cards, receipts, and forms.

[0036] See also Figure 2 In the embodiment, the set business image data mainly includes several types of data used in the experiment of the present invention, such as settlement sheets, nameplates, etc., and the content or coordinate frame of the information to be extracted is manually marked.

[0037] After the above data is processed by PaddleOCR (https: / / gitee.com / paddlepaddle / PaddleOCR), the text content and its corresponding text box coordinates are obtained. Figure 2 Some of the results obtained are "[[797.0,36.0],[1150.0,20.0],[1151.0,59.0],[799.0,75.0]],('ZXDU58W121',0.9939807057380676)"; In order to make the multimodal large model adapt to diverse instructions, fixed question prompts are not used during fine-tuning. For example, for Figure 2 The data shown, a pre-set optional question is stated as " \nPlease give the bounding box coordinates of the area described by this sentence: outdoor power system", and the answer of the multimodal large model is " <ref> Outdoor power system< / ref> <box> [[346.0,143.0,657.0,175.0]]< / box> ”;

[0038] The purpose of the visual reference data is to enable the multimodal large model to strengthen its understanding of the marked frame, so that the multimodal large model can focus on the text marked by the eye-catching color frame (in the embodiment, a red frame is used), so that the user can guide the multimodal large model by only selecting or marking the target text area on the picture without providing the coordinates of the target text area; the visual reference data comes from pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data. After PaddleOCR recognizes the text content and text box, opencv-python is used to mark these text boxes in the picture with eye-catching colors;

[0039] For example, Figure 3 The data shown, a pre-set optional question is stated as " \nPlease identify the content marked by the red box in the picture", there is no fixed prompt; the answer of the multimodal large model is " <ref> Outdoor power system< / ref> <box> [[346.0,143.0,657.0,175.0]]< / box> ”;

[0040] The purpose of the chain thinking data is to improve the performance of the multimodal large model in complex document scenarios through Chain-of-Thought reasoning; the thought chain used is the layout thinking chain, and the chain thinking process of the layout thinking chain includes three steps: problem analysis, relevant area division and answer determination; the chain thinking data is based on pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data, and is formed through Zhipu Qingyan ChatGLM (https: / / chatglm.cn) and manual annotation.

[0041] Taking the table scenario as an example, the question might be "What is the total of the confirmed amount (excluding tax)?" The model's answer would be: "This is a table comprehension question. The question content is the value of the field 'Confirmed amount (excluding tax) total'; the relevant area is in the left area of ​​the table in the document, with coordinates [x1, y1, x2, y2]; the answer to the question is in the same column as 'Confirmed amount (excluding tax)' and in the same row as 'Total', and the content is <ref> XXX< / ref> <box> [[x3,y3,x4,y4]]< / box> ”.

[0042] See also Figure 4The multimodal large model adopts InternVL2 (https: / / internvl.github.io / blog / 2024-07-02-InternVL-2.0 / ), whose architecture is a combination of a visual encoder ViT module + a projection connection layer MLP module + a large language model LLM module;

[0043] A two-stage fine-tuning strategy is used to fine-tune the multimodal large model;

[0044] In the first stage, the ViT module and the LLM module are frozen first, and the MLP module is only fine-tuned using the text location data and visual reference data to enhance the multimodal model's understanding of text location and parsing capabilities of visual references, which helps ensure that the multimodal model can accurately locate text elements in the document and understand the marked content;

[0045] In the second stage, the ViT module and the LLM module are kept frozen, and the chain thinking data is used to adopt the low-rank adaptation technology to fine-tune the parameters of the small adapter layer attached to the ViT and LLM modules of the multimodal large model. This not only reduces the number of additional parameters, but also effectively avoids the risk of overfitting, while keeping the core functions of the multimodal large model unchanged.

[0046] Low-Rank Adaptation technology is a lightweight adaptive mechanism that fine-tunes parameters by adding small adapters to the original modules. This method not only reduces the number of additional parameters, but also effectively avoids the risk of overfitting while keeping the core functions of the model unchanged.

[0047] See also Figure 5 The content of the template library is composed of the user's conversation history record instance and the meta-information summarized by the multimodal large model; each conversation history record instance contains a document image of a specific type, the user's question regarding specific key information and the model's reply, and the meta-information includes a description of the document image and the question; the model's reply has been manually corrected to ensure the accuracy of the result;

[0048] See also Figure 5 When "Answer according to template library" is triggered, the multimodal large model retrieves the most relevant template from the template library. The specific content is: the multimodal large model summarizes the information of the current target image and the question content, mainly involving elements such as document type and title content, and then uses the BM25 algorithm (Best Match 25) to search and match the most relevant template instance in the meta-information part of the template library.

[0049] The specific content of the multimodal large model adding relevant context information to the current conversation is: after finding a matching instance, the multimodal large model will attach the conversation history record of the matching instance to the context of the current conversation, so as to perform In-Context Learning.

[0050] See also Figure 6 In step (3), the specific content of the judgment and error correction processing of the preliminary response result is: the response result of the multimodal large model is "green electricity consumption certificate", and its coordinate information is matched with the text box obtained by the previous OCR, and the matching result with the largest intersection over union (IoU) value and greater than the preset threshold is selected. The first item of the OCR result "Certificate No." has an IOU of 0 with the large model result, while the second item "green electricity consumption certificate" has the highest IOU, so the latter is selected. After obtaining the matching result, the text content in the OCR result is used to replace the large model response result to complete the error correction.

[0051] The inventors conducted a large number of experiments on the method proposed in the present invention. The experiments selected various types of document images as test samples, including settlement forms, invoices, nameplate images, etc., to test the adaptability of the method in different application scenarios. The experiments were divided into two main parts depending on whether the template library was used.

[0052] The first part focuses on testing the performance improvement of multimodal large models by fine-tuning training. In this stage, the experiment took the form of asking questions with a single picture. The results showed that the fine-tuned model has significantly improved its ability to output text coordinate boxes, and its fine-grained text understanding ability has also been greatly improved. It is worth noting that the model can also understand visual reference cues marked by humans, and performs particularly well when processing complex documents. Due to the introduction of chain thinking data, the fine-tuned model will perform detailed step analysis when outputting, which makes the accuracy of its responses significantly higher than the original model.

[0053] The second part further tests the actual effect of the whole method. In this process, multiple rounds of questions and answers were conducted in advance to expand the template library. The experimental results show that the multimodal large model is very effective in contextual few-sample learning, and can quickly correct the original wrong answer results with only a few samples. Compared with the model without contextual learning support, this method with contextual learning has greatly improved the accuracy of the reply results. In addition, in practical applications, the text recognition of the multimodal large model may be wrong, resulting in deviations in the preliminary reply results. However, by combining the OCR recognition results and the preliminary reply results of the multimodal large model, the method can effectively judge and correct errors. This mechanism has shown good results in the experiment. It can identify text errors and correct them, thereby ensuring the accuracy of the final reply results.

[0054] The above experimental results prove that the method proposed in the present invention is effective.

Claims

1. A visual information extraction method based on a multimodal large model, characterized in that: The method comprises the following steps: (1) Fine-tune the multimodal large model using three types of specialized data, namely text location data, visual reference data, and chain thinking data, to ensure that it can accurately understand the content of document images and extract key information from them; (2) After completing fine-tuning of the multimodal large model, the user uploads the target image to the multimodal large model and enters a text prompt; if the entered text prompt does not contain the trigger word "answer according to the template library", it directly goes to step (3); otherwise, the multimodal large model retrieves the most relevant template from the template library and adds the relevant context information to the current conversation, and then goes to step (3); (3) The target image is first subjected to optical character recognition (OCR) to perform text recognition on the entire image, thereby obtaining the text content and the coordinates of its location; combined with relevant context information, a complete graphic prompt is formed, which is then submitted to the multimodal large model for processing to generate a preliminary response result; the preliminary response result includes not only the text content but also the coordinate information of the text; The preliminary reply result is combined with the aforementioned OCR text recognition result and subjected to error correction processing to generate a final reply result; (4) When the user enters "make template" in the next conversation, the multimodal large model will give a detailed description of the image entered by the user, including the document type, title, main information, etc., and save these descriptions together with the user's question as the meta-information of the template, so that accurate retrieval and matching can be performed in step (2); then, the user's conversation history will be saved; the multimodal large model will request the user to confirm whether the content of the model's reply is accurate; if the user finds that the reply is incorrect, he or she can make manual corrections; finally, the manually corrected conversation history and the meta-information summarized by the model will be integrated and saved in the template library.

2. The method for extracting visual information based on a multimodal large model according to claim 1, characterized in that: The purpose of the text positioning data is to enable the multimodal model to output accurate text coordinates, so as to facilitate subsequent error correction and improve the overall credibility; the text positioning data comes from pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data. After these data are processed by PaddleOCR recognition, the text content and its corresponding text box coordinates are obtained; in order to enable the multimodal model to adapt to diverse instructions, fixed question prompts are not used during fine-tuning. A pre-set question is expressed as " \nPlease give the bounding box coordinates of the area described by this sentence: XXX", and the answer of the multimodal large model is " <ref> XXX< / ref> <box> [[x1,y1,x2,y2]]< / box> ”; The purpose of the visual reference data is to enable the multimodal model to strengthen its understanding of the marked box, so that the multimodal model can focus on the text marked by the eye-catching color box, so that the user can guide the multimodal model by simply selecting or marking the target text area on the picture without providing the coordinates of the target text area; the visual reference data comes from the pre-set data sets including DocVQA-ZH, XFUND-ZH and the set business image data. After the text content and text box are recognized by PaddleOCR, these text boxes are marked in the picture with eye-catching colors using opencv-python; a pre-set problem is expressed as " \nPlease identify the content marked by the eye-catching color box in the picture", there is no fixed prompt; the answer of the multimodal large model is " <ref> XXX< / ref> <box> [[x1,y1,x2,y2]]< / box> ”; The purpose of the chain thinking data is to improve the performance of the multimodal large model in complex document scenarios through Chain-of-Thought reasoning; the thought chain used is the layout thinking chain, and the chain thinking process of the layout thinking chain includes three steps: problem analysis, relevant area division and answer determination; the chain thinking data is based on pre-set data sets including DocVQA-ZH, XFUND-ZH and set business image data, and is formed by Zhipu Qingyan ChatGLM in combination with manual annotation.

3. The method for extracting visual information based on a multimodal large model according to claim 1, characterized in that: The multimodal large model adopts InternVL2, whose architecture is a combination of a visual encoder ViT module + a projection connection layer MLP module + a large language model LLM module; A two-stage fine-tuning strategy is used to fine-tune the multimodal large model; In the first stage, the ViT module and the LLM module are frozen first, and the MLP module is only fine-tuned using the text location data and visual reference data to enhance the multimodal model's understanding of text location and parsing capabilities of visual references, which helps ensure that the multimodal model can accurately locate text elements in the document and understand the marked content; In the second stage, the ViT module and the LLM module are kept frozen, and the chain thinking data is used to adopt the low-rank adaptation technology to fine-tune the parameters of the small adapter layer attached to the ViT and LLM modules of the multimodal large model. This not only reduces the number of additional parameters, but also effectively avoids the risk of overfitting, while keeping the core functions of the multimodal large model unchanged.

4. The method for extracting visual information based on a multimodal large model according to claim 1, characterized in that: The content of the template library is composed of user's conversation history record instances and meta-information summarized by the multimodal large model; each conversation history record instance contains a document image of a specific type, a user's question regarding specific key information and the model's reply, and the meta-information includes a description of the document image and the question; the model's reply has been manually corrected to ensure the accuracy of the result; When "Answer according to template library" is triggered, the multimodal large model retrieves the most relevant template from the template library. The specific content is: the multimodal large model summarizes the information of the current target image and question content, mainly involving elements such as document type and title content, and then uses the BM25 algorithm to search and match the most relevant template instance in the meta-information part of the template library. The specific content of the multimodal large model adding relevant context information to the current conversation is: after finding a matching instance, the multimodal large model will attach the conversation history record of the matching instance to the context of the current conversation, so as to perform In-Context Learning.

5. The method for extracting visual information based on a multimodal large model according to claim 1, characterized in that: In step (3), the specific content of the judgment and error correction processing of the preliminary reply result is: using the coordinate information in the multimodal large model reply result to match the text box obtained previously by OCR, and selecting the matching result with the largest intersection over union (IoU) value and greater than a preset threshold.

Images