Bill information extraction method and system and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and particularly relates to a bill information extraction method and system and a storage medium, and the method comprises the following steps: obtaining an electronic file containing a bill, and preprocessing the electronic file into an image file facilitating information extraction; extracting bill information in the image file by using a multi-modal large model and an OCR model, and outputting text information in a structured format; and analyzing, correcting and comparing text information respectively output by the multi-modal large model and the OCR model through the large language model, and performing fine tuning or cue word engineering guidance on the large language model for different tasks, so that an output result conforms to expectation. Compared with the prior art, the problems that in the prior art, OCR recognition accuracy is not high, and extracted information is incomplete are solved, the bill information processing speed is increased, the precision of the automatic processing process is greatly improved, and in financial, auditing and other related business scenes, the working efficiency and the data reliability are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a bill information extraction method, system and storage medium. Background Art

[0002] Optical character recognition (OCR) technology is one of the most commonly used technologies for converting printed documents into machine-readable documents, especially in the process of archiving bill information. Through OCR technology, key information in the bill can be extracted, such as bill number, billing date, purchaser information, seller information, etc. However, although OCR technology has great advantages in many aspects, there are still many shortcomings and pain points in practical applications.

[0003] First, although the OCR engine can recognize text printed in a standardized form very well, the OCR recognition accuracy is still low for handwritten text or text processed by various outdated digital technologies. This type of text usually contains a lot of digital noise, which will seriously affect the OCR processing results, resulting in output errors or missing data. For example, handwritten bills are often misrecognized by the OCR engine due to irregular handwriting, resulting in errors in the extraction of key information. The documents generated by some old printers or scanners have low resolution of the equipment itself, or the documents have been copied and scanned many times, resulting in deterioration in quality. These factors will increase the difficulty and error rate of OCR recognition.

[0004] Secondly, although OCR technology can extract text information from text, it lacks the ability to classify and organize information. During the bill processing process, the extracted text information needs to be further organized and classified to match different fields to the corresponding directories. However, OCR technology cannot complete this process automatically, and usually requires manual setting of segmentation rules, such as specifying segmentation positions or using regular expressions for matching. For bills with standardized formats, this process is relatively simple, but for bills with irregular or diverse formats, manually setting segmentation rules becomes very complicated and cumbersome. Whenever a new bill format is encountered, the segmentation rules need to be readjusted, which not only increases the workload, but is also prone to errors, resulting in mismatched information or missing information.

[0005] In addition, OCR technology has high requirements for image quality. If the quality of the input image is poor, such as low resolution, poor contrast, blur or shadow, the recognition effect of OCR will be greatly reduced. When processing these low-quality images, additional preprocessing is required, such as denoising, contrast enhancement, tilt correction, etc., to improve recognition accuracy. These preprocessing steps not only increase processing time, but also require professional image processing knowledge, further increasing the threshold for use.

[0006] Furthermore, OCR technology also has certain limitations when processing multilingual texts. Although many OCR engines currently support multilingual recognition, in actual applications, switching between different languages ​​often leads to recognition errors. For example, for documents mixed with Chinese and English, due to the different character sets, the OCR engine may mistakenly recognize Chinese characters as English characters or vice versa, resulting in the inability to correctly interpret the extracted information. In addition, for some niche languages ​​or special symbols, such as company logos and QR codes in invoices, the recognition ability of the OCR engine is also relatively limited.

[0007] In summary, although OCR technology plays an important role in the filing of bills, it still has many pain points in practical applications, such as low recognition accuracy, difficulty in information classification and sorting, high image quality requirements, and limitations in multi-language processing. To solve these problems, it is necessary to continuously innovate and improve technology, and accumulate more experience and optimization strategies in practical applications. Summary of the invention

[0008] The purpose of the present invention is to provide a bill information extraction method, system and storage medium in order to solve at least one of the above problems, so as to solve the problems of low OCR recognition accuracy and incomplete extracted information in the prior art. This solution not only improves the speed of bill information processing, but also greatly improves the accuracy of the automated processing process, thereby significantly enhancing work efficiency and data reliability in finance, auditing and other related business scenarios.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] The first aspect of the present invention discloses a method for extracting bill information, comprising the following steps:

[0011] Obtaining an electronic file containing a bill, and preprocessing the electronic file into an image file for easy information extraction;

[0012] Use the multimodal large model and OCR model to extract the bill information from the electronic file and image file respectively, and output the text information in a structured format;

[0013] The large language model is used to analyze, correct, and compare the text information output by the multimodal large model and the OCR model. The large language model is fine-tuned or guided by prompt word engineering for different tasks to ensure that the output results meet expectations.

[0014] Preferably, the electronic file includes a file in the PDF format and a picture format of the bill.

[0015] Preferably, the preprocessing includes format conversion and image enhancement, wherein the format conversion is used to convert files in PDF format and picture format into bitmap format for subsequent processing.

[0016] Preferably, the extracted text information is output according to a user-specified format.

[0017] Preferably, the user-specified format includes key-value pairs.

[0018] The second aspect of the present invention discloses a bill information extraction system, which adopts any of the above methods;

[0019] The system includes a working agent and a supervisory decision agent;

[0020] The working agent is used for information extraction and analysis and comparison;

[0021] The supervisory decision agent is used to decide the next action or end the process according to the output of the working agent.

[0022] Preferably, the working agent includes a first sub-agent for extracting information and a second sub-agent for analyzing and comparing information.

[0023] Preferably, there are several first sub-agents, each of which is configured with a different model, and the bill information is extracted and output in a structured manner through the different models.

[0024] Preferably, the system also includes an output agent for outputting the bill information extraction results.

[0025] Preferably, the system is deployed locally.

[0026] A third aspect of the present invention discloses a storage medium for computer-readable storage, wherein the storage medium stores a program or instruction, and the program or instruction executes the steps of any of the above-mentioned ticket information extraction methods.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1) Application of multimodal large models: This invention innovatively combines multimodal large models with large language models and optical character recognition (OCR) models to build a fully automated bill information extraction system. Unlike traditional OCR technology, multimodal large models combine visual and semantic information, benefiting from their large training sets, and can better understand and process complex layouts and diverse inputs. They can also better identify low-quality documents such as handwritten documents with irregular handwriting and high digital noise, thereby improving the accuracy and comprehensiveness of information extraction.

[0029] 2) Autonomous operation of AI agents: This invention upgrades the semi-automatic post-processing mode to fully automatic processing. It introduces autonomous AI agents that can make decisions independently based on environmental information and task requirements without user intervention. It allows the definition of processes involving loops, provides fine-grained control over processes and states, and supports a memory function that records the conversation between the user and the AI.

[0030] 3) Natural language interaction: The present invention allows users to interact with the system through natural language to specify output fields and output formats. Users can use natural language instructions to flexibly control the type and format of extracted information. This interactive method greatly simplifies the operation process and makes the system easier to use and more efficient.

[0031] 4) Wide application scenarios: This method has significant application value in finance, auditing and other related business scenarios, and can enhance work efficiency and data reliability. For example, in financial auditing, a large amount of bill information can be quickly and accurately extracted and analyzed, reducing manual intervention and improving work efficiency.

[0032] 5) Security considerations: The present invention supports local deployment of the model, ensuring that all data is processed locally without leaking information to cloud services or external suppliers. This feature greatly improves the security of the system, especially when processing sensitive financial and business information, ensuring data privacy and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the technical implementation process of this method;

[0034] Figure 2 is a schematic diagram of the workflow in this method;

[0035] Figure 3 A schematic diagram showing an example output of this method. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example

[0038] like Figure 1-3 shown.

[0039] This embodiment includes a method for extracting bill information, comprising the following steps:

[0040] Obtaining an electronic file containing a bill, and preprocessing the electronic file into an image file for easy information extraction;

[0041] Use the multimodal large model and OCR model to extract the bill information from the image file and output the text information in a structured format;

[0042] The large language model is used to analyze, correct, and compare the text information output by the multimodal large model and the OCR model. The large language model is fine-tuned or guided by prompt word engineering for different tasks to ensure that the output results meet expectations.

[0043] This embodiment also includes a bill information extraction system, which adopts any of the above methods;

[0044] The system includes a working agent and a supervisory decision agent;

[0045] The working agent is used for information extraction and analysis and comparison;

[0046] The supervisory decision agent is used to decide the next action or end the process according to the output of the working agent.

[0047] More specifically, in this embodiment:

[0048] This solution aims to fully automate the extraction of bill information, including bill number, billing date, buyer information, seller information and other key information. The extracted information must be complete, accurate and in a standardized format. This solution also uses artificial intelligence agents (AI Agents) to dispatch multiple models to complete bill information extraction, result comparison, result analysis, self-correction and other operations, and can output the extracted information in a user-specified format.

[0049] like Figure 1 As shown, the technical solution of this embodiment is as follows:

[0050] S1. Data input:

[0051] 1) Upload the PDF file (including scanned copy) or image format file (including JPEG, PNG, etc.) of the bill to the system.

[0052] S2. Preprocessing:

[0053] 1) If the uploaded file format is a PDF file, convert the PDF file into an image format (preferably a bitmap format) for subsequent processing.

[0054] 2) Improve image quality through image enhancement technology.

[0055] S3. Define tools for extracting bill information:

[0056] 1) Use a multimodal model to extract ticket information and output it in a structured format.

[0057] 2) Use the OCR model to parse the file and output structured text information.

[0058] S4. Define tools for analysis, comparison and information extraction

[0059] 1) Fine-tune the large language model or guide the prompt word engineering for different tasks to make the results meet expectations. This may include whether to extract only specific information, whether to process the extracted information, or whether the extracted information is in a specific dictionary.

[0060] 2) Use large language models to analyze and compare the extracted information to ensure data accuracy and comprehensiveness.

[0061] S5. Define tools to output the final extracted information

[0062] 1) Output the extracted information in a user-specified format.

[0063] S6. Define and configure each agent and node

[0064] 1) Create multiple information extraction agents and assign different bill information extraction tools.

[0065] 2) Create an agent for analyzing and comparing information, and assign tools for extracting information for analysis and comparison.

[0066] 3) Create an agent that outputs results and assign a tool to output the results.

[0067] 4) Create an agent responsible for supervision and decision-making (Supervisor). This agent uses a large language model and fine-tunes it or guides it with prompt word engineering according to different tasks, so that it can judge the results of analysis and comparison and decide whether to output data or re-extract data.

[0068] S7. Construction and configuration of workflow:

[0069] 1) Build and configure workflows to coordinate the extraction and analysis work of various models.

[0070] S8. Run the workflow:

[0071] 1) Run the entire workflow, schedule each model to extract and analyze information, and finally output the results.

[0072] In step S5, the user can specify the extracted fields, output format, etc. through natural language interaction. For example, specify a json file in the output key-value pair format to facilitate subsequent field value matching and extraction, as well as database docking and other tasks.

[0073] In step S7, the workflow is constructed and configured, which mainly consists of two parts: the working agent and the supervisory decision agent. The supervisory decision agent decides the next action or ends the process based on the output of the working agent. This architecture makes task allocation more orderly and multi-agent collaboration more efficient. Figure 2 As shown:

[0074] Specifically in this embodiment, the supervisory decision agent can issue instructions to call multimodal models (including but not limited to GPT-4o, InternVL2, self-training models, etc.) and OCR models (PaddleOCR, CNOCR, self-training models, etc.) to recognize the same bill, and collect the results output by different models. Then use the comparative analysis agent to summarize, analyze and correct the results. If it is not enough to get the final accurate output, the supervisory decision agent can get an accurate and comprehensive answer by repeating the above operations multiple times. The final output example is as follows: Figure 3 As shown, Figure 3 Some of the required text extraction targets are shown in FIG. It should be additionally noted that the self-training model mentioned above can be trained using public data (such as electronic data from the Internet) and internal bill information as training data sets in an existing manner.

[0075] Corresponding to the above-mentioned extraction method, this embodiment provides an extraction system, which includes a working agent, a supervisory decision agent and an output agent, wherein the working agent is used to extract information and analyze and compare information, the supervisory decision agent is used to decide the next action or end the process according to the output of the working agent, and the output agent is used to output the bill information extraction results to the user.

[0076] The working agent can be divided into a first sub-agent for information extraction and a second sub-agent for information analysis and comparison according to specific functions. There are several first sub-agents, each of which is configured with different models (including different multimodal large models and different OCR models), and the agent can obtain multiple results for analysis and comparison, so as to obtain accurate information extraction output results.

[0077] This extraction system is deployed in a localized manner to avoid the leakage of key privacy content on the public network.

[0078] In this solution, through theoretical expansion, multiple multimodal large models and OCR models can realize diversified information sample collection, and compare and analyze the data extracted by various models, so as to maximize the recognition accuracy and comprehensiveness. In particular, the application of multimodal large models has significant advantages over traditional OCR technology.

[0079] First, multimodal large models have obvious advantages in terms of accuracy. These models can combine the contextual information of images and text for character recognition. They can not only recognize individual characters, but also understand the semantic structure of the entire document, thereby reducing the occurrence of misrecognition. In addition, multimodal large models perform well when processing complex layouts. Traditional OCR often cannot cope with complex layouts such as multi-column text and text embedded in images, while multimodal large models can combine visual and language information to better understand and process these complex layouts and improve recognition accuracy.

[0080] Multimodal large models also excel in processing diverse inputs. They are able to process data in multiple forms, such as text, images, and charts, at the same time. For example, in a report containing text and charts, a multimodal large model can not only recognize the text content, but also extract chart information. At the same time, combined with image processing techniques such as denoising and contrast enhancement, the quality of images input to these models can be significantly improved, thereby further improving the accuracy of text content recognition and extraction by multimodal large models and OCR models.

[0081] In addition, multimodal large models also have great advantages in robustness. They have stronger processing capabilities for low-quality images (such as blur, noise, distortion, etc.), and also have strong processing capabilities for handwritten documents or outdated old documents. By combining the contextual information of images and texts, they can better process and correct low-quality inputs. At the same time, by training on large-scale multilingual and multi-font data, multimodal large models can better recognize texts in different languages ​​and fonts, showing stronger adaptability and generalization capabilities.

[0082] What is more innovative is that this solution upgrades the traditional semi-automatic extraction method to fully automatic extraction. Compared with the traditional method of extracting bill information by relying on only one OCR model, this solution uses an intelligent system to fully automatically perform multiple model operations. Combining different tool functions built with multimodal models and OCR models (tools defined in the previous steps), key information is automatically extracted and structured, reducing the tedious process of manually setting segmentation rules and solving the problem of mismatched and missing information when processing non-standard bills.

[0083] Intelligent agents not only have intelligent perception capabilities and can understand the environment, including visual, auditory, textual and other sensor data, but also can self-reflect and correct errors. They can perform operations multiple times until comprehensive and accurate information is extracted. In addition, intelligent agents have the ability to interact with users in natural language. Users can control the screening and extraction of information through natural language instructions and require the intelligent agent to output the answer in a specified format (such as a key-value pair).

[0084] These agents show great potential in terms of task scalability. They can operate autonomously without human intervention and make independent decisions based on environmental information and task requirements. Agents also have the ability to interact with humans and other agents, understand and respond to natural language instructions, cooperate and collaborate, and ultimately perform specific tasks such as information retrieval, data analysis, and automated operations.

[0085] Finally, the use of OCR technology also faces the problem of data security and privacy protection. In the process of bill processing, the bill contains a lot of sensitive information, such as company name, tax number, amount, etc. Once this information is leaked, it will cause significant losses to enterprises and individuals. At present, the cloud recognition services provided by some OCR service providers are convenient for users, but they also increase the risk of data leakage. The system and method can provide localized deployment to ensure data security and privacy protection.

[0086] In summary, this solution fully utilizes the comprehensive advantages of multimodal large models, OCR models, large language models and intelligent agent systems to significantly improve the accuracy and efficiency of bill information extraction. Its core advantages are reflected in the following aspects:

[0087] 1) Multimodal large models improve accuracy: By integrating multimodal large models, multi-model optimization can be performed in OCR results. Different model results can complement each other and be selected based on merit, significantly improving the accuracy of data extraction. The traditional single method can only rely on a single data source and is difficult to achieve this effect.

[0088] 2) Large language model enhances data processing capabilities: By fine-tuning the large language model and optimizing prompt words, data can be screened, compared, corrected and supplemented in a natural language interactive manner to output results that meet user expectations. This flexible data processing method is difficult to achieve with traditional OCR model systems.

[0089] 3) Intelligent workflow automation processing: The workflow system based on the intelligent agent can automatically review the OCR output results according to user instructions, intelligently determine the next operation, reduce manual intervention, and greatly improve work efficiency.

[0090] 4) Powerful scalability: This system has a highly modular design and can be plugged into different tools or system interfaces. It can realize automatic data transmission and processing through intelligent agent scheduling, support the expansion of more application scenarios, such as automatic docking with other systems, and further improve the automation level of business processes.

[0091] The intelligent agent system provides flexibility and reliability that traditional OCR systems cannot achieve through self-reflection and error correction, natural language interaction, and multiple execution. This makes the solution more adaptable in information extraction and processing, and comprehensively overcomes the limitations of traditional OCR technology.

[0092] The present invention focuses on the application innovation of artificial intelligence agents. The method integrates a highly energy-efficient large language model, a multimodal model, and an optical character recognition model to achieve highly accurate information extraction of bill information.

[0093] This innovative solution fully utilizes the decision-making ability of AI Agent and coordinates multiple core models to form a collaborative intelligent system. The multimodal model combines visual and semantic information to enhance the ability to identify key elements from complex backgrounds. The OCR model accurately captures and converts unstructured text in images, ensuring the accuracy of information extraction. The large language model, with its deep understanding and ability to generate text, processes the text information on the bill and provides standardized extracted information in a way that users want.

[0094] Through such comprehensive application, the present invention not only improves the speed of bill information processing, but also greatly improves the accuracy of the automated processing process, thereby significantly enhancing work efficiency and data reliability in finance, auditing and other related business scenarios.

[0095] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A method for extracting bill information, characterized in that: The steps include: Obtaining an electronic file containing a bill, and preprocessing the electronic file into an image file for easy information extraction; Use the multimodal large model and OCR model to extract the bill information from the image file and output the text information in a structured format; The large language model is used to analyze, correct, and compare the text information output by the multimodal large model and the OCR model. The large language model is fine-tuned or guided by prompt word engineering for different tasks to ensure that the output results meet expectations.

2. A method for extracting bill information according to claim 1, characterized in that: The electronic files include documents in PDF format and image format.

3. A method for extracting bill information according to claim 1, characterized in that: The extracted text information is output according to the format specified by the user.

4. A method for extracting bill information according to claim 3, characterized in that: The user-specified format includes key-value pairs.

5. A bill information extraction system, characterized in that: Adopt the method as claimed in any one of claims 1 to 4; The system includes a working agent and a supervisory decision agent; The working agent is used for information extraction and analysis and comparison; The supervisory decision agent is used to decide the next action or end the process according to the output of the working agent.

6. A bill information extraction system according to claim 5, characterized in that: The working agent includes a first sub-agent for extracting information and a second sub-agent for analyzing and comparing information.

7. A bill information extraction system according to claim 6, characterized in that: There are several first sub-agents, each of which is configured with a different model, and the bill information is extracted and output in a structured manner through the different models.

8. A bill information extraction system according to claim 5, characterized in that: The system also includes an output agent for outputting the bill information extraction results.

9. A bill information extraction system according to claim 5, characterized in that: The system is deployed locally.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores a program or instruction, and the program or instruction executes the steps of the bill information extraction method as described in any one of claims 1-4.

Images