Multi-modal document information processing method, device and equipment based on large model agent and storage medium

Abstract

The invention relates to the technical field of intelligent auditing, in particular to a multi-modal document information processing method, device and equipment based on a large model agent and a storage medium. An input document and an auditing instruction are reasoned through a business information auditing model so as to determine a part to be extracted in document content and an auditing rule; and then calling a corresponding extraction tool to perform information extraction on the document, performing item-by-item check on the extracted content based on an auditing rule, and summarizing check results to obtain a final check result. According to the scheme, through the reasoning ability and the multi-modal data analysis ability of a large model, the verification instruction under the natural language can be understood and converted into the verification logic of the document, and meanwhile key information in the input document can be intelligently recognized and extracted based on the verification logic; according to the scheme, various document types including texts, tables, images and the like can be processed in a large-batch and multi-angle mode, the flexibility of auditing work is enhanced, the checking period is effectively shortened, and the auditing work efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent audit technology, and in particular to a multimodal document information processing method, device, equipment and storage medium based on a large model intelligent agent. Background Art

[0002] With the rapid development of information technology and the deepening of digital transformation, data has become one of the most valuable assets of enterprises. In many industries, the accuracy and compliance of business data are crucial to the decision-making, risk management and business operations of enterprises. However, in the face of massive business data, traditional data verification methods often rely on manual operations, which is not only inefficient but also easily affected by human factors.

[0003] With the increase in business volume and the diversification of data types, the requirements for verification technology are getting higher and higher. Some intelligent auxiliary tools on the market, such as rule-based systems and simple automation tools, can no longer meet the needs of modern enterprises. These methods are particularly inadequate when processing unstructured data (such as contract texts, financial invoices, etc.), especially lacking the ability to understand complex data patterns and analyze the deep connections between various documents. In addition, as business rules continue to change and update, traditional verification systems need to be frequently updated and maintained manually, which not only increases the operating costs of enterprises, but also reduces the flexibility and response speed of verification work.

[0004] Therefore, how to develop a business data verification method that can adapt to diverse data types, understand complex data patterns, automatically update to adapt to changes in business rules, and efficiently analyze the deep connections between documents has become an urgent problem to be solved in this field. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal document information processing method, device, equipment and storage medium based on a large model intelligent agent, aiming to solve the technical problem in the prior art of how to develop a business data verification method that can adapt to diverse data types, understand complex data patterns, automatically update to adapt to changes in business rules, and efficiently analyze the deep connections between documents.

[0006] To achieve the above object, the present invention provides a multimodal document information processing method based on a large model agent, the method comprising the following steps: Obtain documents to be reviewed and verification instruction information; Input the verification instruction information into the business information audit model for reasoning to determine information extraction instruction information and audit rule information; Extracting information from the document to be reviewed according to the information extraction instruction information to obtain a review material extraction result; Based on the audit rule information, the audit material extraction result is verified to obtain the verification result of the document to be audited.

[0007] Optionally, the step of inputting the verification instruction information into a business information audit model for inference to determine information extraction instruction information includes: Input the verification instruction information into the business information audit model for semantic reasoning to obtain the verification task type; Determining, according to the verification task type and the verification case database, an associated document type corresponding to the verification task type; Obtaining verification task association features according to the associated document type and the verification task type; The associated document type and the verification task associated feature are used as the information extraction indication information.

[0008] Optionally, obtaining the verification task association feature according to the associated document type and the verification task type includes: Determining document business characteristics according to the associated document type, wherein the document business characteristics include key business information fields, business data formats, and business logic relationships; Determine verification requirement information according to the verification task; According to the verification requirement information, the verification task associated features in the document business features are determined.

[0009] Optionally, the step of inputting the verification instruction information into a business information audit model for inference to determine audit rule information includes: Inputting the verification instruction information into the business information audit model for semantic reasoning to obtain a semantic reasoning result; Matching the semantic reasoning result with the verification case library to obtain a matching case, and using the verification rule of the matching case as a reference verification rule; The reference audit rules and the verification instruction information are input into the business information audit model for fusion reasoning to obtain the audit rule information corresponding to the current verification task.

[0010] Optionally, extracting information from the document to be reviewed according to the information extraction indication information to obtain a review material extraction result includes: Determining a target document and target extraction content features of the target document according to the information extraction indication information; Determine the location for extracting audit-related information based on the target extracted content features and the document structure of the target document; According to the document structure of the target document and the target extracted content features, the extraction tool under the corresponding content features is called to obtain the audit material extraction result.

[0011] Optionally, before determining the location for extracting the audit-related information based on the target content features and the document structure of the target document, the method further includes: Dividing the target document into content to obtain a text area and a non-text area; By using a text retrieval enhancement model, content feature annotation is performed on the text region to obtain a content feature annotation result of the text region; Using a multimodal data analysis model, content feature annotation is performed on the non-text area to obtain a non-text area feature annotation result; The document structure of the target document is obtained according to the text region content feature labeling result and the non-text region feature labeling result.

[0012] Optionally, the verifying the audit material extraction result based on the audit rule information to obtain the verification result of the document to be audited includes: Based on the audit rule information, the audit material extraction results are grouped and matched according to the audit rule logic to obtain multiple verification subtasks; According to the audit rule logic, data is derived from the audit material extraction results in the verification subtask to obtain derived verification data; Perform multiple checks on the audit material extraction results and the derived verification data to obtain the verification results of the verification subtasks, wherein the multiple checks include value verification, date verification, business logic verification, and relevance verification; The verification results of the multiple verification subtasks are aggregated and deduplicated to obtain the verification result of the document to be reviewed.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also proposes a multimodal document information processing device based on a large model agent, and the multimodal document information processing device based on a large model agent comprises: Data input module, used to obtain documents to be reviewed and verification instruction information; A verification reasoning module, used to input the verification instruction information into the business information audit model for reasoning, and determine the information extraction instruction information and the audit rule information; An information extraction module, used to extract information from the document to be reviewed according to the information extraction indication information, and obtain a review material extraction result; The verification execution module is used to verify the verification material extraction result based on the audit rule information to obtain the verification result of the document to be audited.

[0014] In addition, to achieve the above-mentioned objectives, the present invention also proposes a multimodal document information processing device based on a large model agent, wherein the multimodal document information processing device based on a large model agent comprises: a memory, a processor, and a multimodal document information processing program based on a large model agent stored in the memory and executable on the processor, wherein the multimodal document information processing program based on the large model agent is configured to implement the steps of the multimodal document information processing method based on the large model agent as described above.

[0015] In addition, to achieve the above-mentioned objectives, the present invention also proposes a storage medium, on which a multimodal document information processing program based on a large model intelligent agent is stored. When the multimodal document information processing program based on a large model intelligent agent is executed by a processor, the steps of the multimodal document information processing method based on a large model intelligent agent as described above are implemented.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: the solution can understand the verification instructions in natural language and convert them into the verification logic of the document through the reasoning ability of the big model and the multimodal data analysis ability, and can intelligently identify and extract key business information in the input document based on the verification logic, so that the solution can process various document types including text, tables, and images in large quantities and from multiple angles, thereby enhancing the flexibility of the audit work, effectively shortening the verification cycle, and improving the efficiency of the audit work. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flowchart of the first embodiment of the multimodal document information processing method based on a large model agent of the present invention; Figure 2 It is a flow chart of the second embodiment of the multimodal document information processing method based on a large model agent of the present invention; Figure 3 It is a flowchart of a third embodiment of a multimodal document information processing method based on a large model agent of the present invention; Figure 4It is a flowchart of a fourth embodiment of a multimodal document information processing method based on a large model agent of the present invention; Figure 5 It is a structural block diagram of the first embodiment of the multimodal document information processing device based on the large model agent of the present invention; Figure 6 It is a structural diagram of a multimodal document information processing device based on a large model intelligent agent in the hardware operating environment involved in the embodiment of the present invention.

[0020] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: obtain the document to be reviewed and the verification instruction information; input the verification instruction information into the business information review model for reasoning, and determine the information extraction indication information and the review rule information; according to the information extraction indication information, extract information from the document to be reviewed to obtain the review material extraction result; based on the review rule information, verify the review material extraction result to obtain the verification result of the document to be reviewed.

[0024] Some intelligent auxiliary tools currently on the market, such as rule-based systems and simple automation tools, have been unable to meet the needs of modern enterprises. These tools are particularly inadequate when processing unstructured data (such as contract texts, financial reports, etc.). At the same time, they lack the ability to understand complex data patterns and analyze the deep connections between various documents. As business rules continue to change and update, frequent manual updates and maintenance are required, which not only increases the operating costs of the enterprise, but also reduces the flexibility and response speed of verification work.

[0025] This application provides a solution. Through the powerful reasoning ability and multimodal data analysis ability of the big model, you only need to upload the document to be reviewed and the verification instructions at the natural language level. The big model can perform reasoning on its own, determine the information extraction instruction information and the audit rule information, and then accurately extract information from the document based on this information. Finally, apply the audit rule information to verify the extraction results to obtain accurate verification results. This method not only improves the automation and intelligence level of the verification work, but also reduces the operating costs of the enterprise by reducing manual intervention, and enhances the flexibility and adaptability of the system, enabling enterprises to manage and analyze business data more effectively, thereby gaining an advantage in the fierce market competition.

[0026] Based on this, the embodiment of the present invention provides a multimodal document information processing method based on a large model agent, referring to Figure 1 , Figure 1 It is a flowchart of the first embodiment of a multimodal document information processing method based on a large model agent of the present invention.

[0027] In this embodiment, the multimodal document information processing method based on a large model agent includes the following steps: Step S10: Obtain the document to be reviewed and the verification instruction information.

[0028] It should be noted that the business data verification method of the present application is applicable to business information verification of complex documents such as contracts, financial statements, bills and other documents. For example, when reviewing a contract document, the focus of the review is to ensure whether the key information in the contract is correct, such as the contract number, signing date, amount and other contents. The basis for the verification comes from other types of documents. For example, the amount in the contract document needs to be compared with the data in the invoice, withdrawal slip, and financial statement, and if necessary, it needs to be correlated with time. Therefore, in the method of the present application, the input document to be reviewed is usually more than one, but a collection of multiple different types of documents. In addition, the verification instruction information here refers to input instructions described in natural language. For example, when the input verification instruction information is "extract key information such as contract number, signing date, amount, etc. from the contract to ensure that the total revenue in the financial statements is equal to the sum of the revenues of each department", the business data processing big model of this application can automatically parse out the document types involved in the text and the key information fields in these documents after receiving such natural language instructions, so as to conduct subsequent audit processes by itself. In addition, the instruction "ensure that the total revenue in the financial statements is equal to the sum of the revenues of each department" is the audit rule part of the verification instruction. The audit rule will be converted by the big model into specific audit steps and logic to guide the subsequent information extraction and verification process.

[0029] It is understandable that, because in the traditional mode, auditors determine the types of files that need to be compared based on the audit rules, for example, when comparing account information or amount data, only files with economic data need to be compared, and other document contents such as product research reports and market analysis reports do not need to participate in this verification. Therefore, when there are multiple different audit purposes at the same time, the types of files required for the audit and the valid information in the documents are also different, and the traditional audit requires auditors to group the audit files for each rule. In this application, the staff does not need to perform the corresponding grouping work. It is only necessary to package and upload all the files involved in the audit to the business information audit model. The large model automatically parses and processes these file collections, determines the connection between the documents and the audit rules one by one, and extracts the key information that needs to be compared from each document. This process no longer relies on manual tedious file classification and grouping, thereby greatly simplifying the audit process.

[0030] It should be understood that the core advantage of the business data verification method lies in its automated and intelligent processing capabilities, which has brought significant improvements to the existing audit process in many aspects. First, through the natural language processing capabilities of the large model, the method can directly understand and parse verification instructions in natural language, thereby reducing the reliance on manual interpretation and conversion of instructions. Secondly, the large model can automatically identify and extract key information fields in relevant documents without human intervention, which not only improves efficiency but also reduces audit risks caused by human errors. The verification process of this method can automatically adapt to different audit purposes and rules, dynamically filter out relevant documents from various types of file collections, and perform necessary data comparisons and logical verifications, with a high degree of flexibility and scalability.

[0031] Step S20: input the verification instruction information into the business information audit model for reasoning, and determine the information extraction instruction information and the audit rule information.

[0032] It should be noted that since the business information audit model here is modified based on the large language model, it can identify and analyze various types of documents and understand the business attributes of the documents themselves. Therefore, the audit model can not only understand the various document types involved in the instructions, and understand the role and importance of these documents in the business process, but also understand the actual verification business needs and audit objectives, thereby generating key information fields that need to be paid attention to. For example, contract documents need to pay attention to the contract number, signing date, amount, etc., financial statements need to pay attention to information such as total revenue, net profit, assets and liabilities, and bill documents need to pay attention to the bill's anti-counterfeiting certificate, date, amount, payee information and other elements, and determine the location and format of this information in the corresponding document.

[0033] It can be understood that through the above steps, the data content that needs to be extracted from the input document and the location of the data in the document can be obtained. The business information audit model will automatically generate a set of detailed information extraction instructions. This set of instructions will clearly indicate which documents need to be analyzed, which specific areas or fields need to be extracted, and which document extraction tools need to be called to identify and extract this information from the document.

[0034] It should be understood that the audit model will also determine multiple detailed audit rules based on the semantic analysis results of the verification instruction information. These rules will cover a series of logical verification steps, verification objects and target verification results to ensure that the data involved in the verification is not only accurate but also meets the requirements of the actual business process.

[0035] In one embodiment, the verification instruction information is input into a business information audit model for reasoning to determine the audit rule information, including: inputting the verification instruction information into the business information audit model for semantic reasoning to obtain a semantic reasoning result; matching the semantic reasoning result with a verification case library to obtain a matching case, and using the audit rules of the matching case as a reference audit rule; inputting the reference audit rule and the verification instruction information into the business information audit model for fusion reasoning to obtain the audit rule information corresponding to the current verification task.

[0036] It should be noted that verification instruction information is usually input in natural language, describing the specific requirements of the verification task, such as "check whether the contract amount is consistent with the invoice amount". The business information audit model can understand the intention of these instructions through semantic reasoning and extract key information, such as the goal of the verification task, the document type involved and key fields. Based on these contents, the model compares the semantic reasoning results with historical cases in the case library. The model can quickly find cases similar to the current task and extract their audit rules as reference. This process not only improves the verification efficiency, but also ensures the consistency and accuracy of the verification rules.

[0037] It is understandable that although the reference audit rules provide a reference to historical experience, they may not fully meet the specific needs of the current task. Therefore, the model needs to adjust the reference audit rules in combination with the current verification instruction information to generate audit rule information suitable for the current task. The fusion reasoning process ensures the dynamic adaptability of the audit rules and can flexibly handle diverse verification needs.

[0038] Step S30: extracting information from the document to be reviewed according to the information extraction instruction information to obtain a review material extraction result.

[0039] It should be noted that the process of information extraction includes extracting effective information from text, images, tables and even audio from different types of documents. Different information extraction tools need to be called in this process. For example, the OCR engine is used to extract text information from images or scanned documents and convert unstructured image data into editable and searchable text formats; the table parsing tool is used to identify and extract table data from documents and convert the table content into a structured format, such as CSV or Excel spreadsheets, for further analysis and processing; the image recognition model is used to identify and extract key information from images, such as product images, seals, signatures, etc. These non-text information are different from text content and are often difficult to obtain through traditional text extraction methods; audio transcription tools, if the document contains audio content, such as meeting recordings or voice notes, these tools can convert audio into text for content analysis and review; document layout analysis tools are used to identify the structure and layout of documents, such as titles, paragraphs, lists, etc., which helps to locate and extract specific information more accurately; based on the above tools, it also includes natural language processing plug-ins for understanding and analyzing text content, extracting key entities (such as names of people, places, organization names, etc.), concepts and relationships, and providing a deeper level of text understanding.

[0040] It is understandable that by calling these specialized information extraction tools, the audit model is able to extract valid information from different types of documents, whether they are text, images, tables or audio. This not only improves the accuracy and comprehensiveness of information extraction, but also lays a technical foundation for subsequent audits and analysis. In this way, the audit model can automatically process complex multimodal documents and meet the diverse needs of modern enterprises in data management and auditing.

[0041] It should be understood that the information extraction process is not a comprehensive extraction, but only captures valid information. Since the information extraction indication information has been obtained in the previous steps, it is only necessary to capture the document content of a specific part of a specific document. For example, when verifying the date, it is only necessary to extract the time information of the relevant files and the events anchored by these time information. When verifying the amount, it is only necessary to extract the relevant amount data and ensure the consistency of the data. It is not necessary to extract all the information in the document. This can improve the efficiency of information extraction and shorten the verification cycle.

[0042] Step S40: Based on the audit rule information, the audit material extraction result is verified to obtain the verification result of the document to be audited.

[0043] It is understandable that the verification process is a key link in the entire business data verification method. Its purpose is to verify and compare the extracted audit materials according to the audit rule information to ensure the accuracy, consistency and compliance of the data. This process includes both the correctness of data levels such as amount and time, and the correctness of the logical level. For example, whether the terms in the contract comply with industry standards or company policies, whether the procurement process includes the necessary approval links, and whether the documentation of a certain regulation is complete. Through the above-mentioned verification content, the verification process can not only ensure the numerical accuracy of the data, but also verify the logical relationship and compliance between the data, thereby comprehensively evaluating the quality and reliability of the documents to be audited.

[0044] It should be understood that the verification process requires not only a simple comparison of the extracted data, but also an in-depth analysis combined with business logic. For example, when verifying financial statements, it is necessary not only to check whether the numerical values ​​of the data are consistent, but also to verify whether the logical relationship between the data is reasonable, such as whether the total income is equal to the sum of the income of each department, whether the balance sheet is balanced, etc. In addition, the verification process also needs to consider time relevance, such as whether the signing date of the contract is consistent with the date of the relevant bill, etc. The audit model will strictly follow these rules to verify the extracted audit materials one by one.

[0045] This embodiment obtains the document to be reviewed and the verification instruction information; inputs the verification instruction information into the business information review model for reasoning to determine the information extraction instruction information and the review rule information; extracts information from the document to be reviewed based on the information extraction instruction information to obtain the review material extraction result; verifies the review material extraction result based on the review rule information to obtain the verification result of the document to be reviewed.

[0046] In summary, in this embodiment, the business information audit model is used to infer the input documents and verification instructions to determine the part to be extracted in the document content and the audit rules, and then the corresponding extraction tool is called to extract information from the document, and then the extracted content is checked item by item based on the audit rules, and the verification results are summarized to obtain the final verification results. Through the reasoning ability and multimodal data analysis ability of the large model, this solution can understand the verification instructions in natural language and convert them into the verification logic of the document. At the same time, it can intelligently identify and extract key business information in the input document based on the verification logic, so that the solution can process a variety of document types including text, tables, and images in large quantities and from multiple angles, enhancing the flexibility of the audit work, effectively shortening the verification cycle, and improving the efficiency of the audit work.

[0047] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2, the step S20 further includes: Step S201: input the verification instruction information into the business information audit model for semantic reasoning to obtain the verification task type.

[0048] It should be noted that since verification instruction information allows input in the form of natural language, this method provides great convenience for relevant staff, allowing customers to describe verification requirements in an intuitive and flexible manner without following a strict format or coded language.

[0049] It is understandable that semantic reasoning is at the core of this process. Through semantic reasoning, the audit model can extract key information from natural language instructions, such as the goal of the verification task, the document types involved, and the key fields that need to be focused on. For example, when the input verification instruction is "extract the contract number, signing date, and amount from the contract, and ensure that the amount is consistent with the invoice", the model can identify the task type as "contract and invoice amount verification" and clearly specify that the document types to be processed are contracts and invoices.

[0050] It should be understood that semantic reasoning not only relies on the understanding of vocabulary and grammar, but also needs to be combined with context and business background knowledge. This is because natural language is often ambiguous and diverse, and the same vocabulary may have different meanings in different contexts. For example, "amount" may refer to the total contract amount, the advance payment amount, or the invoice amount, and the specific meaning needs to be inferred based on the context. In addition, the model also needs to be able to handle complex instructions, such as those involving multiple verification tasks or cross-document relevance verification. Through powerful semantic reasoning capabilities, the audit model can convert natural language instructions into specific verification task types, laying the foundation for subsequent audit processes and ensuring the accuracy and efficiency of the entire verification process.

[0051] Step S202: Determine the associated document type corresponding to the verification task type according to the verification task type and the verification case database.

[0052] It should be noted that due to the strong repetitiveness and regularity of the audit process of business data, many verification tasks involve similar document types and verification logic in different scenarios. For example, in contract audits, it is usually necessary to associate documents such as invoices, financial statements, or project progress reports to verify the execution of contract terms and the accuracy of the amounts. Therefore, a verification case database is added, and a large number of historical verification tasks and their corresponding document types and verification logic are accumulated to provide an important reference for the associated document types of the current verification tasks.

[0053] It is understandable that when the verification tasks are consistent or similar, the document types they call are also similar. The model can quickly identify patterns similar to the current task by comparing historical cases, and infer the document types that need to be associated based on this.

[0054] Step S203: Obtain verification task association features according to the associated document type and the verification task type.

[0055] It should be noted that verification task-related features refer to key information fields in document types that are directly related to the task objectives in a specific verification task. These fields are the core elements for completing the verification task and they directly affect the accuracy and efficiency of the verification.

[0056] It is understandable that if the current task is "check whether the contract amount is consistent with the invoice amount", the model will identify through the case database that this task usually involves contract documents and invoice documents, and further determine the key fields that need to be extracted, such as contract number, contract amount, invoice number and invoice amount. In this case, contract and invoice are related document types, and contract number, contract amount, invoice number and invoice amount are related features of the verification task.

[0057] In one embodiment, the verification task association characteristics are obtained based on the associated document type and the verification task type, including: determining document business characteristics based on the associated document type, the document business characteristics including key business information fields, business data formats and business logic relationships; determining verification requirement information based on the verification task; determining the verification task association characteristics in the document business characteristics based on the verification requirement information.

[0058] It should be noted that the determination of document business features is based on the understanding of document types and the analysis of business background knowledge. Different types of documents have different business features, and business features are functionally divided into key business information fields, business data formats, and business logic relationships.

[0059] It can be understood that key business information fields refer to fields in the document that are closely related to business logic. For example, in a contract document, key fields include contract number, signing date, contract amount, information of both parties, etc. In an invoice document, key fields may include invoice number, invoice amount, invoice date, payee information, etc.; business data format refers to the data type and format requirements of the field. For example, the date field may need to be filled in the "YYYY-MM-DD" format, and the amount field needs to be extracted in a numeric format and retain two decimal places. Business logic relationship refers to the logical relationship between fields. For example, in a financial statement, total revenue should be equal to the sum of revenues of each department; in a contract, the contract amount should be consistent with the invoice amount.

[0060] It should be understood that the determination of associated features is based on the combination of verification requirement information and document business features. The model analyzes the verification requirement information and selects fields directly related to the current task from the document business features. For example, although a contract document may contain many fields (such as signing date, information of both parties, etc.), in the amount verification task, only the contract number and contract amount are fields directly related to the verification requirement, so they are determined as associated features.

[0061] Step S204: using the associated document type and the verification task associated features as the information extraction indication information.

[0062] It should be noted that different verification tasks may involve different associated features. For example, the associated features of contract and invoice verification include contract number, contract amount, invoice number and invoice amount; the associated features of financial statement verification may include total revenue, net profit, assets and liabilities and other fields; the associated features of bill review may include bill number, amount, payee information, bill date and other forms. In addition, associated features include not only text fields, but also seals and signatures in images, or other forms of key information in audio.

[0063] It can be understood that through the above steps, the model can dynamically generate verification task related features according to the associated document type and verification task type, providing clear guidance for subsequent information extraction and verification. This feature-based extraction method not only improves the accuracy and efficiency of verification, but also ensures the pertinence and flexibility of the verification process.

[0064] In this embodiment, the verification instruction information is input into the business information audit model for semantic reasoning to obtain the verification task type; based on the verification task type and the verification case database, the associated document type corresponding to the verification task type is determined; based on the associated document type and the verification task type, the verification task associated feature is obtained; and the associated document type and the verification task associated feature are used as the information extraction indication information.

[0065] In summary, the business data processing model in this embodiment realizes the automatic conversion from natural language instructions to specific verification tasks through natural language processing and semantic reasoning technology, which greatly simplifies the work intensity of business data auditors. This way of conveying audit intentions allows relevant personnel to describe verification requirements in natural language without following strict formats or coded languages, which reduces the entry threshold for relevant personnel. Secondly, combined with the verification case database, the model can quickly identify the document types and key fields related to the task, avoiding the tedious process of manually screening documents and fields, and significantly improving the verification efficiency. In addition, by dynamically generating information extraction indication information, the model can accurately locate key information, reduce the interference of irrelevant data, and ensure the accuracy of the verification results. At the same time, the method supports multimodal data processing, and can process various types of documents such as text, images, and tables to meet the diversified verification needs of large batches of files in complex business scenarios. Therefore, this solution not only improves the efficiency and accuracy of business data verification, but also reduces labor costs and improves the intelligence level of enterprise data management and auditing.

[0066] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S30, comprising: Step S301: Determine a target document and target extraction content features of the target document according to the information extraction indication information.

[0067] It can be understood that the information extraction instruction information is a detailed instruction based on the audit rules and verification task related features generated in the previous steps. It provides the information extraction module with clear document type and key field information to ensure the pertinence and efficiency of the extraction process; the target document refers to the document type directly related to the current verification task, and the target extraction content features refer to the key information fields that need to be extracted from these documents; by clarifying the target document and the target extraction content features, the model can accurately locate the required information and avoid processing irrelevant content, thereby improving the efficiency and accuracy of information extraction.

[0068] Step S302: Determine the location for extracting audit-related information based on the target extraction content features and the document structure of the target document.

[0069] It should be noted that the document structure of the target document refers to the layout and organization of the document, including the distribution of elements such as titles, paragraphs, tables, and images. Understanding the document structure can help the model determine the specific location of the target content features in the document.

[0070] It is understandable that the process of determining the location of information extraction requires analysis in combination with the target content features and document structure. For example, in a contract document, "Contract Amount" may be located in the "Amount Clause" section, while "Contract Number" may be located in the title section at the beginning of the document. By analyzing the document structure, the model can quickly locate the specific location of these key fields, thereby improving the efficiency and accuracy of information extraction.

[0071] It should be understood that document structure analysis is not only applicable to text data, but also supports the processing of multimodal data. For example, for documents containing images (such as contracts with seals), the model can determine the location of the seal through image recognition technology; for table data, the model can locate the data in a specific cell through table parsing tools. This multimodal support capability ensures the comprehensiveness and accuracy of information extraction.

[0072] In one embodiment, before the content features and the document structure of the target document are extracted according to the target and the audit-related information extraction position is determined, it also includes: dividing the target document into content areas and obtaining text areas and non-text areas; annotating the text areas with content features through a text retrieval enhancement model to obtain text area content feature annotation results; annotating the non-text areas with content features through a multimodal data analysis model to obtain non-text area feature annotation results; and obtaining the document structure of the target document based on the text area content feature annotation results and the non-text area feature annotation results.

[0073] It should be noted that in order to more efficiently determine the location for extracting audit-related information, the model will first perform content division and feature annotation on the target document before performing information extraction, including dividing the target document into text areas and non-text areas, and annotating content features of different areas, thereby generating complete and clear document structure information.

[0074] It is understandable that the text area contains directly readable text content, such as contract terms, text descriptions in financial statements, text information in invoices, etc., and the non-text area contains non-text content such as images, tables, charts, audio, signatures, seals, etc. For example, seals in contracts, QR codes in invoices, charts in financial statements, etc. Through content division, the model can clearly identify the distribution of different types of data in the document, providing a basis for subsequent feature annotation and information extraction.

[0075] It should be understood that the text retrieval enhancement model is a model based on natural language processing (NLP) that can identify and annotate information content in text. The model will annotate text information related to the target content features according to the needs of the verification task. For example, in a contract document, the location and content of fields such as "contract number", "contract amount", and "signing date" are annotated to facilitate subsequent content extraction. On the other hand, the multimodal data analysis model is a model that can process multiple data types (such as images, tables, and audio) and can identify and annotate key information in non-text areas. Similarly, the model will annotate non-text information related to the target content features according to the needs of the verification task. For example, in a contract document, the location and type of the seal are annotated; in an invoice document, the location of the QR code or barcode is annotated.

[0076] Step S303: According to the document structure of the target document and the target extracted content features, the extraction tool under the corresponding content features is called to obtain the audit material extraction result.

[0077] It is understandable that after determining the document structure and target content features of the target document, the model needs to call the appropriate extraction tool based on this information to complete the data extraction. Different content features require different extraction tools, such as text content requires natural language processing tools, image content requires OCR engines, table content requires table parsing tools, and audio content requires audio transcription tools. By accurately calling these tools, the model can efficiently extract the required key information from the document.

[0078] It should be understood that the selection of extraction tools is based on the dynamic matching of the target extraction content features and the document structure. For example, for the same document, if the target extraction content feature is "contract amount" and the field is located in a table cell in the document, the model will call the table parsing tool to extract the data; if the target extraction content feature is "seal image", the model will call the image recognition tool to extract the seal information. This dynamic matching mechanism ensures that the selection of extraction tools can adapt to different audit needs and improve the flexibility and efficiency of information extraction. At the same time, the calling process of the extraction tool is highly automated and can be dynamically adjusted according to different verification tasks and document types. For example, for documents containing audio content, the model will call the audio transcription tool to convert the audio into text and then extract key information. In addition, by combining document structure information and target extraction content features, the model can accurately locate the location of the required information, avoid processing irrelevant content, and further improve the efficiency of information extraction.

[0079] In this embodiment, the target document and its target extraction content features are clarified according to the information extraction indication information to ensure the pertinence and efficiency of the extraction process. Secondly, the target document is divided into content, and the text area and non-text area are distinguished. The text and non-text areas are annotated with content features respectively through the text retrieval enhancement model and the multimodal data analysis model to generate complete document structure information. Finally, in combination with the document structure and the target extraction content features, the corresponding extraction tools (such as NLP tools, OCR engines, table parsing tools, etc.) are dynamically called to accurately extract the required key information. This process supports multimodal data processing and can adapt to the diverse verification task requirements.

[0080] In summary, this embodiment significantly improves the efficiency and accuracy of information extraction through refined content division, feature annotation and dynamic tool calling, avoids the processing of irrelevant information, reduces redundant operations, and improves extraction efficiency. Secondly, by annotating documents through text retrieval enhancement models and multimodal data analysis models, the model can accurately locate the location of key information to ensure the comprehensiveness and accuracy of extraction. In addition, the dynamic calling of extraction tools can be flexibly adjusted according to different document types and content features, adapting to complex business scenarios, reducing manual intervention, and reducing error rates. This intelligent information extraction method not only improves the efficiency of audit tasks, but also improves the flexibility and adaptability of data processing, reduces operating costs, and improves overall business efficiency.

[0081] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S40, comprising: Step S401: Based on the audit rule information, the audit material extraction results are grouped and paired according to the audit rule logic to obtain multiple verification subtasks.

[0082] It should be noted that since audit rules are generally not just for a single type of data verification, there may be multiple types of data (such as values, dates, text, etc.) and complex logical relationships (such as amount verification, date order, text consistency, etc.) at the same time. Therefore, the audit rule logic needs to group and pair the extracted audit materials according to different verification requirements to form multiple specific verification subtasks. For example, an audit rule may require simultaneous verification of whether the contract amount is consistent with the invoice amount (value verification) and whether the contract signing date is earlier than the invoice issuance date (date verification). By decomposing these different verification requirements into multiple subtasks, the model can handle complex audit rules more efficiently and ensure that each audit point can be accurately verified.

[0083] Step S402: According to the audit rule logic, data is derived from the audit material extraction results within the verification subtask to obtain derived verification data.

[0084] It should be noted that the extracted data needs to be further processed before subsequent verification. For example, when verifying whether the sum of each quarter's expenditure data is equal to the total annual expenditure, the model needs to extract the information from each quarter's documents and perform addition calculations, and compare the calculation results with the extracted annual expenditure data. The addition calculation results here are the derived verification data.

[0085] It is understandable that the types of derived verification data are not limited to numerical derivation, but also include logical derivation. For example, in contract review, the model may need to deduce the logical result of whether a guarantee is required based on the conditions in the contract terms (such as "guarantee is required when the contract amount is greater than 100,000 yuan"), and based on this, check whether the uploaded file contains relevant supporting documents to obtain the verification result. This kind of logical derivation is also a form of data derivation, which can help the model verify more complex business rules.

[0086] It should be understood that the purpose of data derivation is to meet the complex logical verification requirements in the audit rules and ensure that the verification process can cover a wider range of business scenarios. By generating derived data, the model can convert the original extracted data into more meaningful verification data, thereby improving the accuracy and comprehensiveness of the verification.

[0087] Step S403: Perform multiple checks on the audit material extraction results and the derived verification data to obtain the verification results of the verification subtasks, wherein the multiple checks include numerical verification, date verification, business logic verification, and correlation verification.

[0088] It should be noted that since the audit rules may involve verification in multiple dimensions, a single-dimensional verification often cannot fully cover all requirements. Therefore, the model needs to be comprehensively verified and reviewed in multiple dimensions such as numerical verification, date verification, business logic verification, and relevance verification to generate reliable verification results.

[0089] Understandably, numerical verification is mainly used to verify the numerical consistency of data. For example, verify whether the contract amount is consistent with the invoice amount, or whether the total revenue in the financial statement is equal to the sum of the revenues of each department. Date verification is used to verify the logic and sequence of dates, such as whether the contract signing date is earlier than the execution date, or whether the invoice issuance date is within the validity period of the contract. Business logic verification is used to verify whether the data complies with business rules, such as whether the financial statements are balanced, or whether the contract terms comply with the terms set by the company. Relevance verification is used to verify the relationship between different data, such as whether the contract number is consistent with the invoice number, or whether the project progress is consistent with the contract agreement.

[0090] It should be understood that through multiple checks, the model can verify the data from multiple angles to ensure the accuracy and reliability of the verification results. This multi-dimensional verification method can not only identify potential errors and anomalies, but also discover possible logical contradictions between data, thereby providing strong support for subsequent decision-making. In addition, multiple checks can also improve the flexibility and adaptability of the verification process, enabling it to cope with complex business scenarios and diverse audit needs.

[0091] Step S404: Summarize and remove duplicate verification results of the multiple verification subtasks to obtain the verification result of the document to be reviewed.

[0092] It is understandable that after completing the verification of multiple verification subtasks, each subtask will generate independent verification results, which may contain repeated or redundant verification conclusions, and there may also be conflicts in the verification conclusions. Therefore, in order to generate a clear, accurate and easy-to-understand final verification report, the model needs to summarize and deduplicate these scattered verification results. For example, multiple subtasks may verify the consistency of the contract amount and the invoice amount. When summarizing, only one verification result needs to be retained to avoid duplicate reports.

[0093] It should be understood that the process of aggregation and deduplication not only ensures the integrity and consistency of the verification results, but also improves the readability and practicality of the verification report. By integrating the results of all subtasks, the model can generate a comprehensive verification report that reflects the overall situation of this batch of documents to be reviewed. At the same time, removing duplicate information can avoid redundancy, reduce unnecessary interference, and ensure the simplicity and accuracy of the verification results.

[0094] In this embodiment, based on the audit rule information, the audit material extraction results are grouped and paired according to the audit rule logic to obtain multiple verification sub-tasks; based on the audit rule logic, data is derived from the audit material extraction results within the verification sub-task to obtain derived verification data; multiple verifications are performed on the audit material extraction results and the derived verification data to obtain the verification results of the verification sub-task, and the multiple verifications include numerical verification, date verification, business logic verification and correlation verification; the verification results of the multiple verification sub-tasks are summarized and deduplicated to obtain the verification results of the document to be audited.

[0095] In summary, this embodiment significantly improves the efficiency and accuracy of business document verification through the process of group matching, data derivation, multiple verification and summary deduplication. First, group matching decomposes complex audit rules into multiple subtasks, simplifies the verification process, and improves processing efficiency. Secondly, data derivation can generate valid data related to the review rules in a targeted manner to meet the complex business logic verification requirements. Multiple verification verifies data from multiple dimensions, which can effectively identify potential errors and logical contradictions and improve the reliability of verification results. Finally, summary deduplication integrates the results of all subtasks, avoids duplication and redundant information, generates concise and clear verification reports, and improves the readability and practicality of the reports. This systematic verification method not only reduces manual intervention and reduces error rates, but also improves the data management and decision support capabilities of enterprises. It is particularly suitable for complex multimodal data verification scenarios and has broad application prospects and practical value.

[0096] This application also provides a multimodal document information processing device based on a large model agent, please refer to Figure 5 , the multimodal document information processing device based on the large model agent comprises: Data input module 10, used to obtain documents to be reviewed and verification instruction information; A verification reasoning module 20 is used to input the verification instruction information into the business information audit model for reasoning, and determine the information extraction instruction information and the audit rule information; An information extraction module 30 is used to extract information from the document to be reviewed according to the information extraction indication information to obtain a review material extraction result; The verification execution module 40 is used to verify the verification material extraction result based on the verification rule information to obtain the verification result of the document to be reviewed.

[0097] In one embodiment, the verification reasoning module 20 is also used to input the verification instruction information into the business information audit model for semantic reasoning to obtain the verification task type; determine the associated document type corresponding to the verification task type based on the verification task type and the verification case database; obtain the verification task association feature based on the associated document type and the verification task type; and use the associated document type and the verification task association feature as the information extraction indication information.

[0098] In one embodiment, the verification reasoning module 20 is also used to determine the document business characteristics based on the associated document type, and the document business characteristics include key business information fields, business data formats, and business logic relationships; determine the verification requirement information based on the verification task; and determine the verification task-related characteristics in the document business characteristics based on the verification requirement information.

[0099] In one embodiment, the information extraction module 30 is also used to input the verification instruction information into the business information audit model for semantic reasoning to obtain semantic reasoning results; match the semantic reasoning results with the verification case library to obtain matching cases, and use the audit rules of the matching cases as reference audit rules; input the reference audit rules and the verification instruction information into the business information audit model for fusion reasoning to obtain the audit rule information corresponding to the current verification task.

[0100] In one embodiment, the information extraction module 30 is also used to determine the target document and the target extraction content features of the target document based on the information extraction indication information; determine the audit-related information extraction location based on the target extraction content features and the document structure of the target document; and call the extraction tool under the corresponding content features based on the document structure of the target document and the target extraction content features to obtain the audit material extraction results.

[0101] In one embodiment, the information extraction module 30 is further used to divide the target document into content areas and obtain text areas and non-text areas; perform content feature annotation on the text areas through a text retrieval enhancement model to obtain text area content feature annotation results; perform content feature annotation on the non-text areas through a multimodal data analysis model to obtain non-text area feature annotation results; and obtain the document structure of the target document based on the text area content feature annotation results and the non-text area feature annotation results.

[0102] In one embodiment, the verification execution module 40 is also used to group and pair the audit material extraction results according to the audit rule logic based on the audit rule information to obtain multiple verification sub-tasks; according to the audit rule logic, derive data on the audit material extraction results within the verification sub-task to obtain derived verification data; perform multiple verifications on the audit material extraction results and the derived verification data to obtain the verification results of the verification sub-task, and the multiple verifications include numerical verification, date verification, business logic verification and correlation verification; summarize and remove duplicate verification results of the multiple verification sub-tasks to obtain the verification results of the document to be reviewed.

[0103] This application uses the business information audit model to infer the input documents and verification instructions to determine the parts to be extracted from the document content and the audit rules, then calls the corresponding extraction tool to extract information from the document, and then verifies the extracted content item by item based on the audit rules, and summarizes the verification results to obtain the final verification results. Through the reasoning ability and multimodal data analysis ability of the large model, this solution can understand the verification instructions in natural language and convert them into the verification logic of the document. At the same time, it can intelligently identify and extract key business information in the input document based on the verification logic, so that the solution can process a variety of document types including text, tables, and images in large quantities and from multiple angles, enhancing the flexibility of the audit work, effectively shortening the verification cycle, and improving the efficiency of the audit work.

[0104] The multimodal document information processing device based on a large model agent provided by the present application adopts the multimodal document information processing method based on a large model agent in the above-mentioned embodiment, which can solve the technical problem of how to develop a business data verification method that can adapt to diverse data types, understand complex data patterns, automatically update to adapt to changes in business rules, and efficiently analyze the deep connections between documents. Compared with the prior art, the beneficial effects of the multimodal document information processing device based on a large model agent provided by the present application are the same as the beneficial effects of the multimodal document information processing method based on a large model agent provided by the above-mentioned embodiment, and the other technical features of the multimodal document information processing device based on a large model agent are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0105] The present application provides a multimodal document information processing device based on a large model intelligent agent, and the multimodal document information processing device based on a large model intelligent agent includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal document information processing method based on the large model intelligent agent in the above-mentioned embodiment one.

[0106] Reference below Figure 6 , which shows a schematic diagram of the structure of a large model agent-based multimodal document information processing device suitable for implementing the embodiment of the present application. The large model agent-based multimodal document information processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The multimodal document information processing device based on a large model agent is shown as merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0107] like Figure 6 As shown, the multimodal document information processing device based on the large model agent may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 to the RAM (Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the multimodal document information processing device based on the large model agent are also stored. The processing device 1001, the ROM 1002 and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multimodal document information processing device based on the large model agent to communicate with other devices wirelessly or by wire to exchange data. Although the multimodal document information processing device based on the large model agent with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0108] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0109] The multimodal document information processing device based on a large model agent provided by the present application adopts the multimodal document information processing method based on a large model agent in the above-mentioned embodiment, which can solve the technical problem of how to develop a business data verification method that can adapt to diversified data types, understand complex data patterns, automatically update to adapt to changes in business rules, and efficiently analyze the deep connections between documents. Compared with the prior art, the beneficial effects of the multimodal document information processing device based on a large model agent provided by the present application are the same as the beneficial effects of the multimodal document information processing method based on a large model agent provided by the above-mentioned embodiment, and the other technical features of the multimodal document information processing device based on a large model agent are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0110] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0111] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0112] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the multimodal document information processing method based on a large model agent in the above-mentioned embodiment.

[0113] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0114] The above-mentioned computer-readable storage medium may be included in the multimodal document information processing device based on the large model intelligent agent; or it may exist independently without being assembled into the multimodal document information processing device based on the large model intelligent agent.

[0115] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a multimodal document information processing device based on a large model intelligent agent, the multimodal document information processing device based on a large model intelligent agent: obtains the document to be reviewed and the verification instruction information; inputs the verification instruction information into the business information review model for reasoning, and determines the information extraction indication information and the review rule information; extracts information from the document to be reviewed according to the information extraction indication information, and obtains the review material extraction result; based on the review rule information, verifies the review material extraction result, and obtains the verification result of the document to be reviewed.

[0116] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0119] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal document information processing method based on a large model agent, and can solve the technical problem of how to develop a business data verification method that can adapt to diverse data types, understand complex data patterns, automatically update to adapt to changes in business rules, and efficiently analyze the deep connections between documents. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the multimodal document information processing method based on a large model agent provided by the above-mentioned embodiment, and will not be repeated here.

[0120] The computer program product provided by the present application can solve the technical problem of multimodal document information processing based on large model agents. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the multimodal document information processing method based on large model agents provided by the above-mentioned embodiments, and will not be repeated here.

[0121] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multimodal document information processing method based on a large model agent, characterized in that: The multimodal document information processing method based on a large model agent comprises: Obtain documents to be reviewed and verification instruction information; Input the verification instruction information into the business information audit model for reasoning to determine information extraction instruction information and audit rule information; Extracting information from the document to be reviewed according to the information extraction instruction information to obtain a review material extraction result; Based on the audit rule information, the audit material extraction result is verified to obtain the verification result of the document to be audited.

2. The multimodal document information processing method based on a large model agent according to claim 1 is characterized in that: The step of inputting the verification instruction information into the business information audit model for reasoning to determine the information extraction instruction information includes: Input the verification instruction information into the business information audit model for semantic reasoning to obtain the verification task type; Determining, according to the verification task type and the verification case database, an associated document type corresponding to the verification task type; Obtaining verification task association features according to the associated document type and the verification task type; The associated document type and the verification task associated feature are used as the information extraction indication information.

3. The multimodal document information processing method based on large model agent according to claim 2 is characterized in that: The obtaining of the verification task association feature according to the associated document type and the verification task type includes: Determining document business characteristics according to the associated document type, wherein the document business characteristics include key business information fields, business data formats, and business logic relationships; Determine verification requirement information according to the verification task; According to the verification requirement information, the verification task associated features in the document business features are determined.

4. The multimodal document information processing method based on large model agent according to claim 1 is characterized in that: The step of inputting the verification instruction information into the business information audit model for inference to determine the audit rule information includes: Inputting the verification instruction information into the business information audit model for semantic reasoning to obtain a semantic reasoning result; Matching the semantic reasoning result with the verification case library to obtain a matching case, and using the verification rule of the matching case as a reference verification rule; The reference audit rules and the verification instruction information are input into the business information audit model for fusion reasoning to obtain the audit rule information corresponding to the current verification task.

5. The multimodal document information processing method based on large model agent according to claim 1 is characterized in that: The extracting information from the document to be reviewed according to the information extraction indication information to obtain the review material extraction result includes: Determining a target document and target extraction content features of the target document according to the information extraction indication information; Determine the location for extracting audit-related information based on the target extracted content features and the document structure of the target document; According to the document structure of the target document and the target extracted content features, the extraction tool under the corresponding content features is called to obtain the audit material extraction result.

6. The multimodal document information processing method based on large model agent according to claim 5 is characterized in that: Before determining the location for extracting audit-related information based on the target content features and the document structure of the target document, the method further includes: Dividing the target document into text areas and non-text areas; By using a text retrieval enhancement model, content feature annotation is performed on the text region to obtain a content feature annotation result of the text region; Using a multimodal data analysis model, content feature annotation is performed on the non-text area to obtain a non-text area feature annotation result; The document structure of the target document is obtained according to the text region content feature labeling result and the non-text region feature labeling result.

7. The multimodal document information processing method based on large model agent according to claim 1 is characterized in that: The step of verifying the audit material extraction result based on the audit rule information to obtain the verification result of the document to be audited includes: Based on the audit rule information, the audit material extraction results are grouped and matched according to the audit rule logic to obtain multiple verification subtasks; According to the audit rule logic, data is derived from the audit material extraction results in the verification subtask to obtain derived verification data; Perform multiple checks on the audit material extraction results and the derived verification data to obtain the verification results of the verification subtasks, wherein the multiple checks include value verification, date verification, business logic verification, and relevance verification; The verification results of the multiple verification subtasks are aggregated and deduplicated to obtain the verification result of the document to be reviewed.

8. A multimodal document information processing device based on a large model agent, characterized in that: The multimodal document information processing device based on a large model agent comprises: Data input module, used to obtain documents to be reviewed and verification instruction information; A verification reasoning module, used to input the verification instruction information into the business information audit model for reasoning, and determine the information extraction instruction information and the audit rule information; An information extraction module, used to extract information from the document to be reviewed according to the information extraction indication information, and obtain a review material extraction result; The verification execution module is used to verify the verification material extraction result based on the audit rule information to obtain the verification result of the document to be audited.

9. A multimodal document information processing device based on a large model agent, characterized in that: The multimodal document information processing device based on a large model agent comprises: a memory, a processor, and a multimodal document information processing program based on a large model agent stored in the memory and executable on the processor, wherein the multimodal document information processing program based on a large model agent is configured to implement the steps of the multimodal document information processing method based on a large model agent as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a multimodal document information processing program based on a large model intelligent agent, and when the multimodal document information processing program based on a large model intelligent agent is executed by a processor, the steps of the multimodal document information processing method based on a large model intelligent agent are implemented as described in any one of claims 1 to 7.

Images