Big language model-based document intelligent auditing method and system, and medium

Abstract

The invention discloses an intelligent document auditing method and system based on a large language model and a medium, and the method comprises the steps: receiving an intelligent auditing request, and obtaining corresponding structured document data, document form structure definition and auditing rules; generating an auditing task based on the rule and the data; calling a large language model service processing task, and enabling the large language model to perform semantic analysis and compliance reasoning on the data based on the structure definition by taking the rule, the data and the structure definition as input together; and receiving and collecting structured auditing results returned by the medium-large language model, forming a response, and returning the response. According to the method, the document form structure definition is introduced to serve as a semantic translator, the natural language understanding ability of enterprise structured data and a large language model is effectively bridged, efficient, accurate and reason-traceable automatic intelligent auditing of business documents such as reimbursement bills is achieved, and the limitation of traditional manual auditing or rule engines is overcome.

Description

Technical Field

[0001] This invention relates to the field of enterprise financial information technology, and in particular to a method, system and medium for intelligent document review based on a large language model. Background Technology

[0002] In the field of corporate expense management, the review of expense reports is a demanding and critical task. Currently, companies typically rely on the rule engines built into expense reporting systems or entirely manual methods for review. While rule-based review methods can achieve partial automation, their review logic depends on preset, fixed logical judgments, making it difficult to handle complex scenarios not explicitly covered by rules, effectively understand and review unstructured data (such as key information in invoice images), and cross-check with semi-structured data. This results in poor flexibility and limited coverage. On the other hand, purely manual review, especially with large volumes of documents, suffers from low efficiency, high labor costs, and is prone to inconsistent review standards, increased error rates, and omissions of anomalies due to fatigue. Therefore, existing technological solutions cannot simultaneously guarantee review efficiency while achieving accurate, automated, and interpretable intelligent review of complex business scenarios and unstructured and semi-structured data. Summary of the Invention

[0003] To address the problem that existing manual and rule-based review methods cannot efficiently and accurately handle the cross-review of unstructured data and complex semantics, this invention proposes a document intelligent review method, system, and medium based on a large language model.

[0004] The specific technical solution is as follows:

[0005] A document intelligent verification method based on a large language model includes the following steps:

[0006] Receive intelligent review request, wherein the intelligent review request includes at least the identification information of the document to be reviewed;

[0007] In response to the intelligent audit request, obtain the structured document data, document form structure definition and at least one audit rule corresponding to the document's identification information;

[0008] Based on the audit rules and the structured document data, generate one or more audit tasks;

[0009] The large language model service is invoked to process the audit task, wherein the audit rules, structured document data, and document form structure definition are provided to the large language model as input information, and semantic parsing and compliance reasoning are performed on the structured document data based on the document form structure definition.

[0010] Receive the structured audit results returned by the large language model for each audit task;

[0011] The structured audit results are aggregated to form an audit response and returned. By inputting the document form structure definition, structured data, and audit rules into the large language model, the key technical obstacle that LLM cannot directly understand the semantics of specific structured data in business systems is fundamentally solved, achieving a leap from simple keyword matching to deep semantic understanding and reasoning.

[0012] Furthermore, the structured document data includes the expense report master data and at least one associated expense detail item data;

[0013] The acquisition steps include: obtaining detailed information about the expense report form structure definition, expense detail item form structure definition, expense report master data, expense detail item data, and the audit rules by calling multiple external data interfaces in parallel. Parallel calls to multiple external data interfaces significantly improve the efficiency of the data acquisition stage, avoid delays caused by serial requests, and thus shorten the overall response time of the entire audit task. This is particularly suitable for expense report scenarios with complex data associations and large data volumes.

[0014] Furthermore, the step of generating one or more audit tasks based on audit rules includes:

[0015] Match the scope of the audit rules with the type of each expense item;

[0016] Generate an independent audit task for each successfully matched cost item and audit rule combination;

[0017] A skip-review task flag is generated for each mismatched combination. By introducing a matching mechanism between rule scope and fee type, accurate distribution of review tasks is achieved, unnecessary LLM calls are avoided, and computational resources and costs are saved. At the same time, generating skip-review task flags ensures the integrity and traceability of the review process, making the review logic more rigorous.

[0018] Furthermore, the step of calling the large language model service to process the audit task is executed in parallel;

[0019] Rate limiting is implemented to control the frequency of requests to the large language model service. Parallel execution of LLM audit tasks significantly improves the audit throughput for batch or complex documents. Combined with rate limiting, excessive requests to the LLM service are effectively prevented, ensuring the stability and robustness of the entire system and avoiding system crashes due to upstream service overload.

[0020] Furthermore, the steps of the method are executed through a state machine workflow engine. This workflow engine defines multiple nodes, including at least: a data acquisition node, a rule processing node, an LLM review node, and a result aggregation node. The execution of each node transmits context data through state objects. By orchestrating the review process through the state machine workflow engine, complex review business logic is decomposed into standardized, manageable nodes, making the entire process clear, controllable, and easy to maintain and extend. Transmitting context through state objects ensures the consistency and integrity of data flow between nodes.

[0021] Furthermore, the state machine workflow engine supports dynamic edge configuration, enabling it to dynamically create and execute multiple LLM audit call subtasks in parallel based on the current list of expense details and audit rules. This dynamic edge configuration allows the state machine workflow to flexibly and dynamically create workflow paths based on actual document data and rule lists, achieving a highly adaptive and scalable audit process that can easily handle complex business scenarios with different document structures and rule combinations.

[0022] Furthermore, during the process of calling the large language model service, streaming technology is used to return the intermediate execution status and some audit results to the requester in real time;

[0023] The structured audit results include audit rule codes, audit conclusions, result summaries, and detailed reasons for audit failures, where the detailed reasons reference the Chinese descriptions in the document form structure definition. Utilizing streaming technology, the audit process is visualized, providing users with real-time feedback and significantly improving the interactive experience. Simultaneously, the use of Chinese descriptions of failure reasons in the structured audit results makes the audit conclusions clear, easy to understand, and traceable, greatly improving audit efficiency and communication effectiveness.

[0024] A document intelligent verification system based on a large language model, used to execute the aforementioned document intelligent verification method based on a large language model, includes:

[0025] The request processing module is used to receive and process intelligent audit requests;

[0026] The data processing module is used to acquire the structured document data, document form structure definition, and audit rules;

[0027] The task orchestration module is used to generate audit tasks based on the data, structure definition, and rules.

[0028] The large language model service module, used to process the audit task, is configured to receive the audit rules, structured document data, and document form structure definition as input, and perform semantic parsing and compliance reasoning on the structured document data based on the document form structure definition; and

[0029] The response generation module is used to aggregate results and generate an audit response.

[0030] Furthermore, the task orchestration module is implemented by a state machine workflow engine;

[0031] The data processing module is configured to acquire data by calling multiple external data interfaces in parallel.

[0032] The system also includes a rate limiting module for limiting the frequency of calls to the large language model service module. By integrating a state machine workflow engine, parallel data interface, and rate limiting module, the system solidifies the high efficiency, stability, and adaptability at the method level into the system architecture, forming a high-performance, highly reliable complete solution.

[0033] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0034] The above technical solution has the following advantages or technical effects:

[0035] 1. This invention innovatively introduces the definition of document form structure as a semantic bridge, enabling large language models to accurately understand business data, realizing in-depth semantic auditing of complex rules and unstructured scenarios, and significantly improving the accuracy and intelligence level of auditing.

[0036] 2. This invention constructs a high-efficiency, high-throughput asynchronous processing pipeline by employing parallel data acquisition, parallel LLM task processing, and state machine workflow orchestration, which greatly improves the processing efficiency of large-scale document review.

[0037] 3. This invention provides users with a real-time, transparent, and easy-to-understand review process and conclusions through streaming response and structured, readable review result output, greatly optimizing user experience and operational efficiency.

[0038] 4. This invention ensures the maintainability, scalability, and operational stability under high pressure of the entire intelligent auditing system through modular system design and integrated fault tolerance mechanisms such as rate limiting. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention;

[0040] Figure 2This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0041] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example 1

[0043] like Figure 1 As shown, a document intelligent review method based on a large language model includes the following steps:

[0044] Receive intelligent review requests, which must include at least the identification information of the documents to be reviewed;

[0045] In response to the intelligent audit request, obtain the structured document data corresponding to the document's identification information, the document form structure definition, and at least one audit rule;

[0046] Based on the audit rules and structured document data, generate one or more audit tasks;

[0047] The large language model service is invoked to process the audit task. The audit rules, structured document data, and document form structure definition are provided to the large language model as input information. Based on the document form structure definition, semantic parsing and compliance reasoning are performed on the structured document data.

[0048] Receive structured audit results for each audit task returned by the large language model;

[0049] Collect structured review results, generate a review response, and return it.

[0050] After the reimbursement system starts, it provides web services through the FastAPI framework. The main program, main.py, is responsible for application initialization and core configuration; the pseudocode is as follows:

[0051] import FastAPI from fastapi

[0052] from app.core.configs.config import settings

[0053] from app.routers.v1.api import api_router

[0054] app = FastAPI(

[0055] title=settings.PROJECT_NAME,

[0056] description="AI Agent Service Platform",

[0057] version=settings.VERSION )

[0059] app.include_router(api_router, prefix=settings.API_V1_STR)

[0060] When the expense reimbursement system requires intelligent approval, the intelligent approval interface is called. The specific processing flow is as follows:

[0061] Step S1: Receive intelligent review request

[0062] The system receives POST requests through the / audit / interface. The request body contains parameters such as the enterprise code, expense report code, and a list of audit rule codes.

[0063] Step S2: Obtain structured data and rules

[0064] The system responds to the request by obtaining three key pieces of information through the data service layer: structured document data (reimbursement form master data and associated expense details), document form structure definition (field metadata), and audit rule details.

[0065] Step S3: Generate an audit task

[0066] Based on the acquired audit rules and structured document data, the system performs analysis and matching to dynamically generate audit tasks.

[0067] Step S4: Call LLM for semantic parsing and reasoning

[0068] The system takes audit rules, structured document data, and form structure definitions as input and provides them to the large language model (LLM). The LLM understands the business semantics of the structured data based on the form structure definitions and performs compliance reasoning.

[0069] Step S5: Receive and process the audit results

[0070] The system receives the structured audit results (JSON format) returned by LLM.

[0071] Step S6: Gather and return the final response

[0072] The results of all audit tasks are compiled into a complete report and returned to the reimbursement system.

[0073] Example 2

[0074] This embodiment is a further optimization of embodiment 1, specifically:

[0075] Structured document data includes the expense report master data and at least one associated expense detail item data;

[0076] The acquisition steps include: by calling multiple external data interfaces in parallel, obtaining detailed information on the expense reimbursement form structure definition, expense detail form structure definition, expense reimbursement master data, expense detail data, and audit rules.

[0077] The steps for generating one or more audit tasks based on audit rules include:

[0078] Match the scope of the audit rules with the type of each expense item;

[0079] Generate an independent audit task for each successfully matched cost item and audit rule combination;

[0080] Generate a task flag to skip review for each mismatched combination.

[0081] The steps for calling the large language model service to process the review task are executed in parallel.

[0082] Rate limiting is implemented to control the frequency of requests to the large language model service.

[0083] In step S2, the system improves efficiency by making parallel calls to multiple external data interfaces. This is specifically implemented in capi_client.py, and the pseudocode is as follows:

[0084] async def get_form_data(ent: str, form_type: str, code: str, lang:str):

[0085] url = f"{_base_url()} / AIX / aiAudit / formData"

[0086] headers = _hdr(ent, lang)

[0087] payload = {"formType": form_type, "formDataCode": code}

[0088] async with httpx.AsyncClient() as c:

[0089] return await c.post(url, headers=headers, json=payload)

[0090] The system retrieves the following in parallel: expense report form structure definition, expense detail form structure definition, expense report master data, expense detail data, and audit rule details.

[0091] In step S3, the system performs refined task filtering:

[0092] The scope of application of the audit rules is matched with the type of expense details. Independent audit tasks are generated for successfully matched combinations, and task flags for skipping audits are generated for unmatched combinations.

[0093] In step S4, the system employs a parallel execution strategy for multiple audit tasks and controls the request frequency through an integrated rate limiter. The pseudocode is as follows:

[0094] def __init__(self, base_url: str | None = None, api_key: str | None =None):

[0095] self.limiter = Limiter(

[0096] Rate(settings.QWEN_ACCESS_LIMIT_RATE, Duration.SECOND * 1),

[0097] max_delay=Duration.SECOND * settings.QWEN_ACCESS_LIMIT_DELAY, )

[0099] By implementing parallel processing and refined task management, the system throughput and resource utilization efficiency have been significantly improved.

[0100] Example 3

[0101] This embodiment is a further improvement on embodiment 1 or 2, specifically:

[0102] The steps of the method are executed through a state machine workflow engine, which defines multiple nodes, including at least: a data acquisition node, a rule processing node, an LLM review node, and a result aggregation node; the execution of nodes passes context data through state objects.

[0103] The state machine workflow engine supports dynamic edge configuration, which can dynamically create and execute multiple LLM audit call subtasks in parallel based on the list of expense details and audit rules in the current state.

[0104] During the process of calling the large language model service, streaming technology is used to return the intermediate execution status and some audit results to the requester in real time;

[0105] The structured audit results include audit rule codes, audit conclusions, result summaries, and detailed reasons for audit failures, with the detailed reasons referencing the Chinese descriptions in the document form structure definition.

[0106] The system uses the LangGraph framework to build a state machine workflow, defining the AuditContextState state object to pass the context. The workflow nodes include: a data acquisition node (parallel acquisition of expense report data, structure definition, and audit rules), a rule processing node (formatting audit rules), an expense breakdown and disbursement node, an LLM audit node, and a result aggregation node.

[0107] The workflow supports dynamic edge configuration, dynamically creating parallel subtasks based on the cost details list and the approval rule list. The pseudocode is as follows:

[0108] async def parallel_audit_per_rule(st: ExepenseState):

[0109] send_list: list[Send] = []

[0110] for rule in st.get("normalizedRules") or []:

[0111] if rule_match(rule, st["expense"]): # Rule matching check

[0112] send_list.append(Send("audit_one", {"rule": rule, "expense": st["expense"][0]}))

[0113] else:

[0114] send_list.append(Send("audit_skip", {"rule": rule, "expense": st["expense"][0]}))

[0115] return send_list

[0116] During the LLM call, the system uses streaming technology to return intermediate states and partial results in real time. The audit results include rule codes, audit conclusions, result summaries, and reasons for failure, with the reasons referencing the Chinese descriptions defined in the form structure.

[0117] This embodiment achieves flexible orchestration and efficient execution of the review process through a state machine workflow engine, providing a good user experience.

[0118] Example 4

[0119] like Figure 2 As shown, a document intelligent review system based on a large language model is used to perform any one of embodiments 1 to 3, including:

[0120] The request processing module is used to receive and process intelligent audit requests;

[0121] The data processing module is used to acquire structured document data, document form structure definitions, and audit rules;

[0122] The task orchestration module is used to generate audit tasks based on data, structure definitions, and rules;

[0123] The large language model service module, used for handling audit tasks, is configured to receive audit rules, structured document data, and document form structure definitions as input, and perform semantic parsing and compliance reasoning on the structured document data based on the document form structure definitions; and

[0124] The response generation module is used to aggregate results and generate an audit response.

[0125] The task orchestration module is implemented by a state machine workflow engine;

[0126] The data processing module is configured to acquire data by calling multiple external data interfaces in parallel;

[0127] The system also includes a rate limiting module, which limits the frequency of calls to the large language model service module.

[0128] The system configures core parameters through environment variables, as shown in the pseudocode below:

[0129] # config.py configuration example

[0130] class Settings(BaseSettings):

[0131] QWEN_BASE_URL: str

[0132] QWEN_API_KEY: str

[0133] QWEN_MODEL: str = "qwen-plus"

[0134] QWEN_ACCESS_LIMIT_RATE: int = 9 # Limit rate to 9 QPS

[0135] CAPI_BASE_URL: str

[0136] CAPI_TIMEOUT: int = 30

[0137] The rate limiting module uses the token bucket algorithm to ensure the stability of the LLM service.

[0138] Example 5

[0139] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any one of Embodiments 1 to 3.

[0140] The computer program stored on the storage medium includes the following functional modules: request receiving and parsing module, parallel data acquisition module, audit task generation module, LLM service calling module (including rate limiting function), audit result processing module, and streaming response generation module.

[0141] When the program is executed by the processor, it implements the method flow of the aforementioned embodiments to complete the intelligent review function of the document.

[0142] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A document intelligent review method based on a large language model, characterized by the following steps: include: Receive intelligent review requests, wherein the intelligent review requests include at least the identification information of the documents to be reviewed; In response to the intelligent audit request, obtain the structured document data, document form structure definition and at least one audit rule corresponding to the document's identification information; Based on the audit rules and the structured document data, generate one or more audit tasks; The large language model service is invoked to process the audit task, wherein the audit rules, structured document data, and document form structure definition are provided to the large language model as input information, and semantic parsing and compliance reasoning are performed on the structured document data based on the document form structure definition. Receive the structured audit results returned by the large language model for each audit task; The structured audit results are aggregated, an audit response is generated, and returned.

2. The document intelligent review method based on a large language model according to claim 1, characterized in that, The structured document data includes the expense report master data and at least one associated expense detail item data; The acquisition steps include: by calling multiple external data interfaces in parallel, acquiring the expense reimbursement form structure definition, expense detail form structure definition, expense reimbursement master data, expense detail data, and detailed information of the audit rules.

3. The document intelligent review method based on a large language model according to claim 2, characterized in that, The step of generating one or more audit tasks based on audit rules includes: Match the scope of the audit rules with the type of each expense item; Generate an independent audit task for each successfully matched cost item and audit rule combination; Generate a task flag to skip review for each mismatched combination.

4. The document intelligent review method based on a large language model according to claim 3, characterized in that, The steps of calling the large language model service to process the review task are executed in parallel. Rate limiting is applied to the frequency of requests calling the large language model service.

5. The document intelligent review method based on a large language model according to claim 1, characterized in that, The steps of the method are executed through a state machine workflow engine, which defines multiple nodes, including at least: a data acquisition node, a rule processing node, an LLM review node, and a result aggregation node; the execution of the nodes passes context data through state objects.

6. The document intelligent review method based on a large language model according to claim 5, characterized in that, The state machine workflow engine supports dynamic edge configuration, and can dynamically create and execute multiple LLM audit call subtasks in parallel based on the list of expense details and the list of audit rules in the current state.

7. The document intelligent review method based on a large language model according to claim 1, characterized in that, During the process of calling the large language model service, streaming technology is used to return the intermediate execution status and some audit results to the requester in real time; The structured audit results include audit rule codes, audit conclusions, result summaries, and detailed reasons for audit failures, wherein the detailed reasons refer to the Chinese descriptions in the document form structure definition.

8. A document intelligent review system based on a large language model, used to execute the document intelligent review method based on a large language model as described in any one of claims 1 to 7, characterized in that, include: The request processing module is used to receive and process intelligent audit requests; The data processing module is used to acquire the structured document data, document form structure definition, and audit rules; The task orchestration module is used to generate audit tasks based on the data, structure definition, and rules. The large language model service module is used to process the audit task. It is configured to receive the audit rules, structured document data and document form structure definition as input, and perform semantic parsing and compliance reasoning on the structured document data based on the document form structure definition. as well as The response generation module is used to aggregate results and generate an audit response.

9. A document intelligent review system based on a large language model according to claim 8, characterized in that, The task orchestration module is implemented by a state machine workflow engine; The data processing module is configured to acquire data by calling multiple external data interfaces in parallel. The system also includes a rate limiting module for limiting the frequency of call requests to the large language model service module.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images