Document auditing method and system and electronic equipment

Abstract

The invention discloses a document auditing method, a document auditing system and electronic equipment. Relates to the field of artificial intelligence. The method comprises the steps that the document type of a to-be-audited document is recognized through a type recognition agent, and a target document type is obtained; based on the target document type, checking rules associated with the target document type are queried from the target knowledge graph through a query agent, and the query agent is a pre-constructed agent used for querying the checking rules in the knowledge graph; based on the auditing rule, the to-be-audited document is audited through multiple auditing agents, a target auditing result is obtained, and the multiple auditing agents are used for auditing the document content and format. The technical problem that the false alarm rate is high when a large model based on deep learning is used for auditing official documents in related technologies is solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a document review method, system, and electronic device. Background Technology

[0002] With the development of information technology, document review work faces unprecedented challenges, including but not limited to a surge in the number of documents, increased complexity, and ever-increasing demands for review efficiency and accuracy. Document review in related technologies primarily relies on manual labor, which is not only time-consuming and labor-intensive but also prone to human error.

[0003] To address these challenges, rule-based automated systems, machine learning algorithms, and even large-scale modeling techniques have been introduced into document review processes in recent years. For example, intelligent parsing and fine-tuning of large-scale models can improve review efficiency and accuracy; large-scale model-based text review methods aim to directly improve document review efficiency through deep learning technology. However, rule-based systems struggle to adapt to complex document structures and unpredictable real-world application scenarios; while large-scale models based on deep learning face challenges in controlling false positive rates (i.e., higher false positive rates).

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a document review method, system, and electronic device to at least solve the technical problem of high false alarm rate in the review of official documents using large deep learning-based models.

[0006] According to one aspect of the present invention, a document review method is provided, comprising: identifying the document type of a document to be reviewed by a type recognition agent to obtain a target document type; based on the target document type, querying a review rule associated with the target document type from a target knowledge graph by a query agent, wherein the query agent is a pre-built agent for querying review rules in the knowledge graph; and reviewing the document to be reviewed by multiple review agents based on the review rules to obtain a target review result, wherein the multiple review agents are used to review the document content and format.

[0007] Further, the target knowledge graph is obtained through the following methods: acquiring a historical document set and a document template set, wherein the historical document set includes various types of official documents that have been approved in the historical process, and the document template set includes writing templates for each type of official document; extracting information from the historical document set and the document template set to obtain extraction results, wherein the extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document; and constructing a knowledge graph based on the extraction results to obtain the target knowledge graph.

[0008] Further, based on the extraction results, a knowledge graph is constructed to obtain the target knowledge graph, including: constructing multiple triples based on the extraction results; performing deduplication processing on the multiple triples to obtain multiple deduplicated triples; constructing a knowledge graph based on the multiple deduplicated triples to obtain an initial knowledge graph; constructing multiple target query statements; verifying whether the auditing rules queried by the target query statements can be found in the initial knowledge graph based on the multiple target query statements, and obtaining a verification result; if the verification result indicates that the auditing rules queried by the target query statements can be found in the initial knowledge graph, the initial knowledge graph is determined as the target knowledge graph.

[0009] Further, identifying the document type of the document to be reviewed by the type recognition agent to obtain the target document type includes: extracting document features of the document to be reviewed by the type recognition agent, wherein the document features include at least one of the following: document title, document structure; obtaining a standard for document type to obtain a type standard; and determining the target document type based on the type standard and the document features.

[0010] Furthermore, based on the target document type, the query agent retrieves the review rules associated with the target document type from the target knowledge graph, including: generating a data query statement based on the target document type through the query agent; and executing the data query statement to obtain the review rules associated with the target document type.

[0011] Furthermore, the plurality of review agents include: a content review agent for reviewing document content and a format review agent for reviewing document format. The review rules include: content review rules and format review rules. Based on the review rules, the document to be reviewed is reviewed by the plurality of review agents to obtain a target review result, including: reviewing the content of the document to be reviewed by the content review agent based on the content review rules to obtain a first review result; reviewing the format of the document to be reviewed by the format review agent based on the format review rules to obtain a second review result; and determining the target review result based on the first review result and the second review result.

[0012] Furthermore, after reviewing the document to be reviewed by multiple review agents based on the review rules to obtain the target review result, the method further includes: generating a review report based on the target review result, wherein the review report includes: the problems existing in the document to be reviewed and the location of the problems.

[0013] According to another aspect of the present invention, a document review system is also provided, comprising: an input layer for receiving the document to be reviewed; an intelligent agent layer, wherein the intelligent agent layer includes: a type identification intelligent agent, a query intelligent agent, and multiple review intelligent agents, for identifying the document type of the document to be reviewed by the type identification intelligent agent to obtain a target document type, querying review rules associated with the target document type from a target knowledge graph by the query intelligent agent based on the target document type, and reviewing the document to be reviewed by the multiple review intelligent agents based on the review rules to obtain a target review result, wherein the type identification intelligent agent is used to identify the document type, the query intelligent agent is used to query the target knowledge graph for the required data, and the multiple review intelligent agents are used to review the document content and format; and an output layer for outputting the review result.

[0014] Furthermore, the document review system also includes: a standard construction layer, used to obtain a historical document set and a document template set, extract information from the content in the historical document set and the document template set to obtain extraction results, and construct a knowledge graph based on the extraction results to obtain the target knowledge graph. The historical document set includes: various types of official documents that have been reviewed and approved in historical time. The document template set includes: writing templates for each type of official document. The extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document.

[0015] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the document review method of any one of the above-mentioned methods by executing the executable instructions.

[0016] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the document review method described above when it is running.

[0017] In this invention, the following approach is adopted: a type recognition agent identifies the document type of the document to be reviewed, obtaining the target document type; based on the target document type, a query agent queries the review rules associated with the target document type from the target knowledge graph, wherein the query agent is a pre-built agent used to query review rules in the knowledge graph; based on the review rules, multiple review agents review the document to be reviewed, obtaining the target review result, wherein the multiple review agents are used to review the document content and format. This solves the technical problem of high false positive rates in related technologies where large deep learning models are used to review official documents. In this invention, multiple agents and a knowledge graph containing review rules are used to review the document to be reviewed, avoiding the high false positive rate in related technologies where large deep learning models are directly used to review official documents, thus improving the accuracy of document review. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a flowchart of an optional document review method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of an optional document review system according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of an optional document review device according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] For ease of description, the following explanations are provided for some terms or nouns involved in the various embodiments of the present invention:

[0026] A knowledge graph is a data structure used to represent entities and their relationships, and is commonly used in information retrieval and intelligent question-answering systems.

[0027] Cypher queries are the query language used by the Neo4j graph database for retrieving and updating graph data.

[0028] Large Language Models (LLMs) are language processing models with a large number of parameters, such as Qwen2.5:32B, used for tasks such as text generation and classification.

[0029] Intent recognition is the process of determining a user's intent by analyzing user input or document content.

[0030] Document type categorizes documents based on their content characteristics, such as meeting minutes, official notices, etc.

[0031] Review standards are a set of format and content requirements for specific types of documents.

[0032] An agent, or intelligent agent, is a proxy that can perceive its environment and take actions to achieve specific goals. Intelligent agents are widely used in the field of artificial intelligence, and their core strength lies in their ability to learn autonomously and continuously evolve to better complete tasks and adapt to complex environments.

[0033] A Document Review Tool for Official Documents Based on KnowledgeGraph and Agent (DRT-KGA) is a document review tool for official documents based on knowledge graphs and multi-agent collaboration.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), the collected information and data (including but not limited to data used for analysis, stored data, displayed data, document data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse.

[0035] Example 1

[0036] According to an embodiment of the present invention, an optional document review method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] Figure 1 This is a flowchart of an optional document review method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0038] Step S101: The document type of the document to be reviewed is identified by the type recognition agent to obtain the target document type.

[0039] The aforementioned documents to be reviewed can be official documents. The aforementioned type recognition agent can parse the document format and content of the documents to be reviewed and identify the document type (such as notices, requests, etc.). For example, by analyzing the features of the document title, body structure, etc., and combining them with the official document type standards in the knowledge graph, the specific type of the document can be determined.

[0040] Step S102: Based on the target document type, query the review rules associated with the target document type from the target knowledge graph through a query agent. The query agent is a pre-built agent used to query review rules in the knowledge graph.

[0041] The target knowledge graph can be used as a database to store document standards (e.g., document review standards or review rules) and document types (i.e., document types). Review standards and document types are organized in the graph database as triples, where nodes represent entities (e.g., document types, review rules) and edges represent relationships between entities (e.g., "contains" or "applies to"). The structure of the knowledge graph allows query agents to perform efficient queries and reasoning to find review rules that match specific document types. It should be noted that the graph database can be Neo4j, or other graph databases. In one optional example, a more flexible data structure, such as JSON or XML files, can be used to store and manage review standards, simplifying the maintenance of these standards.

[0042] The review rules associated with the target document type can include: review standards and requirements for the target document type, including document formatting specifications (such as the font, size, and position of the title, body text, and signature), content requirements (such as required keywords and syntactic structures), and any domain-specific compliance regulations. These review rules constitute the basis and standards for the review process by the review agent.

[0043] In this embodiment, the query agent can use a graphical query language to filter out the review rules related to the target document type from the target knowledge graph. For example, it can first find the document type node, and then find the corresponding format and content requirement nodes along the predefined relationship path, and finally obtain the complete set of review rules.

[0044] Step S103: Based on the review rules, multiple review agents review the document to be reviewed to obtain the target review result. The multiple review agents are used to review the document content and format.

[0045] To improve the efficiency and accuracy of the review process, this embodiment employs multiple review agents working collaboratively. Each agent reviews different aspects of the document; for example, the agents may include a content review agent and a format review agent.

[0046] The content moderation agent checks the completeness, logical consistency, and language accuracy of the document to be reviewed, including the presence of sensitive words. It records all identified issues and their locations. The agent understands the meaning of the text and identifies and marks content that violates the review rules. The formatting agent reviews the appearance and structure of the document, such as the format of titles, paragraph alignment, signatures, the presence of headers and footers, the presence of attachments, and the format of attachments (e.g., whether the font, size, and color of attachments conform to regulations). Based on pre-defined formatting rules, the formatting agent detects any formatting errors in the document.

[0047] In an alternative example, rule-based methods or other machine learning algorithms can be used to replace the vetting agent.

[0048] After the review process is completed, the review results of all review agents can be summarized to generate a comprehensive review report. This report can not only point out problems in the document (such as errors, omissions, non-standard formatting, etc.), but also provide modification suggestions to help users quickly understand and correct the document.

[0049] In this embodiment, each type of intelligent agent (e.g., type recognition intelligent agent, query intelligent agent, review intelligent agent, etc.) can be set with a corresponding prompt word template. By adding the information of the document to be reviewed and prior knowledge to the prompt word template, the corresponding prompt words are obtained. By inputting the prompt words into the corresponding intelligent agent, the intelligent agent can complete the corresponding task to perform efficient and accurate review of the document to be reviewed.

[0050] Through the above steps, this embodiment utilizes multiple intelligent agents and a knowledge graph containing review rules to review the documents to be reviewed. This avoids the high false positive rate that occurs in related technologies where large deep learning models are directly used to review official documents, thus improving the accuracy of document review. Furthermore, it solves the technical problem of high false positive rates in related technologies where large deep learning models are used to review official documents.

[0051] Optionally, in the document review method provided in this embodiment of the invention, the target knowledge graph is obtained in the following way: obtaining a historical document set and a document template set, wherein the historical document set includes: various types of official documents that have been reviewed and approved in the historical process, and the document template set includes: writing templates for each type of official document; extracting information from the historical document set and the document template set to obtain extraction results, wherein the extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document; and constructing a knowledge graph based on the extraction results to obtain the target knowledge graph.

[0052] In this embodiment, all approved official documents can be collected from the historical document library to form a "historical document collection". The historical document collection may contain instances of approved official documents for learning and identifying correct formats and content. Simultaneously, a "document template collection" can also be obtained, which contains standard drafting templates for each type of official document (such as requests, notices, and reports).

[0053] In this embodiment, Natural Language Processing (NLP) techniques and machine learning models, such as large models, can be used to extract information from historical document sets and document template sets. The goal of information extraction can be to identify and extract the format requirements (such as the font size of the title and the paragraph spacing of the body text), content elements (such as required paragraphs and keywords), and filling rules (such as the way to fill in a specific form and the construction rules of a specific sentence) for each type of official document. By training the model to identify and understand the key information in the documents, this information can then be converted into structured data.

[0054] After obtaining the information extraction results, a knowledge graph can be constructed. This knowledge graph is a structured data model used to represent entities (such as document types and format requirements) and the relationships between them. In a knowledge graph, entities can be represented as nodes, and relationships between entities can be connected by edges. For example, the node "notice" can establish an "inclusion" relationship with the node "title format," indicating that the document type "notice" includes specific title format requirements. The process of constructing a knowledge graph may involve storing the information extraction results in the form of triples (subject, relation, object), such as (notice, inclusion, title is bold, size 2). By converting complex information into a graph structure, the review standards and template information of historical documents can be stored and retrieved in a more intuitive and efficient way, facilitating subsequent document review.

[0055] In this embodiment, a target knowledge graph is constructed to optimize the efficiency and accuracy of document review. The powerful natural language processing capabilities of large-scale models and the structural advantages of graph databases are leveraged to provide a solid foundation for subsequent document review work.

[0056] Optionally, in the document review method provided in this embodiment of the invention, a knowledge graph is constructed based on the extraction results to obtain a target knowledge graph, including: constructing multiple triples based on the extraction results; performing deduplication on the multiple triples to obtain multiple deduplicated triples; constructing a knowledge graph based on the multiple deduplicated triples to obtain an initial knowledge graph; constructing multiple target query statements; verifying whether the review rules queried by the target query statements can be found in the initial knowledge graph based on the multiple target query statements to obtain a verification result; if the verification result indicates that the review rules queried by the target query statements can be found in the initial knowledge graph, the initial knowledge graph is determined as the target knowledge graph.

[0057] In this embodiment, for information such as format requirements, content elements, and filling rules for official documents extracted from the historical document set and document template set, this information can first be converted into structured data that the knowledge graph can understand. For example, a "triple" can be constructed, which can consist of three elements: subject, relation, and object, such as "notice → contain → title format". In this way, each piece of extracted information can be converted into one or more triples, which facilitates subsequent storage and retrieval.

[0058] Next, these triples can be deduplicated to eliminate redundant information. During information extraction, the same rules may be extracted multiple times, leading to redundancy in the triple set. Deduplication ensures the uniqueness of information in the knowledge graph, avoiding unnecessary resource consumption and query confusion, and improving the efficiency and accuracy of the graph.

[0059] The deduplicated set of triples can then be used to construct an "initial knowledge graph". For example, triples are imported into a graph database to represent different entities (such as "notifications") and the relationships between them (such as "containment") in the form of nodes and edges.

[0060] To verify the correctness and completeness of the knowledge graph, multiple target queries can be constructed. These queries can retrieve specific review rules, such as the "title format" requirement for "notice" type documents. The target queries can be designed based on the graph structure and written using a graphical query language. The "initial knowledge graph" can then be tested using these target queries. If the knowledge graph returns review rules that match the queries, the information in the knowledge graph can be confirmed to be correct and complete. This ensures that the knowledge graph meets the needs of document review and prevents review failures due to missing or incorrect data.

[0061] Finally, if the verification results show that the "initial knowledge graph" can successfully retrieve all the review rules, then the initial knowledge graph can be confirmed as the "target knowledge graph".

[0062] Rules extracted from historical data and templates are accurately and uniquely converted into knowledge graph data. The validity of the knowledge graph is verified through rigorous validation, making it an indispensable part of the document review process and helping agents to conduct efficient and accurate document review.

[0063] For example, for different types of official documents, based on large language models (e.g., Qwen2.5:32B large model) and agent technology, the standards and filling opinions of official documents are extracted from historical document databases and official document templates to form knowledge graph triples. Using a thinking chain approach, the large model reflects and summarizes to ensure the correct extraction of official document standards.

[0064] The document standards and filling opinions extracted from the large language model can be stored in a graph database in the form of triples. The structure of a triple can be: (subject, relation, object), for example (notification, inclusion, title format). Triples constitute a structured knowledge graph, facilitating subsequent querying and reasoning.

[0065] In the process of building a knowledge graph, we can first define node types such as document type labels, regular content of documents, and filling suggestions, and establish corresponding attributes for them, such as the name of the document type, the specific content of the document, and the format. Then, we connect these nodes through relationships, such as "document type → contains → approval items".

[0066] To ensure the quality of the knowledge graph, large models can be used for multiple rounds of reflection and validation to ensure the accuracy of the extracted criteria. For example, large models can be used to generate multiple queries to verify whether the criteria in the knowledge graph can correctly match the expected review rules.

[0067] Optionally, in the document review method provided in this embodiment of the invention, the document type of the document to be reviewed is identified by a type recognition agent to obtain the target document type, which includes: extracting document features of the document to be reviewed by the type recognition agent, wherein the document features include at least one of the following: document title, document structure; obtaining the standard for document type to obtain the type standard; and determining the target document type based on the type standard and the document features.

[0068] In one optional example, the document title may contain key information about the document's nature. In this embodiment, the type recognition agent can preliminarily determine the document's category by analyzing keywords or phrases in the title. For example, a document with a title containing words such as "notice," "request," or "meeting minutes" can be identified as a corresponding official document.

[0069] Official documents can follow specific structural patterns. For example, notices may include fixed parts such as a title, body, and signing date, while requests may additionally include the recipient and reason for the request. Type recognition agents can deeply analyze the logical structure and physical layout of documents to identify these structural features, thereby assisting in document type identification.

[0070] Document type standards are a set of predefined rules or templates used to describe the unique attributes and formatting requirements of various document types. These standards can be extracted from a company's document management policies, industry standards, or historical document collections to form a structured knowledge system that agents can understand and apply. For example, standards for the "notice" type might include the font size of the title, the layout of the body text, and the placement requirements for the signature and date.

[0071] In this embodiment, the type recognition agent can use machine learning techniques or rule-based algorithms to compare the features extracted from the document with known document type standards to find the most matching document type. Finally, the agent will output the "target document type".

[0072] For example, a type recognition agent can use a large model to analyze documents uploaded to the front end for review and call a Python script to parse the document format and content, identifying the document type (such as a notice or request). By analyzing features such as the document's title and body structure, combined with document type standards from a knowledge graph within the prompt words, the specific document type can be determined.

[0073] Optionally, in the document review method provided in this embodiment of the invention, based on the target document type, a query agent is used to query the review rules associated with the target document type from the target knowledge graph, including: generating a data query statement based on the target document type through the query agent; and executing the data query statement to obtain the review rules associated with the target document type.

[0074] In this embodiment, after obtaining the document type of the document to be reviewed, the query agent can construct a data query statement using a specific query language (e.g., a graph query language). This data query statement can retrieve all review rules associated with the target document type from the knowledge graph. The query agent can ensure the comprehensiveness and accuracy of the query from multiple dimensions, such as title rules, content format, and terminology standards.

[0075] After executing the query, the graph database can return a result set containing all the auditing rules related to the target document type. Auditing rules can include details such as: what style the title should be, what parts the body should include, and what wording guidelines should be followed.

[0076] By querying intelligent agents, review rules can be dynamically retrieved from the target knowledge graph, laying the foundation for subsequent content and format review. Leveraging the knowledge graph's ability to store and retrieve complex relationships, and through a graphical query language, the query intelligent agent can extract the necessary information from the vast knowledge graph, thereby improving the efficiency of reviewing official documents.

[0077] Optionally, in the document review method provided in this embodiment of the invention, the multiple review agents include: a content review agent for reviewing document content and a format review agent for reviewing document format. The review rules include: content review rules and format review rules. Based on the review rules, the document to be reviewed is reviewed by the multiple review agents to obtain a target review result, including: reviewing the content of the document to be reviewed by the content review agent based on the content review rules to obtain a first review result; reviewing the format of the document to be reviewed by the format review agent based on the format review rules to obtain a second review result; and determining the target review result based on the first review result and the second review result.

[0078] Content moderation rules are rules used to review the content of documents, including logical coherence, word choice, sensitive word checks, and grammatical and spelling errors. These rules can be obtained from knowledge graphs.

[0079] A content reviewing agent can check whether a document's content complies with content review rules. The agent can scan the document content and identify problems such as illogical sentences, the use of prohibited sensitive words, and duplicate or missing information.

[0080] After the content review is completed, the content review agent can obtain the first review result, which may include all the problems found in the document content.

[0081] The format review rules can review the format of the document to be reviewed, such as the format of the title and signature, font size and color, paragraph spacing, and the use of tables and lists.

[0082] The Format Reviewing Agent can check the format of a document according to format review rules. For example, it can check whether the title uses the required font and size, whether the signature is in the correct position, and whether the document uses the necessary tables and lists in accordance with standards.

[0083] After completing the review, the format review agent can obtain a second review result, which may include all non-compliance in the document format, such as titles that do not meet the standards, paragraph formatting errors, and attachments that are not added in the required format.

[0084] After reviewing the content and format, the "first review result" and "second review result" can be combined to form a comprehensive "target review result." This result will include all review issues related to the document's content and format, as well as suggested modifications for each issue.

[0085] By setting up multiple review agents to review document content and format, the comprehensiveness and professionalism of the review process are ensured. Through the collaborative work of content review agents and format review agents, detailed review reports can be generated to guide users in document correction, thereby improving overall work efficiency and document standardization.

[0086] Optionally, in the document review method provided in this embodiment of the invention, after reviewing the document to be reviewed through multiple review agents based on review rules and obtaining the target review result, the method further includes: generating a review report based on the target review result, wherein the review report includes: the problems existing in the document to be reviewed and the location of the problems.

[0087] In this embodiment, the process of converting the target audit results into a user-readable format is described. To make the audit report more practical, this embodiment can precisely mark the specific location of each issue in the document, which may include line numbers, paragraph numbers, or specific heading levels. For example, if the wording in a paragraph in the document is inappropriate, the audit report will clearly indicate which sentence in which paragraph has the problem; if there is a formatting error, such as incorrect column width settings in a table, it can also indicate which table in the document needs to be adjusted.

[0088] In this embodiment, a large model is used to extract review standards and comments, and a thought-chain approach is employed for reflection and summarization, ensuring the accuracy and comprehensiveness of document standards. This intelligent review method can more accurately understand and process complex document content.

[0089] Storing the extracted standard information in the form of triples in a knowledge graph database not only helps maintain data consistency and integrity but also allows for rapid retrieval and association analysis. Compared to rule-based or template-based methods in related technologies, this approach is more advantageous when handling complex relationships.

[0090] From document type identification to the generation of the final audit report, the entire process is highly automated, reducing the need for human intervention.

[0091] In this embodiment, multiple agents are employed, each responsible for specific tasks such as document type identification, query generation, content review, and format review. This makes document review more flexible and scalable, and new functions can be added or existing functions improved as needed.

[0092] In addition to providing the overall audit results, this embodiment also provides specific error descriptions, suggested corrections, and the exact location of the problems. This helps users quickly understand and fix issues in the document, improving work efficiency and quality.

[0093] The document review method provided in this embodiment can be used to parse internal company data, enhancing the practicality and adaptability of internal data and providing privacy protection.

[0094] Example 2

[0095] Embodiment 2 of the present invention provides an optional document review system, which can be used to execute the document review method provided in Embodiment 1 of the present invention.

[0096] Figure 2 This is a schematic diagram of an optional document review system according to an embodiment of the present invention, such as... Figure 2 As shown, it includes: input layer, agent layer, and output layer.

[0097] The document is divided into several layers: an input layer for receiving documents to be reviewed; an agent layer comprising a type identification agent, a query agent, and multiple review agents; an input layer for identifying the document type of the document to be reviewed; a query agent for querying the review rules associated with the target document type from the target knowledge graph; and an output layer for outputting the review results.

[0098] The document review system can achieve efficient and accurate review of documents (official documents) by combining knowledge graphs and multi-agent (intelligent agent) systems. The system can extract document standards based on historical data and company templates, and store these standards in a knowledge graph database (e.g., a graph database); then, it uses a multi-agent system for automated review, including document type identification, query generation, and content and format review; finally, it generates a detailed review report and recommendations. The following is a detailed explanation of each module.

[0099] 1. Input layer:

[0100] It can accept pending official documents, document templates, and historical document libraries as input. These documents can be of various types, such as notices, requests for instructions, and reports. Among them, document templates and historical document libraries can be used for standard construction (e.g., building a target knowledge graph) and provide basic data for the review of documents pending approval.

[0101] 2. Agent verification layer:

[0102] (1) Document Type Recognition Agent (corresponding to the type recognition intelligent agent): This agent can determine the type of the input document, such as a notice or request. By analyzing the document's title, body structure, and other features, combined with existing official document type standards, the specific type of the document can be determined. This lays the foundation for subsequent document review.

[0103] (2) Graph query to generate Agent (corresponding to query agent): Based on the document type identification result, the Agent can generate the corresponding Cypher query statement, which is used to retrieve relevant review standards from the target knowledge graph. Cypher is a graph query language that can efficiently perform complex queries in graph databases to ensure that accurate review rules are obtained.

[0104] (3) Content Review Agent: This agent can review the main body of the document based on the content review standards obtained from the knowledge graph. It checks whether the document content is logical, whether the language is standardized, whether there are sensitive words, whether there are typos, etc., and records all the problems found.

[0105] (4) Format Review Agent: Similar to the content review agent, this agent can be used to review the format of documents. It can check whether the title, signature, attachments, and other parts of the document comply with the regulations according to the format review standards, ensuring that the overall format of the document is correct.

[0106] 3. Output layer:

[0107] By combining the results of multiple audit agents, a detailed audit report can be generated. This report can include the overall audit status of the document, as well as specific problems and their locations, making it easy for users to quickly locate and modify them.

[0108] Through the collaborative work of the above modules, DRT-KGA (corresponding to the document review system) can achieve comprehensive and intelligent review of official documents, effectively improving the quality and compliance of documents.

[0109] Optionally, the document review system also includes: a standard construction layer, used to obtain a historical document set and a document template set, extract information from the content in the historical document set and the document template set, obtain extraction results, and construct a knowledge graph based on the extraction results to obtain a target knowledge graph. The historical document set includes: various types of official documents that have been reviewed and approved in the past, the document template set includes: writing templates for each type of official document, and the extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document.

[0110] like Figure 3 As shown, the document review system also includes a standard building layer.

[0111] The standards construction layer can use large model techniques to extract standards and filling instructions for official documents from the input historical document library and document templates. The large model can understand the text content and extract key information such as format requirements, content elements, and filling specifications. This information is then structured to form standardized knowledge graph data. The extracted document standard data can be stored in a knowledge graph database (e.g., a graph database), which can store complex relational data in the form of nodes and edges, facilitating subsequent queries and reasoning. Document standard data is stored as nodes, and the relationships between different standards are represented by edges.

[0112] In one optional example, the implementation steps of the document review system are as follows:

[0113] 1. Input layer:

[0114] (1) First, design a Python script, use the python-docx library to parse the docx file, and design a series of functions to extract the content and format information of paragraphs and tables.

[0115] (2) For official documents to be reviewed, such as notices and requests, the designated tool will provide the function of loading and uploading. The uploaded documents will be parsed by the above-mentioned Python script.

[0116] (3) The official document templates and historical document library are processed offline. The above-mentioned Python script is used in the background of the tool to parse them and extract the format and content of the text.

[0117] 2. Standard Build Layer:

[0118] For different types of official documents, standards and filling suggestions can be extracted from historical document databases and document templates based on large language models and agent technology to form knowledge graph triples. Using a thinking chain approach, the large model can reflect and summarize to ensure the correct extraction of document standards.

[0119] The document standards and feedback extracted by the model are stored in a graph database in the form of triples. The structure of a triple is typically (subject, relation, object), such as (notification, inclusion, title format). These triples constitute a structured knowledge graph, facilitating subsequent queries and reasoning.

[0120] In the process of building a knowledge graph, we can first define node types such as document type labels, regular content of documents, and filling suggestions, and establish corresponding attributes for them, such as the name of the document type, the specific content of the document, and the format. Then, we connect these nodes through relationships, such as "document type → contains → review items".

[0121] To ensure the quality of the knowledge graph, large language models can be used for multiple rounds of reflection and verification to ensure the accuracy of the extracted review criteria. For example, multiple triples can be generated using the large model, and unreasonable or duplicate triples can be automatically removed. Multiple query statements can also be generated using the large model to verify whether the criteria in the knowledge graph correctly match the expected review rules.

[0122] 3. Agent verification layer:

[0123] The Agent review layer consists of multiple Agents, each responsible for different review tasks, including document type identification, Cypher query generation, content review, and format review. Each Agent also carries a thought chain to prevent reasoning errors based on reflection.

[0124] Document Type Recognition Agent: This agent uses a large model to analyze documents uploaded by specified front-end tools and calls Python scripts to parse document format and content, identifying document types (such as notices, requests, etc.). It determines the specific document type by analyzing features such as the document title and body structure, combined with document type standards from the knowledge graph in the prompt words.

[0125] Cypher query to generate Agent: Based on the document type identification results and known document types, this Agent generates corresponding Cypher query statements to retrieve relevant audit standards from the Neo4j knowledge graph.

[0126] Content moderation agent: This agent reviews the main body of the document based on content moderation standards obtained from the knowledge graph. It checks whether the document content is complete, logical, uses standard language, contains sensitive words, etc., and records all issues found.

[0127] Format Review Agent: This agent focuses on document format review. Based on format review standards, it checks whether the font, size, color, and other formatting of the document's title, signature, attachments, etc., comply with regulations, ensuring that the overall document format is correct.

[0128] 4. Output layer:

[0129] After the review is completed, a detailed review report is generated by combining the results of the multi-agent reviews. The report not only includes the overall review status of the document, but also specifically points out the problems and their locations, making it easy for users to quickly locate and modify them.

[0130] In this embodiment, a large model is used to extract review standards and comments, and a thought-chain approach is employed for reflection and summarization, ensuring the accuracy and comprehensiveness of document standards. This intelligent review method can more accurately understand and process complex document content.

[0131] Storing the extracted standard information in the form of triples in a knowledge graph database not only helps maintain data consistency and integrity but also allows for rapid retrieval and association analysis. Compared to rule-based or template-based methods in related technologies, this approach is more advantageous when handling complex relationships.

[0132] From document type identification to the generation of the final audit report, the entire process is highly automated, reducing the need for human intervention.

[0133] In this embodiment, multiple agents are employed, each responsible for specific tasks such as document type identification, query generation, content review, and format review. This makes document review more flexible and scalable, and new functions can be added or existing functions improved as needed.

[0134] In addition to providing the overall audit results, this embodiment also provides specific error descriptions, suggested corrections, and the exact location of the problems. This helps users quickly understand and fix issues in the document, improving work efficiency and quality.

[0135] The document review system provided in this embodiment can be used to parse internal company data, enhancing the usability and adaptability of internal data and providing privacy protection.

[0136] Example 3

[0137] Embodiment 3 of the present invention provides an optional document review device, wherein each implementation unit in the document review device corresponds to each implementation step in Embodiment 1.

[0138] Figure 3This is a schematic diagram of an optional document review device according to an embodiment of the present invention, such as... Figure 3 As shown, it includes: identification unit 31, query unit 32 and review unit 33.

[0139] Among them, the identification unit 31 is used to identify the document type of the document to be reviewed through the type identification agent to obtain the target document type;

[0140] The query unit 32 is used to query the review rules associated with the target document type from the target knowledge graph through a query agent based on the target document type. The query agent is a pre-built agent used to query review rules in the knowledge graph.

[0141] The review unit 33 is used to review the document to be reviewed by multiple review agents based on the review rules, and obtain the target review result. The multiple review agents are used to review the content and format of the document.

[0142] In the document review device provided in this embodiment of the invention, the identification unit 31 identifies the document type of the document to be reviewed through a type identification agent to obtain the target document type. The query unit 32, based on the target document type, queries the review rules associated with the target document type from the target knowledge graph through a query agent. The query agent is a pre-built agent used to query review rules in the knowledge graph. The review unit 33, based on the review rules, reviews the document to be reviewed through multiple review agents to obtain the target review result. The multiple review agents are used to review the document content and format. This solves the technical problem of high false positive rates in related technologies where large deep learning models are used to review official documents. In this embodiment, multiple agents and a knowledge graph containing review rules are used to review the document to be reviewed, avoiding the high false positive rate in related technologies where large deep learning models are directly used to review official documents, thus improving the accuracy of document review.

[0143] Optionally, in the document review device provided in this embodiment of the invention, the target knowledge graph is obtained through the following sub-units: a first acquisition sub-unit, used to acquire a historical document set and a document template set, wherein the historical document set includes: various types of official documents that have been reviewed and approved in the historical process, and the document template set includes: writing templates for each type of official document; a first extraction sub-unit, used to extract information from the historical document set and the document template set to obtain extraction results, wherein the extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document; and a construction sub-unit, used to construct a knowledge graph based on the extraction results to obtain the target knowledge graph.

[0144] Optionally, in the document review device provided in this embodiment of the invention, the construction subunit includes: a construction module, used to construct multiple triples based on the extraction results; a deduplication module, used to perform deduplication processing on the multiple triples to obtain multiple deduplicated triples; a first processing module, used to construct a knowledge graph based on the multiple deduplicated triples to obtain an initial knowledge graph; a second processing module, used to construct multiple target query statements, and based on the multiple target query statements, verify whether the review rules queried by the target query statements can be found in the initial knowledge graph to obtain a verification result; and a determination module, used to determine the initial knowledge graph as the target knowledge graph if the verification result indicates that the review rules queried by the target query statements can be found in the initial knowledge graph.

[0145] Optionally, in the document review device provided in this embodiment of the invention, the identification unit includes: a second extraction subunit, used to extract document features of the document to be reviewed through a type recognition agent, wherein the document features include at least one of the following: document title, document structure; a second acquisition subunit, used to acquire a standard for the document type and obtain a type standard; and a first determination subunit, used to determine the target document type based on the type standard and the document features.

[0146] Optionally, in the document review device provided in this embodiment of the invention, the query unit includes: a generation subunit, used to generate a data query statement based on the target document type through a query agent; and an execution subunit, used to execute the data query statement to obtain the review rules associated with the target document type.

[0147] Optionally, in the document review device provided in this embodiment of the invention, the multiple review agents include: a content review agent for reviewing document content and a format review agent for reviewing document format. The review rules include: content review rules and format review rules. The review unit includes: a first review subunit, used to review the content of the document to be reviewed by the content review agent based on the content review rules, and obtain a first review result; a second review subunit, used to review the format of the document to be reviewed by the format review agent based on the format review rules, and obtain a second review result; and a second determination subunit, used to determine a target review result based on the first review result and the second review result.

[0148] Optionally, in the document review device provided in the embodiments of the present invention, the document review device further includes: a generation unit, used to generate a review report based on the target review result after reviewing the document to be reviewed by multiple review agents according to the review rules and obtaining the target review result, wherein the review report includes: the problems existing in the document to be reviewed and the location of the problems.

[0149] The document review device described above may also include a processor and a memory. The identification unit 31, query unit 32 and review unit 33 are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0150] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, multiple agents and a knowledge graph containing review rules can be used to review documents. This avoids the high false positive rate associated with directly reviewing official documents using large deep learning models, thus improving the accuracy of document review.

[0151] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0152] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the document review method of any one of the above-mentioned methods by executing the executable instructions.

[0153] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the document review method described above when it is running.

[0154] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention, such as... Figure 4 As shown, an embodiment of the present invention provides an electronic device 40, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-mentioned document review methods.

[0155] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0156] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0161] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A document review method, characterized in that, include: The document type is identified by a type recognition agent to determine the document type of the document to be reviewed, thus obtaining the target document type. Based on the target document type, a query agent is used to query the review rules associated with the target document type from the target knowledge graph. The query agent is a pre-built agent used to query review rules in the knowledge graph. Based on the aforementioned review rules, the document to be reviewed is reviewed by multiple review agents to obtain the target review result. The multiple review agents are used to review the document content and format.

2. The document review method according to claim 1, characterized in that, The target knowledge graph is obtained through the following methods: Obtain a historical document set and a document template set, wherein the historical document set includes: various types of official documents that have been reviewed and approved in the historical process, and the document template set includes: writing templates for each type of official document; Information is extracted from the historical document set and the document template set to obtain extraction results, wherein the extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document; Based on the extraction results, a knowledge graph is constructed to obtain the target knowledge graph.

3. The document review method according to claim 2, characterized in that, Based on the extraction results, a knowledge graph is constructed to obtain the target knowledge graph, including: Based on the extraction results, multiple triples are constructed; The triplet is deduplicated to obtain the triplet after deduplication. A knowledge graph is constructed based on the deduplicated triples to obtain an initial knowledge graph. Construct multiple target query statements, and based on the multiple target query statements, verify whether the audit rules queried by the target query statements can be found in the initial knowledge graph, and obtain the verification results; If the verification result indicates that the auditing rule queried by the target query statement can be found in the initial knowledge graph, then the initial knowledge graph is determined as the target knowledge graph.

4. The document review method according to claim 1, characterized in that, The document type is identified by a type recognition agent to determine the document type of the document to be reviewed, resulting in the target document type, including: The document features of the document to be reviewed are extracted by the type recognition agent, wherein the document features include at least one of the following: document title, document structure; Obtain the document type standard to get the type standard; The target document type is determined based on the type criteria and the document characteristics.

5. The document review method according to claim 4, characterized in that, Based on the target document type, the query agent retrieves the associated review rules for the target document type from the target knowledge graph, including: Based on the target document type, the query agent generates a data query statement; Execute the data query statement to obtain the review rules associated with the target document type.

6. The document review method according to claim 1, characterized in that, The multiple review agents include: a content review agent for reviewing document content and a format review agent for reviewing document format. The review rules include: content review rules and format review rules. Based on the review rules, the document to be reviewed is reviewed by the multiple review agents to obtain a target review result, including: Based on the content review rules, the content of the document to be reviewed is reviewed by the content review intelligent agent to obtain a first review result; Based on the format review rules, the format of the document to be reviewed is reviewed by the format review agent to obtain a second review result; Based on the first audit result and the second audit result, the target audit result is determined.

7. The document review method according to claim 3, characterized in that, After reviewing the document to be reviewed by multiple review agents based on the aforementioned review rules to obtain the target review result, the process further includes: Based on the target audit results, an audit report is generated, which includes: the problems existing in the document to be audited and the location of the problems.

8. A document review system, characterized in that, The document review system is used to execute the document review method according to any one of claims 1 to 7, including: The input layer is used to receive the document to be reviewed; The intelligent agent layer includes: a type identification intelligent agent, a query intelligent agent, and multiple review intelligent agents. The type identification intelligent agent identifies the document type of the document to be reviewed, obtaining a target document type. Based on the target document type, the query intelligent agent queries the review rules associated with the target document type from a target knowledge graph. Based on the review rules, the multiple review intelligent agents review the document to be reviewed, obtaining a target review result. The type identification intelligent agent identifies the document type, the query intelligent agent queries the target knowledge graph for the required data, and the multiple review intelligent agents review the document content and format. The output layer is used to output the audit results.

9. The document review system according to claim 8, characterized in that, The document review system also includes: A standard construction layer is used to obtain a historical document set and a document template set, extract information from the content in the historical document set and the document template set to obtain extraction results, and construct a knowledge graph based on the extraction results to obtain the target knowledge graph. The historical document set includes: various types of official documents that have been approved in historical time. The document template set includes: writing templates for each type of official document. The extraction results include at least one of the following: format requirements, content elements, and filling rules for each type of official document.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the document review method according to any one of claims 1 to 7.

Images