A document intelligent review method and system based on multi-modal artificial intelligence and rule engine

Abstract

The application relates to the field of artificial intelligence and natural language processing technology, in particular to a document intelligent review method and system based on a multi-modal artificial intelligence and a rule engine, which solves the problems that an existing review scheme driven by a pure rule engine lacks semantic understanding and cannot understand context, a review scheme model based on a traditional natural language processing model has limited model generalization ability and still needs a large amount of feature engineering, a scheme based on a pure large language model for direct review has untraceable and unverifiable results, and the scheme has high delay and high cost, and the application comprises a document preprocessing and analysis module, a rule intelligent analysis module, a key information intelligent extraction module and a collaborative intelligent review engine. The application positions a large language model LLM as an enhanced component of a rule engine, utilizes the capability of the large language model LLM to understand natural language rules, extract complex information and perform deep semantic reasoning, and the framework of the whole review process, the final output of decision and the accurate positioning of results are dominated by the controllable and explainable rule engine.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically to a document intelligent review method and system based on multimodal artificial intelligence and rule engine. Background Technology

[0002] In many fields such as business, law, and finance, the review of documents such as contracts, agreements, prospectuses, and audit reports is a crucial and extremely demanding task. Traditional manual review methods rely heavily on the experience and attention of professionals, resulting in inefficiency, high costs, susceptibility to oversights, and inconsistent standards. With the development of information technology, some automated document processing technologies have emerged, such as rule-based review systems based on keyword matching and regular expressions, review systems based on single natural language processing models, and document question-answering systems based on large language models.

[0003] Existing review schemes driven by pure rule engines lack semantic understanding and cannot comprehend context. Review schemes based on traditional natural language processing models have limited generalization capabilities and still require extensive feature engineering. Schemes based on pure large language models for direct review have untraceable and unverifiable results, resulting in high latency and high cost. Therefore, they do not meet current needs. To address this, we propose a document intelligent review method and system based on multimodal artificial intelligence and rule engines. Summary of the Invention

[0004] The purpose of this invention is to provide a document intelligent review method and system based on multimodal artificial intelligence and rule engine, in order to solve the problems mentioned in the background art, such as the lack of semantic understanding in existing pure rule engine driven review schemes, the limited generalization ability of review schemes based on traditional natural language processing models, the need for a large amount of feature engineering, the lack of traceability and verification of results in schemes based on pure large language models, and high latency and high cost.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a document intelligent review system based on multimodal artificial intelligence and a rule engine, comprising a document preprocessing and parsing module, a rule intelligent analysis module, a key information intelligent extraction module, and a collaborative intelligent review engine: The document preprocessing and parsing module is used to convert raw documents in various formats into structured data rich in semantic and structural information that can be understood and processed through computer vision models, and to identify different types of visual elements; The rule intelligent analysis module uses a large language model (LLM) to compile natural language rules into executable structured rules, and inputs the structured rules in natural language form and outputs structured JSON objects. The key information intelligent extraction module is used to efficiently and accurately extract all key information fragments required for review from rich document paragraphs according to the requirements of the structured review rule set. The collaborative intelligent review engine uses one of its two sub-engines to execute multiple review rules and aggregates and precisely locates the results.

[0006] Preferably, the computer vision model includes the layout recognition model LayoutLM and the real-time object detection model YOLO. The visual elements consist of text blocks and non-text blocks. The text blocks include, but are not limited to, multi-level headings, paragraphs, list items, headers and footers, and page numbers. The non-text blocks include, but are not limited to, tables and stamps.

[0007] Preferably, the JSON object includes a rule ID, a rule description, a rule type, and a detailed list of key information extraction elements. The JSON object also includes one of the rule target chapter and rule keywords.

[0008] Preferably, the semantic similarity between the document paragraphs and the rule keywords is calculated using cosine similarity. ; Where D is the word vector of the document paragraph, R is the word vector of the rule keyword, and the threshold for semantic similarity is set to 0.7.

[0009] Preferably, the two sub-engines are a traditional algorithm reviewer and a large language model LLM inference reviewer, respectively.

[0010] A document intelligent review method based on multimodal artificial intelligence and rule engine includes the following steps: S1: Convert all types of documents supported by the system into PDF data streams. Utilize a deep learning-based computer vision model to analyze and identify different types of visual elements on the page, and output the type and bounding box coordinates of each element. Specifically, for the identified text type, use an OCR engine to identify the text content and its precise coordinates on the page. Perform special processing on the identified table areas, use a table recognition model to restore the row and column structure of the table, extract the text content within the cells and the logical relationships between cells, and output structured data. At the same time, perform special processing on the identified stamp areas, and use a stamp recognition model to extract stamp data. S2: Input the natural language rules along with the prompt word templates for rule parsing into the large language model LLM, design sophisticated prompt words to guide the large language model LLM to complete rule classification, determine the category of the rule, so that it can be distributed to different review engines in the future, extract the specific elements required to execute the rule from the rule description, and output a structured JSON object. S3: Receives structured rules from the rule intelligent analysis module and uses these rules to perform fast matching and lightweight semantic analysis on the document structure data generated by the document preprocessing and parsing module. By utilizing the layout analysis results and element coordinate information output by the document preprocessing and parsing module, combined with predefined structured templates, it achieves accurate positioning and efficient extraction of text information. For information with complex semantics and non-fixed positions, it extracts key information elements from relevant document fragments and rule descriptions and inputs them into the large language model LLM. It also requires the large language model LLM to directly output the extracted factual information in a specified format. S4: The collaborative intelligent review engine receives a set of key information organized by chapters generated by the key information intelligent extraction module and a set of structured review rules generated by the rule intelligent analysis module. It then classifies and distributes the rules to one of the two most suitable sub-engines for execution. The traditional algorithm reviewer is suitable for rules that are clear and have simple logical judgments. It directly performs logical operations and matching on the structured data in the key information set, which is fast and accurate. The large language model LLM reasoning reviewer is suitable for complex rules that require deep semantic understanding and reasoning.

[0011] Preferably, the precise positioning formula for text information extraction in S3 is: ; Where f(,) represents the data precisely extracted from the document based on predefined coordinates and a structured template.

[0012] Preferably, the formula for rule classification and distribution is: ; .

[0013] Preferably, the traditional algorithm reviewer is applicable to scenarios such as numerical comparison, format checking, existence checking, and keyword matching, while the large language model LLM inference reviewer is applicable to scenarios such as clause fairness judgment, language ambiguity recognition, and contextual logic consistency verification.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention integrates the precision of a rule engine with the deep semantic understanding capabilities of a Large Language Model (LLM). The system can capture both simple risks based on explicit patterns and identify implicit risks requiring complex reasoning, achieving a balance between high precision and high recall. Its modular design and parallel processing strategy significantly improve review speed, enabling it to handle massive amounts of documents. The natural language rule definition approach allows business experts to quickly participate in rule customization and iteration, and the system boasts extremely high scalability. 2. This invention, through its review logic ultimately driven by clear, structured rules, ensures that every risk conclusion can be traced back to specific rules and the original text, meeting the stringent requirements of commercial and legal scenarios for process credibility and verifiable results. It can understand and process various types of information in documents, such as text, tables, seals, and layouts, achieving true "document" understanding and avoiding review blind spots caused by ignoring non-textual information. Business personnel do not need to master programming skills to define complex review requirements through natural language, greatly reducing manpower and time costs, and enabling intelligent review technology to be quickly implemented in a wider range of business scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the intelligent document review method of the present invention; Figure 2 This is a schematic diagram of the intelligent document review system of the present invention; Figure 3 This is a schematic diagram of the document preprocessing and parsing module of the present invention; Figure 4 This is a schematic diagram of the key information intelligent extraction module of the present invention; Figure 5 This is a schematic diagram of the collaborative intelligent review engine of the present invention; Figure 6 A schematic diagram of the implementation code for the document preprocessing and parsing module of this invention; Figure 7 A schematic diagram of the implementation code for the rule-based intelligent analysis module of this invention; Figure 8 This is a schematic diagram of the implementation code for the intelligent extraction module of key information in this invention; Figure 9 This is a schematic diagram of the LLM (Large Language Model) review prompts of the present invention; Figure 10 This is a schematic diagram of the implementation code of the collaborative intelligent review engine of the present invention. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0017] Please see Figure 2 , Figure 3 and Figure 6This invention provides an embodiment of a document intelligent review system based on multimodal artificial intelligence and a rule engine, comprising a document preprocessing and parsing module, a rule intelligent analysis module, a key information intelligent extraction module, and a collaborative intelligent review engine. The document preprocessing and parsing module is used to convert original documents in various formats into structured data rich in semantic and structural information that can be understood and processed through computer vision models, and to identify different types of visual elements. The computer vision models include the layout recognition model LayoutLM and the real-time object detection model YOLO. Visual elements consist of text blocks and non-text blocks. Text blocks include, but are not limited to, multi-level headings, paragraphs, list items, headers and footers, and page numbers. Non-text blocks include, but are not limited to, tables and stamps. The document preprocessing and parsing module adopts integrated pipeline technology to perform in-depth analysis of the document and output rich information dimensions.

[0018] Please see Figure 7 The rule intelligent analysis module uses a large language model (LLM) to compile natural language rules into executable structured rules. These structured rules are then input into and output as structured JSON objects in natural language form. Each JSON object includes a rule ID, rule description, rule type, and a detailed list of key information extraction elements. It also includes one of the following: rule target section and rule keywords. The rule intelligent analysis module outputs a set of structured review rules, where each rule clearly defines its intent, target, and checkpoints, providing precise instructions for subsequent modules. Rules are input in natural language, for example: "Examine whether the rights and obligations of both parties are equal." "Check whether the contract amount is in words and consistent with Arabic numerals." "Confirm whether the confidentiality period is clearly defined and is no less than three years." “Identify all liability limitation clauses and examine their reasonableness.”

[0019] Extract the specific elements required to execute the rule from the rule description. For example, for the rule "Confirm whether the confidentiality period is clearly defined and not less than three years", the Large Language Model (LLM) should extract: Target audience: Confidentiality period; Check condition 1: Is it clear (existence check); Inspection condition 2: Value ≥ 3 years (value comparison).

[0020] Please see Figure 4 and Figure 8The key information intelligent extraction module is used to efficiently and accurately extract all the key information fragments required for the review from rich document paragraphs according to the requirements of the structured review rule set. The semantic similarity between document paragraphs and rule keywords is calculated by cosine similarity. For example, the review rules for "confidentiality clauses" prioritize paragraphs that are highly related to keywords such as "confidentiality" and "trade secrets." This process is not only based on precise string matching, but may also combine semantic matching technologies such as vector similarity to capture synonymous expressions, thereby achieving accurate intelligent anchoring.

[0021] ; Where D is the word vector of the document paragraph, R is the word vector of the rule keyword, and the semantic similarity threshold is set to 0.7.

[0022] For information with complex semantics and non-fixed location, extract key information elements from relevant document fragments and rule descriptions and input them into the Large Language Model (LLM). The LLM is required to directly output the extracted factual information in a specified format. For example, the LLM may output: {“Obligation Party”: “Party A”, “Obligation Description”: “Payment for Goods”, “Right Party”: “Party B”, “Right Description”: “Delivery of Goods”}.

[0023] Please see Figure 5 , Figure 9 and Figure 10 The collaborative intelligent review engine uses one of two sub-engines to execute multiple review rules and aggregates and precisely locates the results. The two sub-engines are a traditional algorithm reviewer and a large language model LLM inference reviewer.

[0024] Please see Figure 1 and Figure 2 A document intelligent review method based on multimodal artificial intelligence and rule engine includes the following steps: S1: Convert all types of documents supported by the system into PDF data streams. Utilize a deep learning-based computer vision model to analyze and identify different types of visual elements on the page, and output the type and bounding box coordinates of each element. Specifically, for the identified text type, use an OCR engine to identify the text content and its precise coordinates on the page. Perform special processing on the identified table areas, use a table recognition model to restore the row and column structure of the table, extract the text content within the cells and the logical relationships between cells, and output structured data. At the same time, perform special processing on the identified stamp areas, and use a stamp recognition model to extract stamp data. S2: Input the natural language rules along with the prompt word templates for rule parsing into the large language model LLM, design sophisticated prompt words to guide the large language model LLM to complete rule classification, determine the category of the rule, so that it can be distributed to different review engines in the future, extract the specific elements required to execute the rule from the rule description, and output a structured JSON object. S3: Receives structured rules from the rule intelligent analysis module and uses these rules to perform fast matching and lightweight semantic analysis on the document structure data generated by the document preprocessing and parsing module. By utilizing the layout analysis results and element coordinate information output by the document preprocessing and parsing module, combined with predefined structured templates, it achieves accurate positioning and efficient extraction of text information. For information with complex semantics and non-fixed positions, it extracts key information elements from relevant document fragments and rule descriptions and inputs them into the large language model LLM. It also requires the large language model LLM to directly output the extracted factual information in a specified format. S4: The collaborative intelligent review engine receives a set of key information organized by chapters generated by the key information intelligent extraction module and a set of structured review rules generated by the rule intelligent analysis module. It then classifies and distributes the rules to one of the two most suitable sub-engines for execution. The traditional algorithm reviewer is suitable for rules that are clear and have simple logical judgments. It directly performs logical operations and matching on the structured data in the key information set, which is fast and accurate. The large language model LLM reasoning reviewer is suitable for complex rules that require deep semantic understanding and reasoning.

[0025] The precise positioning formula for text information extraction in S3 is: ; Where f(,) represents the data precisely extracted from the document based on predefined coordinates and a structured template.

[0026] For example, it can directly locate specific areas near the document title or extract the content of specified cells using table coordinates. It does not rely on deep semantic understanding, but rather on the physical layout and coordinate data of the document for fast and accurate positioning and extraction, featuring extremely fast processing speed and high accuracy.

[0027] Furthermore, the formula for rule classification and distribution is as follows: ; .

[0028] Traditional algorithm reviewers are suitable for scenarios such as numerical comparison, format checking, existence checking, and keyword matching; Execution process: Directly performs logical operations and matching on the structured data in the key information set, which is fast and accurate.

[0029] Example: "For the rule 'check whether the contract amount is in uppercase and consistent with Arabic numerals,' the system first uses a 'precise extraction based on structure and coordinates' strategy to locate the text block containing the contract amount and the table cell below it. Then, the traditional algorithm reviewer performs format verification and numerical comparison on the extracted uppercase amount string and Arabic numerals to complete the review." The Large Language Model (LLM) inference reviewer is suitable for scenarios such as determining the fairness of terms, identifying linguistic ambiguity, and verifying the consistency of contextual logic. Execution process: The key information set of relevant chapters and the rule descriptions that need to be reasoned are input into the Large Language Model (LLM). Through carefully designed prompts, the Large Language Model (LLM) is required to play the role of a "review expert", making judgments based solely on the provided key information and outputting review conclusions in strict accordance with the specified format (e.g., {"Risk Level": "High", "Risk Description": "The terms clearly favor Party B", "Basis for Judgment": "...", "Original Location": [Page Number, Paragraph Index]}). This effectively limits the "illusion" of the Large Language Model (LLM).

[0030] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A multi-modal artificial intelligence and rules engine based intelligent document review system, characterized in that, It includes a document preprocessing and parsing module, a rule-based intelligent analysis module, a key information intelligent extraction module, and a collaborative intelligent review engine: The document preprocessing and parsing module is used to convert raw documents in various formats into structured data rich in semantic and structural information that can be understood and processed through computer vision models, and to identify different types of visual elements; The rule intelligent analysis module uses a large language model (LLM) to compile natural language rules into executable structured rules, and inputs the structured rules in natural language form and outputs structured JSON objects. The key information intelligent extraction module is used to efficiently and accurately extract all key information fragments required for review from rich document paragraphs according to the requirements of the structured review rule set. The collaborative intelligent review engine uses one of its two sub-engines to execute multiple review rules and aggregates and precisely locates the results.

2. The system as claimed in claim 1, wherein the system is based on multi-modal artificial intelligence and rule engine for intelligent review of documents. The computer vision model includes the layout recognition model LayoutLM and the real-time object detection model YOLO. The visual elements consist of text blocks and non-text blocks. The text blocks include, but are not limited to, multi-level headings, paragraphs, list items, headers and footers, and page numbers. The non-text blocks include, but are not limited to, tables and stamps.

3. The system as claimed in claim 2, wherein the system is based on multi-modal artificial intelligence and rule engine for intelligent review of documents. The JSON object includes a rule ID, rule description, rule type, and a detailed list of key information extraction elements. The JSON object also includes one of the rule target chapter and rule keywords.

4. The system as claimed in claim 3, wherein the system is based on multi-modal artificial intelligence and rule engine for intelligent review of documents. The semantic similarity between the document paragraphs and the rule keywords is calculated using cosine similarity. ； Where D is the word vector of the document paragraph, R is the word vector of the rule keyword, and the threshold for semantic similarity is set to 0.

7.

5. The document intelligent review system based on multimodal artificial intelligence and rule engine according to claim 4, characterized in that: The two sub-engines are a traditional algorithm reviewer and a large language model LLM inference reviewer.

6. A document intelligent review method based on multimodal artificial intelligence and rule engine, and a document intelligent review system based on multimodal artificial intelligence and rule engine according to any one of claims 1-5, characterized in that, Includes the following steps: S1: Convert all types of documents supported by the system into PDF data streams. Utilize a deep learning-based computer vision model to analyze and identify different types of visual elements on the page, and output the type and bounding box coordinates of each element. Specifically, for the identified text type, use an OCR engine to identify the text content and its precise coordinates on the page. Perform special processing on the identified table areas, use a table recognition model to restore the row and column structure of the table, extract the text content within the cells and the logical relationships between cells, and output structured data. At the same time, perform special processing on the identified stamp areas, and use a stamp recognition model to extract stamp data. S2: Input the natural language rules along with the prompt word templates for rule parsing into the large language model LLM, design sophisticated prompt words to guide the large language model LLM to complete rule classification, determine the category of the rule, so that it can be distributed to different review engines in the future, extract the specific elements required to execute the rule from the rule description, and output a structured JSON object. S3: Receives structured rules from the rule intelligent analysis module and uses these rules to perform fast matching and lightweight semantic analysis on the document structure data generated by the document preprocessing and parsing module. By utilizing the layout analysis results and element coordinate information output by the document preprocessing and parsing module, combined with predefined structured templates, it achieves accurate positioning and efficient extraction of text information. For information with complex semantics and non-fixed positions, it extracts key information elements from relevant document fragments and rule descriptions and inputs them into the large language model LLM. It also requires the large language model LLM to directly output the extracted factual information in a specified format. S4: The collaborative intelligent review engine receives a set of key information organized by chapters generated by the key information intelligent extraction module and a set of structured review rules generated by the rule intelligent analysis module. It then classifies and distributes the rules to one of the two most suitable sub-engines for execution. The traditional algorithm reviewer is suitable for rules that are clear and have simple logical judgments. It directly performs logical operations and matching on the structured data in the key information set, which is fast and accurate. The large language model LLM reasoning reviewer is suitable for complex rules that require deep semantic understanding and reasoning.

7. The document intelligent review method based on multimodal artificial intelligence and rule engine according to claim 6, characterized in that: The precise positioning formula for text information extraction in S3 is as follows: ； Where f(,) represents the data precisely extracted from the document based on predefined coordinates and a structured template.

8. The document intelligent review method based on multimodal artificial intelligence and rule engine according to claim 7, characterized in that: The formula for rule classification and distribution is as follows: ； 。 9. The document intelligent review method based on multimodal artificial intelligence and rule engine according to claim 8, characterized in that: The traditional algorithm reviewer is suitable for scenarios such as numerical comparison, format checking, existence checking, and keyword matching, while the large language model LLM inference reviewer is suitable for scenarios such as clause fairness judgment, language ambiguity recognition, and contextual logic consistency verification.

Images