A method and system for intelligent structured processing of a regulation document, and a medium

Abstract

The application discloses a kind of intelligent structured processing method and system of rules and regulations document, belong to artificial intelligence and natural language processing technical field.Along with the problem that the semantic of long regulation document in prior art is split due to the input length limit of large language model, output is unreliable, the application fuses the paragraph type by small model and semantic feature recognition, corrects label in combination with business rules, and aggregates semantic complete text fragment according to regulation level logic;The segment is constructed as guide prompt input large language model to generate structured result along with type label, and JSON format and field integrity check are set, and when check fails, it is returned to small model result.The method guarantees the integrity of clause-level semantics, improves the accuracy of key field extraction, and ensures system robustness.It is suitable for enterprise compliance management, regulation knowledge base construction and other scenarios, to realize the reliable conversion of unstructured regulation document to unified structured data.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to an intelligent structured processing method, system and medium for regulations and institutional documents. Background Technology

[0002] In corporate governance, administrative management, and legal compliance scenarios, regulatory documents (such as company articles of association, internal management systems, government normative documents, and industry policies) are generally in the form of unstructured electronic documents such as PDF and Word. Although these documents have a clear hierarchical logical structure in terms of content (usually organized in the form of "chapter-article-section-item"), their text presentation lacks machine-readable semantic markup, making it difficult for computers to automatically identify clause boundaries, hierarchical relationships, and key elements. This limitation severely restricts the development of subsequent intelligent applications, such as precise clause retrieval, cross-regulatory compliance comparison, automated risk review, and the construction of corporate compliance knowledge graphs.

[0003] Currently, the industry mainly attempts to use Large Language Models (LLMs) to perform end-to-end structured parsing of entire regulatory documents. However, regulatory documents are usually quite long (thousands to tens of thousands of words), far exceeding the single input length limit of mainstream LLMs (generally 4K–32K tokens). If the document is simply segmented into fixed lengths and fed into the model separately, semantic units are easily fragmented—for example, assigning the main text "Article 3" and its subordinate items (such as "(I)" and "(II)") to different processing segments, resulting in fragmented parsing results, missing key fields, or disordered logical structure, seriously affecting the output quality.

[0004] Furthermore, large language models inherently exhibit uncertainties when generating structured data (such as JSON format): they may output grammatically invalid formats, omit necessary fields (such as "clause number" and "responsible party"), or even produce illusory content that does not match the original text. If the system lacks an effective validation mechanism for the output results and a reliable rollback strategy, its robustness will be insufficient to meet the stringent requirements of accuracy, consistency, and stability in enterprise-level application scenarios.

[0005] To circumvent the aforementioned problems, some solutions have shifted to rule-based methods, such as using regular expressions to match patterns like "Article X" to identify the start of a clause. However, such methods have extremely poor generalization ability and struggle to adapt to the widespread format diversity in real-world documents (such as mixed Chinese and English text, custom numbering systems, and non-standard formatting). Other solutions rely solely on lightweight models to classify paragraphs (e.g., distinguishing between "headings" and "body text"), which improves processing efficiency but fails to achieve deep semantic analysis (e.g., extracting structured elements such as scope of application, responsible parties, and exceptions).

[0006] In summary, existing technologies struggle to balance ensuring the semantic integrity of long documents, the depth of structured parsing, and the robustness of system operation. There is an urgent need for a novel processing method that integrates multi-model collaboration, structure-aware segmentation, and intelligent fault-tolerance mechanisms to achieve efficient, accurate, and reliable structured conversion of regulatory documents. Summary of the Invention

[0007] This application provides an intelligent structured processing scheme for regulations and institutional documents, aiming to solve problems in existing technologies such as semantic fragmentation caused by input length limitations of long regulations and institutional documents, system vulnerability caused by unreliable output of large models, and insufficient generalization ability of rule-based or single-model methods. This method achieves high-accuracy and high-robustness structured parsing of regulations and institutional documents through collaboration between small models and large language models, structure-aware segmentation, and intelligent backoff mechanisms.

[0008] Specifically, the above method includes the following steps: In the first aspect of this application, a method for intelligent structured processing of rules and regulations documents is provided, comprising the following steps: The system acquires the electronic document of rules and regulations to be processed and performs paragraph-level parsing on the electronic document to obtain multiple text paragraphs. Each text paragraph is a continuous text unit separated by the layout or line breaks of the electronic document. For each text paragraph, extract the text content and layout style features; The text content and layout style features are input into a pre-trained small model, and the small model outputs the first structured data. The first structured data includes at least the original text content of the text paragraph and paragraph type labels. The paragraph type labels include at least one of chapter title, clause main item, clause sub-item, and body paragraph. Based on paragraph type labels, consecutive text paragraphs are aggregated to form multiple text fragments. Each text fragment contains semantically coherent and structurally complete institutional content, and the character length of the text fragment does not exceed the single input length limit of the large language model. The text content of each text fragment and its corresponding paragraph type label are input into the large language model, and the large language model generates a second structured data as the structured result of the text fragment; If the second structured data does not conform to the preset structure specification, then the structured result of the text fragment is generated based on the first structured data corresponding to each text paragraph in the text fragment; Summarize the structured results of all text fragments and output structured rules and regulations data in a unified format.

[0009] This application performs paragraph-level parsing on electronic documents of regulations and rules to be processed, and uses a small model to generate first structured data containing paragraph type tags. Then, based on these tags, it aggregates to form semantically complete text fragments, effectively avoiding the context truncation and semantic fragmentation problems caused by directly inputting long documents into a large language model. On this basis, by inputting the text fragments into a large language model to generate second structured data, and setting a dual guarantee mechanism of "if it does not conform to the preset structure specification, it will fall back to the result generated based on the first structured data", it not only leverages the advantages of the large language model in deep semantic understanding, but also avoids the risks of unstable output and uncontrollable format. Finally, it summarizes the structured results of all fragments and outputs data in a unified format, ensuring that the entire processing flow has strong robustness and engineering usability while maintaining high parsing depth, significantly improving the reliability and practicality of structuring long regulatory documents.

[0010] As a preferred technical solution of this application, the typesetting style features include at least two of the following: paragraph indentation, font size, whether centered, and whether bold.

[0011] This application makes full use of the inherent formatting rules of institutional documents (such as centered and bolded titles, and no indentation in clauses) to enhance the small model's ability to distinguish paragraph types, especially providing key auxiliary information when the semantics of plain text are ambiguous.

[0012] As a preferred technical solution of this application, the input features of the small model also include keyword features extracted based on the word frequency-inverse document frequency (TFIDF) algorithm, and / or semantic features generated based on a pre-trained word vector model.

[0013] This application enhances the sensitivity of small models to institutional domain terms (such as "scope of application" and "responsible entity") by integrating statistical features and deep semantic representations, thereby improving their generalization ability under diverse document styles.

[0014] As a preferred technical solution of this application, after the structured data of the text paragraph is output by the small model, the method further includes the following steps: If a text paragraph meets the preset business rule conditions, its paragraph type label will be forcibly changed to the specified type; Business rule conditions include at least one of the following: The text begins with "Article X", "Section X", or "Item X" and does not end with a period; The corresponding paragraph of the text content is displayed in the center, with bold font, and has fewer than 30 characters; The text content is located in the first 10% of the document and contains the words "Table of Contents" or "Contents".

[0015] This application introduces prior knowledge from the institutional domain to accurately correct misjudgments in typical patterns of small models (such as misjudging "Chapter 1" as the main text), significantly improving label accuracy and providing a reliable basis for subsequent aggregation.

[0016] As a preferred technical solution of this application, the aggregation of consecutive text paragraphs based on paragraph type tags includes: Starting with a text paragraph labeled "Main Clause", aggregate subsequent text paragraphs labeled "Sub-clause" or "Body Paragraph" into the same text segment until the next text paragraph labeled "Main Clause" or "Chapter Title" is encountered.

[0017] This application strictly follows the hierarchical logic of "chapter-article-clause-item" in the system document for semantic segmentation, ensuring that each text fragment sent into the large model is a complete clause unit, fundamentally avoiding the semantic fragmentation problem caused by mechanical truncation.

[0018] As a preferred technical solution of this application, the text content of each text segment and its corresponding paragraph type label are input into the large language model, including: Convert all paragraph type tags in the text fragment into guiding statements according to a preset template; The guiding statement is appended before the text content of the text fragment as an input prompt for the large language model.

[0019] This application guides the large model to focus on the institutional context by explicitly injecting structured context (such as "[Main Clause] Article 3..."), which significantly improves the accuracy and consistency of extracting key fields such as clause number and main text content.

[0020] As a preferred technical solution of this application, the preset structural specifications include: The second structured data is a string that conforms to JSON syntax, and the data structure obtained after parsing contains at least one of the 'clause number' field or the 'main text content' field, and the value of the 'clause number' field is not empty.

[0021] This application establishes an objective and automatically executable structural validation standard to effectively filter out format errors, missing fields, or null values ​​in the output of large models, ensuring the usability of deep parsing results.

[0022] Furthermore, in the structured result of generating text fragments based on the first structured data, the generated structured result is a JSON object containing the following fields: The rollback flag field, whose value is true; The paragraph list field contains the paragraph type label and original text content for each paragraph in the text fragment.

[0023] This application still outputs a lightweight representation with a clear structure and well-defined fields even under the fallback path, and uses fallback identifiers to facilitate downstream systems in distinguishing processing strategies, ensuring the uniformity and traceability of the overall output format.

[0024] A second aspect of this application provides an intelligent structured processing method for rules and regulations documents, including: A smart structured processing system for rules and regulations documents is provided, including a document parsing module, a feature extraction module, a small model processing module, a rule correction module, a semantic aggregation module, a large model parsing module, and a result output module. Each module works together to execute the above methods.

[0025] The configuration of each module of this system corresponds one-to-one with the steps of the aforementioned method. For specific technical effects, please refer to the relevant descriptions in the method section; they will not be repeated here. The above system implements the engineering deployment of the method described in the first aspect, supports high-concurrency, low-latency batch processing of policy documents, and meets the stability and scalability requirements of an enterprise-level compliance management platform.

[0026] A third aspect of this application provides a computer-readable storage medium storing a computer program thereon, facilitating the distribution, integration, or embedding of this application in software form into existing information systems, lowering the technical implementation threshold, and expanding the application coverage. When executed by a processor, the program implements the method described in the first aspect above. Specific technical effects are described in the relevant section on methods and will not be repeated here. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. These drawings may be referenced together unless otherwise specified.

[0028] Figure 1 A flowchart illustrating the intelligent structured processing method for rules and regulations documents provided in this embodiment of the invention; Figure 2 A block diagram of the module structure of the intelligent structured processing system for rules and regulations documents provided in this embodiment of the invention. Detailed Implementation

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, a brief introduction to this application will be given below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the descriptions of these embodiments are intended to help understand this application, but do not constitute a limitation on this application.

[0030] This invention provides an intelligent structured processing method for regulations and rules documents, aiming to solve the semantic fragmentation problem caused by long regulations and rules documents exceeding the input length limit of large language models, as well as the system vulnerability caused by the unreliability of large model outputs. Figure 1 As shown, the overall process of this method includes the following core steps: S1. Obtain the electronic document of the rules and regulations to be processed, and parse it at the paragraph level to obtain multiple text paragraphs.

[0031] The electronic documents of rules and regulations to be processed are in PDF or Word format. Each text paragraph is a continuous text unit separated by the document's layout (such as paragraph spacing and indentation) or line breaks.

[0032] In one specific embodiment, the present invention first performs structured parsing on the electronic document of rules and regulations to be processed. Taking a PDF document titled "Enterprise Employee Code of Conduct" as an example, the system loads the document using an open-source PDF parsing library (such as PyMuPDF or pdfplumber) and automatically segments the entire document into multiple independent text paragraphs based on its internal text flow layout information (including character position, line height, paragraph spacing, etc.) and explicit line breaks (\n or \r\n). Each text paragraph is defined as a continuous text unit separated by layout or line breaks. For example, "Chapter 1 General Provisions", "Article 1 To regulate employee behavior...", "(I) Comply with all company rules and regulations..." are all identified as independent paragraphs.

[0033] In one embodiment, the paragraph-level parsing employs a layout analysis algorithm based on visual blocks. When the vertical spacing between adjacent text lines is greater than 1.5 times the average line height, or the difference in horizontal indentation exceeds 2 character widths, it is determined to be a paragraph separation boundary.

[0034] S2. For each text paragraph, extract its text content and layout style features.

[0035] After obtaining the text paragraphs, the system extracts two types of core features in parallel for each paragraph: The first category is layout style features. The system obtains the visual presentation attributes of each paragraph by parsing the metadata of the PDF or Word document. According to claim 2, these layout style features include at least two of the following four: Paragraph indentation: Measures the offset in pixels of the first line of a paragraph relative to the left margin of the page; Font size: The font size of the main text of a paragraph is recorded in points (pt); Center alignment: Determines whether the paragraph text is centered horizontally; Bold: Detects whether the paragraph text has the bold style applied.

[0036] For example, for the paragraph "Chapter 1 General Provisions", the system might extract: indentation = 0px, font size = 16pt, centering = True, bolding = True. For the body text of a regular clause, "All employees must clock in on time...", its characteristics might be: indentation = 20px, font size = 10.5pt, centering = False, bolding = False. These formatting features directly reflect the hierarchical structure of the policy document.

[0037] The second category is textual semantic features. The system further processes the plain text content of the paragraphs in depth: First, the term frequency-inverse document frequency (TFIDF) algorithm is used to calculate the weight of each word from a pre-built corpus of institutional documents, generating a high-dimensional sparse keyword vector to capture the topic keywords of paragraphs (such as "responsibility", "punishment", "scope of application" etc.). Secondly, a pre-trained word vector model for a general domain (such as BERT-base or Sentence-BERT) is invoked to encode the entire paragraph into a dense 768-dimensional semantic vector to capture contextual semantic information.

[0038] Finally, the system concatenates the aforementioned layout style features, TFIDF keyword features, and pre-trained word vector semantic features to form a hybrid feature vector. This vector serves as the input to the small model processing module for subsequent paragraph type prediction. By fusing multi-source heterogeneous features, this invention significantly improves the small model's discriminative ability and generalization performance when faced with institutional documents with diverse formats and ambiguous semantics.

[0039] Furthermore, in one embodiment, the threshold for determining the paragraph indentation is: if the first-line indentation pixel value is greater than 5% and less than 20% of the page width, it is considered a "sub-item of clause"; if the indentation is 0, it is further determined in conjunction with the font size.

[0040] S3. Input these features into a pre-trained small model. The small model outputs the first structured data, which contains at least the original text content of the paragraph and the paragraph type label.

[0041] After completing the multidimensional feature extraction, the system enters the small model classification stage. Specifically, this embodiment uses a pre-trained XGBoost classifier as the core model. The input of this model is the mixed feature vector generated in the previous stage (including layout, TFIDF, and semantic features), and the output is a paragraph type label. This label belongs to one of the preset sets {"Chapter Title", "Main Clause", "Sub-clause", "Paragraph"}.

[0042] As a feasible implementation method, the model training process is as follows: Over 10,000 paragraph samples from real corporate policy documents were collected and labeled, and cross-labeled by three professionals with legal or compliance backgrounds to ensure label consistency (Kappa coefficient > 0.85). After training, the model was serialized, saved, and integrated into the system.

[0043] In one embodiment, an intelligent structured processing method for rules and regulations documents is provided, wherein the input features of the small model are weighted and fused with TFIDF keyword features and pre-trained word vector semantic features through a gating attention mechanism, wherein the gating weight is dynamically adjusted by the paragraph character length.

[0044] For each input paragraph, the small model processing module performs inference and outputs the first structured data. This data is a structured record containing at least two fields: raw_text: The original text content of the paragraph; type_label: The predicted paragraph type label.

[0045] For example, given the input paragraph "Article 3: All employees must abide by the attendance system...", the small model might output: { "raw_text": "Article 3: All employees must abide by the attendance system..." "type_label": "Main Terms" } However, due to limitations in training data or the extreme diversity of document formats, small models may still misjudge certain typical scenarios. To address this, this invention introduces a rule correction step. Specifically, after the small model outputs the first structured data and before entering the aggregation step, the system checks each paragraph to see if it meets the following preset business rule conditions: Clause number pattern matching: If the raw_text of a paragraph begins with the regular expression [012345678900]+[clause item] or \d+[clause item] (e.g., "Article 1", "Section 3") and does not end with a period, then its type_label will be forcibly modified to "Main Clause Item" or "Sub-clause Item" (based on the specific keywords matched).

[0046] Title visual feature matching: If the paragraph's corresponding layout style features meet the following conditions: centering = True, bold = True, and character count < 30, then force its type_label to be corrected to "Chapter Title".

[0047] Table of contents location feature matching: If a paragraph is located within the first 10% of the total document length and its raw_text contains the words "Table of Contents" or "Contents", then its type_label will be forcibly corrected to "Chapter Title" or the special type "Table of Contents".

[0048] Through the above rule corrections, the system can accurately capture highly regular structural patterns in institutional documents and effectively correct edge case errors in the small model. For example, a "Chapter Two Organizational Structure" that was misjudged as a "body paragraph" by the small model due to abnormal font will be successfully corrected to a "Chapter Title" by the rule correction module because it is centered, bolded, and has fewer than 30 characters.

[0049] This step resulted in each paragraph receiving a high-confidence paragraph type label, providing a solid and reliable foundation for subsequent semantic aggregation.

[0050] In one embodiment, when multiple business rule conditions are met simultaneously, the label is forcibly corrected in the following priority order: directory recognition > chapter title visual features > clause number pattern.

[0051] S4. Aggregate consecutive text paragraphs based on paragraph type tags to form multiple text fragments.

[0052] After obtaining high-confidence paragraph type labels that have been corrected according to rules, the system enters the semantic aggregation stage. The core objective of this step is to reorganize the original, discrete text paragraphs into semantically complete processing units—i.e., text fragments—that conform to the inherent logic of institutional documents, in order to solve the problem of limited input length for large language models.

[0053] In one embodiment, a consecutive text paragraph refers to a text paragraph arranged sequentially in the order of the original electronic document containing regulations. Specifically, when parsing the electronic document at the paragraph level, the system assigns a unique sequence identifier to each text paragraph (e.g., paragraph_id = 1, 2, 3...), which strictly reflects the order in which the paragraph appears in the original document. Aggregation operations are performed only on paragraph sequences with consecutively increasing sequence identifiers.

[0054] For example, if the parsed paragraph sequence is P1, P2, P3, P4, where P1 is the "chapter title", P2 is the "main clause", and P3 and P4 are the "body paragraphs", the aggregation module will aggregate P2, P3, P4 (sequential IDs) into a text fragment until it encounters the next non-sequential type (such as P5 being a new "main clause").

[0055] This application employs an aggregation strategy based on institutional structure rules. Specifically, the system iterates through all annotated text paragraph sequences and executes the following logic: Once a paragraph's type_label is detected as "Clause Main Item" (e.g., "Article 5..."), a new text segment is started from this paragraph.

[0056] Subsequently, the system appends the paragraphs that appear consecutively after the starting point and whose type_label is "Sub-item of Clause" (e.g., "(I)...", "1. ...") or "Paragraph of Main Text" to the current text segment in sequence.

[0057] This aggregation process continues until the next paragraph with a type_label of "Main Item of Clause" or "Chapter Title" is encountered. At this point, the current text fragment is closed, and the construction of the next new fragment begins.

[0058] Assume the original paragraph sequence is: [P1: "Chapter Two Attendance Management" (Chapter Title)] [P2: "Article Four: Working Hours..." (Main Clause)] [P3: "(a) Standard Working Hours System..." (Sub-item of Clause)] [P4: "(II) Special Positions..." (Sub-item of Clause)] [P5: "Article 5: Handling of Late Arrivals..." (Main Clause)] The system will aggregate two text fragments: Text fragment A: contains P2, P3, and P4; Text fragment B: Contains P5.

[0059] Preferably, in one embodiment, when aggregating based on paragraph type tags, if three or more paragraphs of unidentified type appear consecutively, they are automatically classified into the previous valid text segment and marked as "unknown content block".

[0060] During this process, the system monitors the cumulative character count of each text segment in real time. Once the character count approaches but does not exceed the single input limit of the large language model (e.g., 8000 characters), even if a new "main clause" or "chapter title" has not yet been encountered, the system will forcibly end the current segment and start a new segment to ensure that the limit is never exceeded. However, in most policy documents, the total length of a single clause and its sub-items is much smaller than this limit, so it can usually be guaranteed that each segment is a complete semantic unit.

[0061] In one embodiment, when aggregating based on paragraph type tags, if the number of characters in the current text fragment exceeds 90% of the single input limit of the large language model, the aggregation is immediately terminated and a new fragment is started, even if the next "main clause" or "chapter title" has not been encountered.

[0062] In a more preferred embodiment, the maximum single input length of the large language model is dynamically adjusted according to its API billing strategy: when using a token-based billing model, the maximum is set to 80% of the maximum context length; when using a fixed-fee model, the maximum is set to 95% of the maximum context length.

[0063] Through this structure-aware aggregation mechanism, this application successfully decomposes long institutional documents into a series of text fragments with clear boundaries, complete content, and compliant length, fundamentally avoiding the semantic fragmentation problem caused by traditional fixed-length segmentation methods, and laying a solid foundation for the accurate parsing of subsequent large models.

[0064] S5. Input the text content of each text segment and its corresponding paragraph type label into the large language model, and the large language model generates second structured data as the structured result of the text segment.

[0065] After semantic aggregation and the formation of compliant text fragments, the system enters the deep parsing stage. The core of this stage is to call the Large Language Model (LLM) to perform refined structured information extraction for each text fragment.

[0066] Preferably, to maximize the parsing accuracy of the large language model, this application does not directly feed the original text fragment into the model, but instead constructs an input prompt rich in structural prior knowledge. The specific process is as follows: Tag Conversion: For each paragraph in the current text segment, the system reads its type_label (such as "Main Clause" or "Sub-Clause") and converts it into a natural language guiding statement according to a preset template. The template used in this embodiment is: "[{type_label}]{raw_text}".

[0067] For example: If the paragraph type is "Main Clause" and the text is "Article 6 Employees must keep trade secrets...", then it will be converted to: "[Main Clause] Article 6 Employees must keep trade secrets...".

[0068] If the paragraph type is "Sub-item of Clause" and the text is "(I) The company's customer list shall not be disclosed...", then it will be converted to: "[Sub-item of Clause] (I) The company's customer list shall not be disclosed...".

[0069] The system concatenates all the paragraphs in the text fragment into their original order, and appends an explicit instruction at the end, such as: "Please parse the above content into a JSON object containing the fields 'Clause Number' and 'Main Text Content'." The final complete string, as the input suggestion, is sent to a large language model API (such as GPT-4-Turbo) in the cloud. The left side shows the original paragraph and its tags, and the right side shows the constructed final suggestion string.

[0070] In one embodiment, the preset template is: "[{type_label}] Number: {clause_id} Content: {raw_text}", where {clause_id} is extracted from {raw_text} using a regular expression.

[0071] In a preferred embodiment, before inputting the text fragments into the large language model, redundant modifiers are removed by a lightweight summarization model, and the main legal terms are retained, reducing the number of input tokens by 15% to 30% while keeping the key semantics unchanged.

[0072] By explicitly injecting structured context, large language models can clearly identify the role and hierarchical relationship of each sentence in institutional documents. This greatly reduces ambiguity or illusions caused by a lack of domain knowledge, enabling the model to more accurately locate "clause numbers" (such as "Article 6") and extract complete "main text content," thereby generating high-quality second-structured data.

[0073] S6. Perform structure verification on the second structured data: If it does not conform to the preset structure specifications, generate a structured result based on the first structured data corresponding to each paragraph in the text fragment.

[0074] After the large language model generates the second structured data, the system does not directly adopt the result. Instead, it immediately initiates a rigorous structure verification process to ensure the reliability of the output. This process is a key step in ensuring the robustness of the system in this invention.

[0075] Specifically, the verification process includes two levels: Syntax validity check: The system first attempts to parse the second structured data using a standard JSON parser (such as Python's json.loads()). If parsing fails (e.g., due to incomplete parentheses or extra commas in the large model output), it is determined that the data does not conform to the preset structure specifications.

[0076] Semantic integrity check: If the syntax check passes, the parsed JSON object is further checked to see if it contains at least one required field, namely "Clause Number" or "Main Text Content", and the value of the field is not empty (i.e., not null, not an empty string "" or only contains whitespace characters).

[0077] Only if both of the above checks pass simultaneously will the second structured data be accepted as the final result. Otherwise, the system will immediately trigger the intelligent rollback mechanism.

[0078] Furthermore, in one embodiment, the preset structure specification includes three levels of validation: the first level is the validity of the JSON syntax, the second level is the existence of required fields, and the third level is the rationality of field values ​​(e.g., "clause number" must match the regular expression [012345678900\d]+). It is considered valid only when all three levels of validation pass.

[0079] The fallback mechanism doesn't simply abandon the process; instead, it generates a lightweight but well-structured alternative result based on the more reliable initial structured data. Specifically, for text fragments that fail validation, the system constructs a new JSON object as its structured result, containing the following two core fields: fallback: A boolean field whose value is always true, used as an explicit fallback identifier to help downstream applications identify that the result comes from the fallback path.

[0080] paragraphs: A list field containing records of all the original paragraphs within this text segment. Each record is a sub-object containing type_label (paragraph type label) and raw_text (original text content).

[0081] For example, if the large model fails to parse a certain segment, the system may generate the following fallback result: { "fallback": true, "paragraphs": [ { "type_label": "Main Terms", "raw_text": "Article 7 Employees leaving the company must complete handover procedures..." }, { "type_label": "Sub-item of Clause", "raw_text": "(I) The handover work includes..." } ] } In one embodiment, when the structured result contains the "fallback": true field, the lightweight rule engine is automatically triggered to perform a secondary parsing of the paragraphs list and mark the result as "semi-automatic validation passed".

[0082] In a preferred embodiment, the first structured data adopts a unified schema definition and includes at least the following fields: paragraph_id (unique paragraph ID), raw_text (raw text), type_label (type label), and confidence_score (small model confidence score), where confidence_score is used for weight allocation during subsequent aggregation.

[0083] In another preferred embodiment, when the second structured data validation fails, the system first attempts to fix the JSON syntax errors; if the key fields are still missing after the fix, a fallback mechanism is triggered to generate a structured result based on the first structured data.

[0084] This design ensures that even in extreme cases where the large model fails, the system can still output structured data that is uniformly formatted, complete in content, and traceable in origin. This not only guarantees the continuity of the overall processing flow but also provides a clear data foundation for subsequent manual review or secondary processing, greatly enhancing the practical value and stability of this invention in enterprise-level production environments.

[0085] S7. Summarize the structured results of all text fragments and output structured rules and regulations data in a unified format.

[0086] After completing the structured processing of all text fragments (whether successfully parsed by a large model or generated through a fallback mechanism), the system performs the final step: result aggregation.

[0087] The system collects the structured results of all text fragments (including secondary structured data and fallback results) in their original order in the original document and integrates them into a single, top-level JSON array object. Each element of this array is the structured result of a text fragment.

[0088] The final output of structured rules and regulations data in a uniform format has the following characteristics: Format consistency: Regardless of the source, all fragments are presented in standard JSON format; Structural integrity: Includes complete terms and conditions and metadata (such as rollback indicators); Order fidelity: Strictly maintain the same order of clauses as the original document.

[0089] For example, the final output might look like this: [ { / * Successful parsing results from the large model * / }, { "fallback": true, "paragraphs": [...]}, / / Result from the fallback mechanism { / * Another successfully parsed result * / } ] In one embodiment, the structured rules and regulations data in the unified format includes a version field schema_version, which is used to identify the compatibility level of the output data structure and supports downstream systems to parse it by version.

[0090] This unified data format can be seamlessly integrated into downstream systems (such as compliance knowledge bases, audit engines, or front-end display modules) without additional format adaptation or cleaning, thus achieving end-to-end automated conversion from unstructured policy documents to machine-readable and operable structured data.

[0091] This application constructs an intelligent structured processing method for regulations and institutional documents. Its overall technical effect lies in achieving highly complete, accurate, and robust structured parsing of long, unstructured regulatory documents without relying on manual intervention. Specifically, this is reflected in the following three aspects: Ensure the integrity of semantic units By combining layout style features and semantic features with a small model to identify paragraph types, and using business rules to correct typical misjudgments, the original document is then segmented into complete semantic fragments that conform to the hierarchical structure of the system, based on the aggregation logic of "starting with the 'main clause' and ending with the next 'main clause' or 'chapter title'". This mechanism effectively avoids the problem of clause fragmentation caused by the input length limitation of large language models.

[0092] Improve the accuracy of deep analysis By converting paragraph type tags into guiding statements and concatenating them before the text content as prompts for the large model, structural prior knowledge is explicitly injected, guiding the large language model to focus on the institutional context, significantly improving the accuracy and consistency of extracting key fields such as "clause number" and "main text content".

[0093] Ensure system robustness By setting strict structural validation standards, the system verifies the legality and completeness of the output of the large model. In case of validation failure, it automatically reverts to a lightweight structured representation generated from the results of the smaller model, containing a rollback flag and the original paragraph list. This dual-protection mechanism ensures that the system can output structured data with consistent format, usable content, and traceable source under all circumstances.

[0094] In summary, this invention solves three major technical challenges simultaneously—semantic fragmentation of long documents, unreliable output of large models, and poor generalization of rule-based methods—through a multi-stage collaborative processing architecture, achieving end-to-end reliable conversion from unstructured institutional texts to machine-readable structured data.

[0095] To facilitate the engineering implementation of the aforementioned methods and solutions, this invention also provides an intelligent structured processing system for rules and regulations documents. For example... Figure 2 As shown, the system includes the following functional modules, each of which can be deployed on the same server or in a distributed cluster and communicate via API or message queue: Document parsing module: responsible for loading electronic documents in formats such as PDF and Word, performing paragraph-level parsing, and outputting a sequence of text paragraphs.

[0096] Feature extraction module: Receives text paragraphs and extracts layout style features, TFIDF keyword features, and pre-trained word vector semantic features.

[0097] Small model processing module: Integrates a trained XGBoost classifier to predict paragraph type labels based on mixed feature vectors, generating the first structured data.

[0098] Rule correction module: Based on preset business rules, it forcibly corrects the labels output by the small model to improve the accuracy of key structural element recognition.

[0099] Semantic aggregation module: Aggregates text fragments based on paragraph type tags, ensuring that their semantics are complete and their length is compliant.

[0100] Large model parsing module: responsible for constructing guided prompts, calling the large language model API, and performing structure validation; if validation fails, it triggers fallback logic.

[0101] Results output module: Summarizes the structured results of all fragments, generates a JSON array in a uniform format, and outputs it.

[0102] In a typical deployment scenario, the system runs as a microservice on a Linux server, using the Python Flask framework to build a RESTful API. The small model processing module runs on the local CPU, achieving millisecond-level response times; the large model parsing module calls cloud-based large model services via HTTPS. The entire system supports concurrent processing, with a single node capable of processing hundreds of policy documents per hour.

[0103] Furthermore, the present invention can also be implemented as a computer-readable storage medium. This storage medium (such as a solid-state drive, USB flash drive, or cloud storage object) stores a computer program, which, when executed by the processor of a server or terminal device, enables the aforementioned method. This software distribution format facilitates the integration of the present invention into existing enterprise compliance management platforms, document middleware, or AI application marketplaces, significantly lowering the technological implementation threshold.

[0104] In one embodiment, a method for intelligently structuring regulatory documents is provided, wherein the small model processing module and the large model parsing module adopt an asynchronous pipeline architecture. The small model performs preprocessing and classification of the entire text paragraphs on the local CPU, while the large model only calls the cloud for the aggregated text fragments, thereby reducing the frequency of large model API calls by more than 70%.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] To verify the technical effectiveness of this invention, we conducted a comparative experiment on a test set containing 500 real corporate policy documents (covering company bylaws, human resource policies, financial regulations, etc.). The experimental environment consisted of an Intel Xeon Gold 6248R CPU, 64GB of memory, and an NVIDIA A10 GPU. The large language model used was GPT-4-Turbo (128K context).

[0107] Experimental results show that: Semantic integrity: The semantic aggregation mechanism of this invention successfully retained 98.7% of the clauses and their sub-items within the same text segment, while the retention rate of traditional fixed-length segmentation methods was only 63.2%. Parsing accuracy: In the extraction of the key fields "clause number" and "main text content", the overall accuracy of this invention reached 96.4%, which is significantly better than the 82.1% achieved by using only a large model for end-to-end parsing; System robustness: After the introduction of human interference (such as the insertion of garbled characters or the deletion of punctuation), this invention can still guarantee 100% output of structured data due to its fallback mechanism, of which 87.5% are high-quality parsing results and the remaining 12.5% ​​are reliable fallback results with fallback flags; while the pure large model scheme has 5.8% of requests that fail completely under the same conditions, returning empty or incorrect format.

[0108] In summary, this invention, through a three-tiered collaborative architecture of "small model pre-screening + large model detailed analysis + intelligent backtracking," not only effectively solves the core pain points in the structured parsing of long and complex documents, but also demonstrates superior performance, accuracy, and stability in practical applications. Those skilled in the art can make conventional adjustments to model selection, feature combinations, rule thresholds, etc., based on this embodiment without departing from the core concept of this invention; all such adjustments should fall within the protection scope of this invention.

[0109] Finally, it should be noted that the above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for intelligent structured processing of rules and regulations documents, characterized in that, Includes the following steps: Obtain the electronic document of rules and regulations to be processed, and perform paragraph-level parsing on the electronic document to obtain multiple text paragraphs, wherein each text paragraph is a continuous text unit separated by the layout or line breaks of the electronic document; For each text paragraph, extract the text content and layout style features; For each text paragraph, the text content and the typesetting style features are input into a pre-trained small model, and the small model outputs first structured data. The first structured data includes at least the original text content of the text paragraph and paragraph type labels. The paragraph type labels include at least one of chapter title, main clause, sub-clause, and body paragraph. Based on the paragraph type tags, the text paragraphs arranged sequentially in the order of the electronic document are aggregated to form multiple text fragments. Each text fragment contains semantically coherent and structurally complete institutional content, and the character length of the text fragment does not exceed the single input length limit of the large language model. The text content of each text fragment and its corresponding paragraph type label are input into the large language model, which generates second structured data, and the second structured data is used as the structured result of the text fragment. If the second structured data does not conform to the preset structure specification, then the structured result of the text fragment is generated based on the first structured data corresponding to each text paragraph in the text fragment; Summarize the structured results of all text fragments and output structured rules and regulations data in a unified format.

2. The method as described in claim 1, characterized in that, The typesetting style features include at least two of the following: paragraph indentation, font size, whether it is centered, and whether it is bold.

3. The method as described in claim 1, characterized in that, The input features of the small model also include keyword features extracted based on the term frequency-inverse document frequency (TFIDF) algorithm, and / or semantic features generated based on a pre-trained word vector model.

4. The method as described in claim 1, characterized in that, After the small model outputs the structured data of the text paragraph, the method further includes the following steps: If the text paragraph meets the preset business rule conditions, its paragraph type label will be forcibly corrected to the specified type; The business rule conditions include at least one of the following: The text content begins with "Article X", "Section X" or "Item X" and does not end with a period; The paragraph corresponding to the text content is displayed in the center, with the font bold, and the number of characters is less than 30; The text content is located in the first 10% of the document and contains the words "Table of Contents" or "Contents".

5. The method according to claim 1, characterized in that, The aggregation of text paragraphs arranged sequentially in the electronic document order from among the multiple text paragraphs based on the paragraph type tags includes: Starting with a text paragraph labeled "Main Clause", aggregate subsequent text paragraphs labeled "Sub-clause" or "Body Paragraph" into the same text segment until the next text paragraph labeled "Main Clause" or "Chapter Title" is encountered.

6. The method according to claim 1, characterized in that, The step of inputting the text content of each text segment and its corresponding paragraph type label into the large language model includes: Convert all paragraph type tags in the text fragment into guiding statements according to a preset template; The guiding statement is appended before the text content of the text fragment as an input prompt for the large language model.

7. The method according to claim 1, characterized in that, The preset structural specifications include: The second structured data is a string that conforms to JSON syntax, and the data structure obtained after parsing contains at least one of the "clause number" field or "main text content" field, and the value of the field is not empty.

8. The method according to claim 1, characterized in that, The structured result generated from the text fragment based on the first structured data is a JSON object containing the following fields: The rollback flag field, whose value is true; The paragraph list field contains the paragraph type label and original text content of each text paragraph in the text fragment.

9. An intelligent structured processing system for rules and regulations documents, characterized in that, include: The document parsing module is used to parse electronic documents of rules and regulations at the paragraph level, resulting in multiple text paragraphs; The feature extraction module is used to extract the text content and layout style features of each text paragraph; The small model processing module is configured with trained small models to output the first structured data for each text paragraph; The rule correction module is used to correct paragraph type tags according to preset business rules; The semantic aggregation module is used to generate multiple text fragments based on paragraph type tags; The large model parsing module is used to call the large language model to generate the second structured data, and generate a structured result based on the first structured data when the structure verification fails; The results output module is used to summarize all structured results and output structured rules and regulations data in a unified format.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images