A standardized data chunking processing system supporting multi-source data formats

Abstract

The application discloses a kind of standardization data block processing systems of supporting multi-source data format, it is related to data processing field, the system includes preprocessing and format uniformity module, intelligent pre-analysis and strategy selection module, mixed inference depth block engine, block result optimization and check module, structured data output module.The application supports the compatible processing of multi-source data format;Based on the dynamic division of chapter structure and logic unit of text content, ensure semantic association content, avoid cross-modal semantic fragmentation;Improve the universality of block strategy;Intelligent block of multi-source data is realized, significantly reduces the demand of manual intervention, improves processing speed and efficiency, especially suitable for large-scale document processing scene;It can be directly connected knowledge graph construction, intelligent retrieval and other downstream applications, significantly improve data utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a standardized data block processing system that supports multi-source data formats. Background Technology

[0002] In the development of information technology, data is of paramount importance. During data preprocessing, dividing data into modules is beneficial for subsequent steps such as feature extraction. However, most current data segmentation algorithms divide modules based on a defined number of tokens, which can lead to semantic segmentation of semantically related content, resulting in insufficient information extraction in subsequent steps. Specifically:

[0003] 1. Semantic fragmentation and breakage of key information: Cutting text into a fixed number of tokens, such as dividing it into modules of 500 words, will forcibly truncate complete semantic units, such as a paragraph, a technical clause, or an event description.

[0004] 2. Loss of contextual relevance and increased risk of semantic ambiguity: When semantically related content is divided into different modules, it loses contextual support, which may lead to ambiguity. The extracted indicator object attributes are disconnected from the actual content, resulting in inaccurate structured data.

[0005] 3. Poor domain knowledge adaptability and neglect of industry characteristics: Documents from different domains, such as medical, chemical and IT, have unique semantic structures and professional terminology systems. The fixed number of tokens cannot adapt to industry characteristics.

[0006] 4. Low data utilization efficiency and increased invalid calculations: Irrelevant content is grouped into the same module, such as forcibly merging two paragraphs with unrelated topics, which leads to the need to process redundant information during extraction and wastes computing resources.

[0007] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0008] In view of the problems in related technologies, this invention proposes a standardized data block processing system that supports multi-source data formats, so as to overcome the above-mentioned technical problems existing in the existing related technologies.

[0009] Therefore, the specific technical solution adopted by the present invention is as follows:

[0010] A standardized data chunking processing system supporting multi-source data formats includes:

[0011] The preprocessing and format unification module is used to extract text information from multiple source documents and convert it into standardized format text that contains the original document structure hierarchy, element relationships and meta information;

[0012] The intelligent pre-analysis and strategy selection module is used to obtain the document type of standardized format text and obtain the chunking strategy package based on the document type and the strategy library matching mechanism.

[0013] The hybrid reasoning deep chunking engine is used to perform semantic understanding of standardized format text using a pre-trained language model and output facts; based on the facts, it combines chunking strategy packages to perform logical reasoning and obtain preliminary chunking results.

[0014] The block result optimization and verification module is used to adjust and verify the preliminary block results based on the isolated content merging and boundary checking mechanism to obtain the block results;

[0015] The structured data output module is used to output the content of the block results in a structured data format.

[0016] Furthermore, the preprocessing and format unification module includes a parsing module, a hierarchical structure conversion module, and an associated tag module;

[0017] The parsing module is used to obtain the document's geometric layout information and page layout information, and to perform header and footer removal, multi-column layout reordering, and table reconstruction.

[0018] The hierarchical structure conversion module is used to identify heading levels using heuristic rules or classification models and convert them into standardized heading tags; it also converts lists into corresponding standardized syntax.

[0019] The association tag module is used to identify related content pairs in a document and associate them with non-text elements through placeholders.

[0020] Furthermore, document types for obtaining standardized formatted text include:

[0021] Scan standardized format text to obtain key features for classification;

[0022] Based on key features used for classification, and through keyword matching, metadata analysis, and statistical analysis of title structure, the document types of standardized format texts are divided using a classification model.

[0023] Furthermore, based on the document type and strategy library matching mechanism, the resulting chunking strategy package includes:

[0024] Based on the document type's type tag, match the corresponding chunking strategy from the strategy library;

[0025] If a match is successful, the segmentation strategy is organized into a segmentation strategy package and sent to the hybrid inference deep segmentation engine.

[0026] Furthermore, the policy library includes document type tags and policy objects: policy objects include:

[0027] A set of logical rules used to provide semantic integrity standards for document content;

[0028] Regular expression template set for identifying key structural boundaries in a document;

[0029] A list of domain keywords, used to provide standard keywords for the domain.

[0030] Furthermore, based on the facts and using logical reasoning combined with the segmentation strategy package, the preliminary segmentation results include:

[0031] The reasoning engine matches facts with rules in the logical rule base and continues reasoning until it finds all the most stable and semantically complete block boundaries that satisfy the logical rules.

[0032] Based on the decision path of the block boundaries, preliminary block division results are obtained.

[0033] Furthermore, the construction of logical rules includes:

[0034] Construct a corpus containing multiple source formats; process the documents in the corpus into a standardized format and extract structural, content, and relational features; transform the structural, content, and relational features into facts to obtain a fact database;

[0035] Based on the fact database, a set of logical rules is derived to maximize the coverage of all positive instances in the corpus while minimizing negative instances.

[0036] Furthermore, the hybrid inference deep chunking engine includes a semantic analysis model, and uses this model to label each part of the text, including:

[0037] A labeling system is constructed, including preconditions, conclusions, method descriptions, and object definitions. Based on the labeling system, samples extracted from the target document are labeled to obtain a labeled dataset.

[0038] The labeled dataset is input into the pre-trained language model. The loss between the probability distribution predicted by the pre-trained language model and the labeled true labels is calculated. The backpropagation algorithm is used to update the classification head and parameters of the pre-trained language model to obtain the semantic analysis model.

[0039] The semantic analysis model outputs the label for each part of the text.

[0040] Furthermore, the block result optimization and verification module includes an isolated content merging module and a boundary check module;

[0041] The isolated content merging module is used to obtain isolated blocks from the initial segmentation results; based on the semantic similarity between the isolated blocks and adjacent blocks, the isolated blocks are merged into other blocks.

[0042] The boundary check module is used to check the block boundaries in the initial block division results and make fine adjustments.

[0043] Furthermore, based on the semantic similarity between isolated blocks and adjacent blocks, merging isolated blocks into other blocks includes:

[0044] Traverse the results of the initial block segmentation, identify isolated blocks based on a preset length threshold, and obtain the adjacent blocks of the isolated blocks.

[0045] A pre-trained sentence embedding model is used to convert the text content of isolated blocks and adjacent blocks into high-dimensional semantic vectors, respectively.

[0046] Based on the high-dimensional semantic vector, cosine similarity is used to calculate the semantic association between isolated blocks and adjacent blocks;

[0047] Based on semantic relevance and a preset merging threshold, isolated blocks are merged with adjacent blocks, and the block list is updated.

[0048] The beneficial effects of this invention are as follows:

[0049] 1. Standardization and scalability advantages: This invention supports compatible processing of multiple data formats, effectively solving the problem of format barriers causing the division rules to fail in the existing technology, and can seamlessly adapt to various input formats such as text PDF and DOCX.

[0050] 2. Significantly improved semantic integrity: This invention dynamically divides text content based on chapter structure and logical units to ensure the complete preservation of semantically related content, such as paragraphs, clauses, and indicator systems; at the same time, it supports multimodal data collaborative processing, prioritizing the inclusion of related content such as table row data and formula context into the same module to avoid cross-modal semantic fragmentation.

[0051] 3. Enhanced Domain Adaptability: This invention uses a regular expression rule engine to parse document content and combines it with industry-specific format specifications for intelligent block segmentation, enabling the system to flexibly adapt to document structures in multiple fields such as chemical engineering, medical care, and IT, thereby improving the universality of the block segmentation strategy.

[0052] 4. Improved automation and processing efficiency: This invention achieves intelligent segmentation of multi-source data, significantly reducing the need for manual intervention and improving processing speed and efficiency, making it particularly suitable for large-scale document processing scenarios.

[0053] 5. Empowering Information Reuse and In-Depth Analysis: This invention provides a standardized foundation for the secondary use, in-depth analysis, and knowledge extraction of industry-standard data by converting document content into a unified Markdown format and dividing it into modules. This structured output can be directly connected to downstream applications such as knowledge graph construction and intelligent retrieval, significantly improving data utilization efficiency. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a technical flowchart according to an embodiment of the present invention;

[0056] Figure 2 This is a block diagram of a standardized data block processing system that supports multi-source data formats according to an embodiment of the present invention.

[0057] In the picture:

[0058] 1. Preprocessing and format unification module; 2. Intelligent pre-analysis and strategy selection module; 3. Hybrid inference deep segmentation engine; 4. Segmentation result optimization and verification module; 5. Structured data output module. Detailed Implementation

[0059] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0060] According to embodiments of the present invention, a standardized data segmentation processing system supporting multi-source data formats is provided. This system is a computer data preprocessing technology that includes natural language processing, multi-source data fusion, and intelligent segmentation algorithms.

[0061] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 2 As shown, a standardized data chunking processing system supporting multi-source data formats according to an embodiment of the present invention includes:

[0062] The preprocessing and format unification module 1 is used to extract text information from multiple source documents and convert it into standardized format text that includes the original document structure hierarchy, element relationships and meta information.

[0063] The intelligent pre-analysis and strategy selection module 2 is used to obtain the document type of standardized format text and obtain the chunking strategy package based on the document type and the strategy library matching mechanism.

[0064] Hybrid reasoning deep segmentation engine 3 is used to perform semantic understanding of standardized format text using a pre-trained language model and output facts; based on the facts, it combines segmentation strategy packages to perform logical reasoning and obtain preliminary segmentation results.

[0065] The block result optimization and verification module 4 is used to adjust and verify the preliminary block results based on the isolated content merging and boundary checking mechanism to obtain the block results.

[0066] The structured data output module 5 is used to output the content of the block results in a structured data format.

[0067] In one embodiment, the preprocessing and format unification module 1 includes a parsing module, a hierarchical structure conversion module, and an association tag module;

[0068] The parsing module is used to obtain the document's geometric layout and page layout information, and to remove headers and footers, reorder multi-column layouts, and reconstruct tables; the hierarchical structure conversion module is used to identify heading levels using heuristic rules or classification models and convert them into standardized heading tags; it also converts lists into corresponding standardized syntax; the association tag module is used to identify related content pairs in the document and associate them with non-text elements through placeholders.

[0069] In one embodiment, the document types for obtaining standardized formatted text include:

[0070] Standardized format texts are scanned to obtain key features for classification. Based on these key features, the document types of standardized format texts are classified through keyword matching, metadata analysis, statistical analysis of title structure, and a classification model.

[0071] In one embodiment, the chunking strategy package obtained based on document type and strategy library matching mechanism includes:

[0072] Based on the document type's type tag, the corresponding chunking strategy is matched from the strategy library; when a match is successful, the chunking strategies are organized into a chunking strategy package and sent to the hybrid inference deep chunking engine.

[0073] In one embodiment, the policy library includes document type tags and policy objects: the policy objects include: a set of logical rules for providing semantic integrity standards for document content; a set of regular expression templates for identifying key structural boundaries in the document; and a list of domain keywords for providing domain standard keywords.

[0074] In one embodiment, based on facts and combined with logical reasoning using a segmentation strategy package, a preliminary segmentation result is obtained, including:

[0075] The reasoning engine matches facts with rules in the logical rule base and continues reasoning until it finds all the most stable and semantically complete block boundaries that satisfy the logical rules; based on the decision path of the block boundaries, the preliminary block results are obtained.

[0076] In one embodiment, the construction of logical rules includes:

[0077] Construct a corpus containing multiple source formats; process the documents in the corpus into a standardized format and extract structural, content, and relational features; transform the structural, content, and relational features into facts to obtain a fact database; based on the fact database, summarize a set of logical rules that maximizes the coverage of all positive instances in the corpus while minimizing negative instances.

[0078] In one embodiment, the hybrid inference deep chunking engine 3 includes a semantic analysis model, and the semantic analysis model is used to label each part of the text, including:

[0079] A labeling system is constructed, including preconditions, conclusions, method descriptions, and object definitions. Based on the labeling system, samples extracted from the target document are labeled to obtain a labeled dataset. The labeled dataset is input into a pre-trained language model. The loss between the probability distribution predicted by the pre-trained language model and the labeled true labels is calculated. The backpropagation algorithm is used to update the classification head and parameters of the pre-trained language model to obtain a semantic analysis model. The semantic analysis model outputs the labels for each part of the text.

[0080] In one embodiment, the block result optimization and verification module 4 includes an isolated content merging module and a boundary checking module;

[0081] The isolated content merging module is used to obtain isolated blocks in the preliminary segmentation results; based on the semantic similarity between the isolated blocks and adjacent blocks, the isolated blocks are merged into other blocks; the boundary checking module is used to check the block boundaries in the preliminary segmentation results and make fine adjustments.

[0082] In one embodiment, merging isolated blocks into other blocks based on semantic similarity between isolated blocks and adjacent blocks includes:

[0083] The system iterates through the initial block segmentation results, identifies isolated blocks based on a preset length threshold, and obtains the immediate neighboring blocks of each isolated block. A pre-trained sentence embedding model is used to convert the text content of both isolated and immediate neighboring blocks into high-dimensional semantic vectors. Based on these high-dimensional semantic vectors, cosine similarity is used to calculate the semantic correlation between isolated blocks and their neighboring blocks. Based on the semantic correlation and a preset merging threshold, isolated blocks are merged with their neighboring blocks, and the block list is updated.

[0084] To facilitate understanding of the above technical solutions of the present invention, the working principle or operation method of the present invention in actual process will be described in detail below.

[0085] This invention proposes a standardized data chunking system that supports multi-source data formats. Through text format conversion and regular expression rule writing, it achieves chapter-level division based on the actual text content, ensuring content integrity. It utilizes an adaptive standardized data chunking technology based on a hybrid inference engine that supports multi-source data formats.

[0086] This invention uses a multi-format parsing and conversion module to uniformly convert heterogeneous documents such as PDF and DOCX into structured Markdown format. Subsequently, an intelligent pre-analysis and strategy selection module 2 classifies the macro-structure and type of the documents. Most importantly, this invention introduces an intelligent segmentation engine that combines logical reasoning and deep learning. This engine integrates the interpretability of formal logic rules with the deep semantic understanding capabilities of neural networks, performing semantically coherent, domain-aware, and even proposition-level intelligent segmentation of Markdown text according to pre-selected strategies. This invention aims to solve the core pain points of traditional segmentation technologies, such as semantic fragmentation, context loss, and poor domain adaptability, achieving high-precision, high-efficiency, and highly interpretable automated document segmentation processing. Details of the technical solution are as follows... Figure 1 As shown, it includes a multi-source input and format standardization engine; it extracts text and structural tags through a PDF parser, extracts style hierarchy through a DOCX parser, and provides a unified intermediate representation: XML structured data. The semantic chunking engine includes a chapter recognition module, a dynamic chunking algorithm, a cross-modal relationalizer, and structured output: Markdown chunks.

[0087] I. Preprocessing and Format Unification Module 1:

[0088] Objective: To resolve the issue of inconsistent source file formats and provide standardized input for subsequent processing. Implementation: Utilize the python-docx library to parse .docx files, extracting text, headings, tables, lists, and other elements. Use the pdfplumber or PyMuPDF library to process .pdf files. This step not only extracts text but also leverages PDF metadata and layout information to initially identify the positions of headings, paragraphs, tables, and images, handling complex situations such as scanned PDFs and two-column layouts. Call the pypandoc conversion tool to uniformly convert the parsed content into a Markdown format rich in metadata. For example, during conversion, heading levels are preserved (e.g., # represents level 1 heading, ## represents level 2 heading), tables are converted to Markdown table syntax, and unique identifiers and contextual descriptions are added to non-text elements such as images and formulas. The preprocessing and format unification module is used to achieve the transformation from simple extraction to rich metadata. The core innovation lies in the fact that the output of this module is not plain text, but a standardized format (Markdown) rich in metadata that preserves the original document structure hierarchy and element relationships. This is reflected in the following three aspects:

[0089] (1) Innovation Point 1: Deep PDF Parsing Based on Layout Analysis

[0090] Traditional PDF-to-text tools typically extract text in reading order, which is disastrous for documents with complex layouts. The innovation of this invention lies in: using advanced PDF parsing libraries: leveraging modern Python libraries such as pdfplumber or PyMuPDF, which can access the geometric layout information within the PDF (x, y coordinates of each character, line, and rectangle). Intelligent document element recognition: the system doesn't just read text, but "understands" the page layout by analyzing coordinates and font information. Header and footer removal: by setting fixed areas at the top and bottom of the page, or identifying text that appears repeatedly across all pages, it automatically identifies and removes these as headers and footers, avoiding interference with the main text. Multi-column layout reordering: by recognizing the boundary coordinates of columns, the system correctly reads the first column before the second, ensuring the correct reading order. Table reconstruction: this is particularly crucial. The system doesn't treat table content as scattered lines of text, but rather reconstructs the table's row and column structure in memory by analyzing lines in the PDF—explicit table lines or text alignment coordinates—and then converts it to Markdown table syntax. This perfectly preserves the two-dimensional relationship of the data.

[0091] Comparison of results: Traditional method yields: Parameter A value A, Parameter B value B (structure lost). This invention yields: |Parameter| value||---|---||Parameter A| value A||Parameter B| value B| (structure preserved).

[0092] (2) Innovation point two: Preserving the complete hierarchical structure during conversion:

[0093] The chapter structure of a document is the skeleton for understanding its content logic. The innovation of this module lies in fully transferring this skeleton to Markdown.

[0094] Heading Level Identification. When processing DOCX files, the system directly reads style information such as Heading 1, Heading 2, etc. When processing PDF files, it uses features such as font size, bolding, whether it occupies a single line, and numbering style (e.g., Chapter 1, 1.1, (a)) to determine the heading level of a text line using heuristic rules or a small classification model. Mapping to Markdown Syntax. The identified heading structure is accurately converted into Markdown heading tags. For example: Heading 1 -> #Heading 1; Heading 2 -> ##Heading 2; Heading 3 -> ###Heading 3. Preserving List Structure. Both ordered lists with numbers and unordered lists with dots are recognized and converted into the corresponding Markdown syntax, such as 1., *.

[0095] (3) Innovation point three: context association and non-text element marking: This is a key step in ensuring semantic integrity and a significant innovation point.

[0096] Entity Binding. The system actively identifies content pairs with strong relationships. For example, using regular expressions or layout proximity analysis, it finds table titles (usually above the table) and image captions (usually below the image) and binds them to the corresponding tables / images. Non-Text Element Placeholders and Descriptions. For content such as images and formulas that cannot be directly converted to text, the system does not simply discard them. Instead, it inserts a unique placeholder at the corresponding position in the Markdown text flow, such as ![Image_001]$$Formula_002$$. The element's metadata is associated with this placeholder; the metadata includes the context text of the caption or formula. The key to this step is the conversion rich in metadata. Unlike simple text copying, the conversion process of this invention preserves the original structural hierarchy of the document, which is the basis for subsequent intelligent chunking.

[0097] II. Intelligent pre-analysis and strategy selection module 2, namely adaptive concretization.

[0098] Objective: To answer the question of how to automatically adjust the segmentation strategy based on different document characteristics, achieving adaptive processing. Implementation: Document Type Recognition: The system first performs a rapid scan and analysis of the converted Markdown text. Through keyword matching, metadata analysis, and statistical analysis of the title structure, a lightweight classification model is used to automatically classify the document into preset categories such as industry standards and standard implementation guidelines. Keyword matching includes keywords such as technical specifications and test reports; title structure includes the presence of 1. Scope, 2. Normative references, and 3. Terms and definitions; and lightweight classification models include Naive Bayes or decision trees.

[0099] (1) 1. Input: Receive Markdown text rich in metadata and structured tags from step 1. 2. Analysis and Feature Extraction: The system does not immediately begin chunking, but instead performs a rapid macro-scan of the entire document to extract key features for classification. These features include: Keyword features: Scanning the text for specific high-frequency words that indicate its identity. For example, technical specifications, clinical trials, end-user license agreements, etc. Structural features: Analyzing the heading structure in the Markdown text. For example, the system will count whether there are specific heading sequences, such as 1. Scope, 2. Normative references, 3. Terms and definitions, which are typical structures of standard documents. For legal documents, it will look for patterns such as Article 1, Section X, etc. Metadata features: If metadata such as author, company, and title is parsed from the source file, such as DOCX, it will also be used as a reference for classification. 3. Perform Classification: The system inputs the extracted features into a pre-trained lightweight classification model, such as Naive Bayes, decision tree, or support vector machine. This model is trained to quickly classify the document into a preset category based on the input features. 4. Output: Output a clear document type label, such as DocType:National_Standard, DocType:Medical_Standard, etc. This label will serve as input for the next step.

[0100] (2) Policy Library Matching and Dynamic Loading: Receives the document type tag output from the previous step, such as DocType:National_Standard. Queries the policy library. The system uses this type tag as a key to search for the corresponding lock in the core policy library, which is the chunking strategy tailored for this document type. Loads the strategy. Once a match is successful, the system loads this complete strategy, including a series of specific logical rules, regular expression templates, and keyword lists, from the policy library into memory. The loaded chunking strategy package is then passed to the hybrid inference deep chunking engine in the third step. This way, the chunking engine knows which specific rules and standards should be followed when processing this document.

[0101] Method for constructing a policy library: The policy library is the core of this module and an aggregation of the system's knowledge and experience. Its construction is a systematic project that combines "expert knowledge" and "data-driven".

[0102] 1. Structure of the policy library: Essentially, the policy library is a mapping database or a structured configuration file. Each of its records contains a key and a value. Key: Document type label, such as National_Standard. Value: A policy object containing complete chunking instructions. A policy object usually consists of the following three parts:

[0103] Logical rule set. This is prepared for the inference decision engine in the third step. These rules define the semantic integrity criteria of the content. For example: Rule A: For the National_Standard type, a text block containing three parts: test equipment, test steps, and result determination must be divided into the same chunk without splitting. Rule B: For the Legal_Contract type, clauses defining rights and obligations must be in the same chunk as their qualifying conditions, such as under... circumstances. Regular expression template set. Used to quickly identify key structural boundaries in the document. For example: Title regular expression: ^第\s*[一二三四五六七八九十百]+\s*章.* Used to identify chapter titles in standards. Clause regular expression: ^第\s*\d+\s*条.* Used to identify clauses in legal documents. Domain keyword list. Used to assist the semantic analysis model in more accurately understanding the content. For example: Chemical standard keywords: CAS number, molecular formula, melting point, boiling point. Medical record keywords: Chief complaint, current history, past history, physical examination.

[0104] 2. Construction process of the policy library. The construction of the policy library is mainly carried out in the following ways: Expert-driven construction. This is the main way for the initial construction of the policy library and the definition of core rules. Domain division. First, determine the document domains that the system needs to support, such as chemical industry, law, medicine, IT, etc. Expert interviews and analysis. Invite experts in the corresponding fields, such as standard-setting engineers, lawyers, doctors, to analyze the structural characteristics and reading habits of typical documents in that field. Rule formalization. Collaborate with domain experts by knowledge engineers to transform the expert's judgment logic and analysis methods into machine-readable logical rules, regular expressions, and keyword lists. For example, the lawyer pointed out that the definition clauses in a contract are usually concentrated in Article 2, and each definition is an independent point. The knowledge engineer transforms it into: Policy(Legal_Contract)->Logical rule: IF section_title is "定义", THEN split by each numbered item. Input and testing. Input the formalized policies into the policy library and use typical documents in that field for testing and iterative optimization.

[0105] III. Hybrid Inference Deep Block Engine 3. Engine Components:

[0106] (1) Logical Rule Base: Basic Principles: This is a formal knowledge base that uses a logic programming paradigm similar to Prolog or Datalog to store a large number of IF-THEN rules regarding document structure and content logic. These rules are predefined by domain experts or derived inductively from a large number of chunked documents through machine learning. Specifically, it is a dynamic rule base construction method that combines machine learning with expert knowledge. This method uses an inductive logic programming engine, designed to automatically learn and summarize general, formalized chunking logic rules from a large number of example documents that have been correctly chunked by experts. The entire process strictly follows the following steps:

[0107] The first step is to construct and factualize the training corpus:

[0108] Corpus Construction. First, the system collects and constructs a training corpus containing multiple source formats, such as DOCX and PDF. All documents in this corpus are pre-segmented into high-quality, manually segmented text by domain experts, i.e., standard-setting engineers, serving as the standard answer for machine learning. Feature Extraction and Factification. The system calls the preprocessing and format unification module to uniformly process all documents in the corpus into a Markdown format rich in meta-information. Subsequently, deep analysis is performed on these structured texts to extract their structural, content, and relational features, and these features are transformed into logical facts that the ILP engine can recognize. The transformed facts include, but are not limited to:

[0109] Structural facts, such as `is_title(X,level=2)`, indicate that X is a second-level heading; `is_list_item(Y)` indicates that Y is a list item. Content facts, such as `contains(X,"Experimental Equipment")` and `contains(X,"Result Determination")`, indicate that text block X contains specific keywords. Relational facts, such as `follows(Table_A,Para_B)`, indicate that table A immediately follows paragraph B. Target facts, most importantly, record the attribution relationships between elements based on the expert's chunking results, such as `in_same_chunk(Para_B,Table_A)`, indicating that paragraph B and table A belong to the same chunk in the standard answer.

[0110] The second step is to summarize the learning task definition and engine startup:

[0111] Given a fact database, a set of IF-THEN rules is derived that maximizes coverage of all positive instances in the corpus (in_same_chunk) while minimizing erroneous coverage of negative instances. The engine starts, and the inductive logic programming engine used in this invention initiates the learning process. The engine takes the fact database generated in the first step as input and outputs high-quality chunking rules.

[0112] The third step is the generation, testing, and generalization of rule hypotheses:

[0113] Hypothesis generation. The ILP engine systematically combines facts from the fact database using a bottom-up or top-down search strategy to generate candidate IF-THEN rule hypotheses. For example, the engine might generate the following hypotheses:

[0114] "chunk_together(X,Y):-is_paragraph(X),is_table(Y),follows(Y,X),contains(X,"as shown in the table below")". Hypothesis testing. For each generated hypothesis rule, the engine performs backtesting across the entire fact database, calculating its support and confidence. Generalization and specialization. To improve the generality of the rules and avoid overfitting, the engine generalizes overly specific rules and specializes overly broad rules to find the optimal rule expression. Support refers to the frequency with which the rule applies in the dataset, and confidence refers to the accuracy of the rule's predictions.

[0115] The fourth step is rule selection, formalization, and integration:

[0116] Rule selection. After the learning process, only rule hypotheses whose support and confidence both exceed preset thresholds are accepted and become formal inductive rules. Formalization and integration. The system automatically converts the selected qualified rules into formal logical statements that are exactly the same as the expert-defined rules, and dynamically supplements and integrates them into the logical rule base.

[0117] Example of a rule:

[0118] Rule 1: `chunk_together(Para,Table):-follows(Table,Para),contains(Para,"As shown in the table below")` (If a paragraph contains quotations such as "As shown in the table below," and a table immediately follows, then the paragraph and table should be grouped into one block). Rule 2: `is_definition_block(Section):-has_title(Section,"Terminology and Definitions")` (If the title of a section is "Terminology and Definitions," then each numbered entry under that section should be considered an independent, minimal semantic block). Rule 3: `is_complete_test(Chunk):-contains(Chunk,"Experimental Equipment"),contains(Chunk,"Experimental Procedures"),contains(Chunk,"Result Determination")` (If a content block contains all three elements—experimental equipment, procedures, and result determination—it is considered a complete and indivisible experimental method unit). Rule 1 is a table association rule, Rule 2 is a definition unit rule, and Rule 3 is an experimental method integrity rule.

[0119] (2) Deep Semantic Analysis Model: Basic Principle: It uses a pre-trained language model, namely BERT, to perform deep semantic understanding of text. Its role is not to directly divide the text into blocks, but to label each part of the text, such as sentences and paragraphs, to provide input facts for logical reasoning. This model is built on a large-scale pre-trained language model, such as BERT and its derivative models. Through pre-training on massive amounts of text, the model has acquired a deep general language understanding capability and can convert the input text sequence into a vector representation rich in contextual semantic information.

[0120] Task Definition and Data Labeling: Define a clear labeling system based on business requirements. In the sentence / paragraph functional classification task, labels include is_premise, is_conclusion, is_method, and is_object. A batch of sample sentences or paragraphs extracted from target documents, such as industry standards, are manually labeled according to the predefined labeling system. This is the supervised learning data necessary for training the model.

[0121] Model fine-tuning steps: Take the text unit to be labeled, such as a paragraph, as input. Add special markers, such as [CLS] and [SEP], to the beginning and end of the text according to the requirements of the BERT model, and convert it into an input ID sequence acceptable to the model. Add a task-specific classification head on top of the pre-trained BERT model. This head is usually a simple fully connected neural network layer. Input the labeled dataset into the model. The BERT model is responsible for encoding the text content into a high-dimensional semantic vector, usually taking the output vector corresponding to the [CLS] marker as the aggregate representation of the entire text. This semantic vector is fed into the top classification head, which outputs a probability distribution calculated using the Softmax function. This distribution represents the probability that the input text belongs to each predefined label. Calculate the loss between the model's predicted probability distribution and the manually labeled true labels, such as cross-entropy loss, and use the backpropagation algorithm to update the parameters of the classification head and the BERT model itself. This process is fine-tuning, which adapts a general language model to a specific labeling task.

[0122] The reasoning and application, or labeling steps, involve fine-tuning the model to assign semantic labels to new text. When a new paragraph is input, the model performs a forward propagation and outputs the label with the highest probability as the final semantic function label for that paragraph. These generated labels, such as is_method(Paragraph_5), are provided as facts to the downstream inference decision engine.

[0123] Functions: Semantic role labeling. Identifies the function of a sentence or paragraph, such as is_premise (a precondition), is_conclusion (a conclusion), is_method (a method description), and is_object (an object definition). Entity recognition. Extracts key technical entities, such as experimental methods, performance indicators, and chemical substances. Relationship extraction. Determines the relationships between different sentences or paragraphs, such as causality, elaboration, and contradiction.

[0124] (3) Reasoning Decision Engine, Basic Principle: This is the brain of the entire segmentation engine. It receives facts from the semantic analysis model and target policies from the policy selection module, i.e., activated logical rules, executes the logical reasoning process, and finally makes segmentation decisions.

[0125] The workflow includes: Fact Generation. Markdown text is fed into a semantic analysis model to generate semantic tags and relational descriptions for each paragraph, sentence, and table. For example, the facts obtained are: is_method(Paragraph_5), is_object(Paragraph_6), elaboration(Paragraph_6, Paragraph_5). Rule Matching and Inference. The inference engine matches these facts against rules in a logical rule base. For example, the engine finds that Paragraph_5 describes a method, and Paragraph_6 describes the object of that method. Based on a rule that methods and objects should be considered as one, the engine generates an intermediate conclusion: group(Paragraph_5, Paragraph_6). Chunking Decision. The engine continues inference until it finds all the most stable and semantically complete chunk boundaries that satisfy the logical rules. Compared to regular expressions, which can only recognize surface patterns, this engine can understand content and segment it logically. Interpretability. Because each chunking decision is based on explicit logical rules, the system can output the decision path. For example, paragraphs 5-8 are grouped together because they trigger the experimental method integrity rule #3. This greatly facilitates manual review and intervention.

[0126] IV. Chunking Result Optimization and Verification Module 4. Objective: To post-process the initial chunking results and correct potential flaws. Implementation Method: Merging isolated content. Check for isolated chunks with insufficient content, such as those containing only one word or sentence. Based on their semantic similarity to adjacent chunks, this can be calculated using vector representation and intelligently merged into the preceding or following chunk. Boundary Check. Check whether chunk boundaries have cut off units that should not be segmented, such as a complete list or a code block, and make fine adjustments.

[0127] The core principle of intelligent processing for merging isolated content is based on semantic similarity calculation using text vectorization, combined with pre-defined decision logic, to determine the optimal allocation of isolated blocks. The specific implementation steps are as follows:

[0128] Identifying Isolated Chunks. The system first iterates through the initial chunking results and identifies isolated chunks based on a preset length threshold, such as fewer than N characters or one sentence. These chunks are potential candidates for merging. Context Acquisition. For each identified isolated chunk, denoted as C_isolated, the system acquires its immediate preceding chunk C_prev and subsequent chunk C_next. Semantic Vectorization. Using a pre-trained sentence embedding model, such as the BERT text representation model, the text content of C_isolated, C_prev, and C_next is converted into high-dimensional semantic vectors V_isolated, V_prev, and V_next, respectively. These vectors encode the semantic meaning of the text in a vector space; semantically similar texts have similar vector distances in this space. Calculating Semantic Similarity. Cosine similarity is used to calculate the semantic association between isolated chunks and adjacent chunks. Cosine similarity measures the directional proximity of two vectors, ranging from -1 to 1; values ​​closer to 1 indicate greater semantic similarity. Calculate the similarity between an isolated block and the previous block: Sim_prev=CosineSimilarity(V_isolated,V_prev); Calculate the similarity between an isolated block and the next block: Sim_next=CosineSimilarity(V_isolated,V_next).

[0129] Intelligent merging decision logic: Comparison and decision. The system makes decisions based on the calculated similarity scores: If Sim_prev > Sim_next, and Sim_prev is greater than a preset merging threshold T, for example, T=0.7, then the isolated chunk is determined to have a closer semantic relationship with the preceding chunk and should be merged forward. If Sim_next > Sim_prev, and Sim_next is also greater than the merging threshold T, then the isolated chunk is determined to have a closer relationship with the following chunk and should be merged backward. If both Sim_prev and Sim_next are below the merging threshold T, then the isolated chunk, although brief, may be an independent semantic unit, such as an important short heading or term; in this case, the system chooses not to merge it to preserve its independence. If the isolated chunk has only one adjacent chunk, i.e., it is at the beginning or end of the document, then only its similarity with the only adjacent chunk needs to be calculated; if it exceeds the threshold T, then it is merged. The merging process is then executed. Based on the decision result, the system concatenates the text content of C_isolated with the selected adjacent block C_prev or C_next, and updates the block list, thus completing an intelligent merging. Through the above steps, the system can avoid incorrectly separating semantically related content that has been temporarily split due to formatting issues, effectively improving the coherence and quality of the final block result.

[0130] V. Structured Data Output Module. Objective: To output consistent and easy-to-use chunked results. Implementation: Output the chunked content in JSON or XML format. Each chunk is an object containing the following fields:

[0131] `chunk_id`: A unique identifier. `source_document`: The source filename. `metadata`: Metadata such as the chapter and document type. `content`: The block content in Markdown format. `Justification`, optional but crucial: A description of the logical rules or decisions used to create this chunk, enhancing the system's transparency and credibility.

[0132] This invention includes: 1. Industry-standard format conversion technology: This invention proposes a Python-based text format conversion technology to standardize the conversion of diverse data formats in the industry-standard domain, such as DOCX and PDF, to a unified format, such as Markdown, providing a consistent foundation for subsequent data processing. 2. Semantic-level intelligent module partitioning method: This invention breaks through the limitations of traditional fixed-length segmentation technology, constructing an upgraded solution from character-level mechanical cutting to semantic-level intelligent partitioning. By recognizing chapter structure, it achieves precise alignment between document content and semantic units, significantly improving the accuracy of segmentation and business adaptability. 3. Adaptive document processing strategy: This invention can automatically adjust the segmentation strategy according to the characteristics and content of different documents to achieve more efficient and accurate text parsing.

[0133] The advantages of this invention are: 1) It addresses the issue of inconsistent industry standard file formats: This invention resolves the problem of inconsistent PDF or DOCX formats for industry standard files; 2) It addresses the problem of incomplete information caused by module segmentation: This invention solves the problem that traditional module segmentation based on a fixed number of tokens forcibly truncates semantically related content, leading to broken key elements during subsequent information extraction and the inability to form a complete semantic chain. Semantically related content includes paragraphs, clauses, and logical units; key elements include indicator objects, attributes, and experimental methods.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A standardized data block processing system supporting multi-source data formats, characterized in that, include: The preprocessing and format unification module is used to extract text information from multiple source documents and convert it into standardized format text that contains the original document structure hierarchy, element relationships and meta information; The intelligent pre-analysis and strategy selection module is used to obtain the document type of standardized format text and obtain the chunking strategy package based on the document type and the strategy library matching mechanism. The hybrid reasoning deep chunking engine is used to perform semantic understanding of standardized format text using a pre-trained language model and output facts; based on the facts, it combines chunking strategy packages to perform logical reasoning and obtain preliminary chunking results. The block result optimization and verification module is used to adjust and verify the preliminary block results based on the isolated content merging and boundary checking mechanism to obtain the block results; The structured data output module is used to output the content of the block results in a structured data format; The strategy library includes document type tags and strategy objects; the strategy objects include: a set of logical rules to provide semantic integrity standards for document content; a set of regular expression templates to identify key structural boundaries in documents; and a list of domain keywords to provide domain standard keywords. Based on the facts, logical reasoning is performed using a segmentation strategy package to obtain preliminary segmentation results, including: matching the facts with rules in the logical rule base through the reasoning engine and continuing reasoning until all the most stable and semantically complete segmentation boundaries that satisfy the logical rules are found; and obtaining preliminary segmentation results based on the decision path of the segmentation boundaries.

2. The standardized data block processing system supporting multi-source data formats according to claim 1, characterized in that, The preprocessing and format unification module includes a parsing module, a hierarchical structure conversion module, and an association tagging module; The parsing module is used to obtain the geometric layout information and page layout information of the document, and to remove headers and footers, reorder multi-column layouts, and reconstruct tables. The hierarchical structure conversion module is used to identify heading levels using heuristic rules or classification models and convert them into standardized heading tags; and to convert lists into corresponding standardized syntax. The association tagging module is used to identify related content pairs in a document and associate them with non-text elements through placeholders.

3. The standardized data block processing system supporting multi-source data formats according to claim 1, characterized in that, The document types for which standardized format text is obtained include: Scan standardized format text to obtain key features for classification; Based on key features used for classification, and through keyword matching, metadata analysis, and statistical analysis of title structure, the document types of standardized format texts are divided using a classification model.

4. A standardized data block processing system supporting multi-source data formats according to claim 1, characterized in that, The process of obtaining the chunking strategy package based on document type and strategy library matching mechanism includes: Based on the document type's type tag, match the corresponding chunking strategy from the strategy library; If a match is successful, the segmentation strategy is organized into a segmentation strategy package and sent to the hybrid inference deep segmentation engine.

5. A standardized data block processing system supporting multi-source data formats according to claim 1, characterized in that, The construction of the logical rules includes: Construct a corpus containing multiple source formats; process the documents in the corpus into a standardized format and extract structural, content, and relational features; transform the structural, content, and relational features into facts to obtain a fact database; Based on the fact database, a set of logical rules is derived to maximize the coverage of all positive instances in the corpus while minimizing negative instances.

6. A standardized data block processing system supporting multi-source data formats according to claim 1, characterized in that, The hybrid inference deep chunking engine includes a semantic analysis model, and the semantic analysis model tags each part of the text, including: A labeling system is constructed, including preconditions, conclusions, method descriptions, and object definitions. Based on the labeling system, samples extracted from the target document are labeled to obtain a labeled dataset. The labeled dataset is input into the pre-trained language model. The loss between the probability distribution predicted by the pre-trained language model and the labeled true labels is calculated. The backpropagation algorithm is used to update the classification head and parameters of the pre-trained language model to obtain the semantic analysis model. The semantic analysis model outputs the label for each part of the text.

7. A standardized data block processing system supporting multi-source data formats according to claim 1, characterized in that, The block result optimization and verification module includes an isolated content merging module and a boundary checking module; The isolated content merging module is used to obtain isolated blocks from the preliminary block division results; Based on the semantic similarity between isolated blocks and adjacent blocks, isolated blocks are merged into other blocks. The boundary checking module is used to check the block boundaries in the preliminary block division results and make fine adjustments.

8. A standardized data block processing system supporting multi-source data formats according to claim 7, characterized in that, The step of merging isolated blocks into other blocks based on their semantic similarity to adjacent blocks includes: Traverse the results of the initial block segmentation, identify isolated blocks based on a preset length threshold, and obtain the adjacent blocks of the isolated blocks. A pre-trained sentence embedding model is used to convert the text content of isolated blocks and adjacent blocks into high-dimensional semantic vectors, respectively. Based on the high-dimensional semantic vector, cosine similarity is used to calculate the semantic association between isolated blocks and adjacent blocks; Based on semantic relevance and a preset merging threshold, isolated blocks are merged with adjacent blocks, and the block list is updated.

Images