A system for extracting large model information and restoring structure from long text documents

Abstract

This invention belongs to the field of intelligent document processing technology, specifically relating to a system for extracting and restoring the structure of long text documents from large models. The system includes a module for defining extraction points and a text preprocessing module, used for deep preprocessing and parsing of the input unstructured document, extracting text and coordinate information, and identifying and protecting special structures. Simultaneously, it employs an intelligent dynamic block-segmentation strategy based on semantic boundaries, combined with a dynamic overlap mechanism, to divide the input document into text blocks that maintain semantic integrity. A high-concurrency processing and scheduling control module is used for batch processing of text blocks and calling a large language model for distributed inference, outputting dispersed results. An extraction, inference, and result fusion module is used to perform semantic deduplication, entity alignment, and confidence fusion on the dispersed results returned by the large language model, generating a globally consistent structured output. This invention features high accuracy, high speed, controllable cost, and strong robustness.

Description

Technical Field

[0001] This invention belongs to the field of intelligent document processing technology, specifically relating to a system for extracting large model information and restoring the structure of long text documents. Background Technology

[0002] Intelligent Document Processing (IDP), as one of the core technologies for enterprise digital transformation, has the core task of automatically converting unstructured or semi-structured document content (such as text, tables, key-value pairs, etc.) into operable structured data. Among these technologies, the ability to accurately extract key information from documents is particularly important and has extremely high application value.

[0003] In the field of Intelligent Document Processing (IDP), existing technical solutions for information extraction from long texts mainly include the following categories:

[0004] 1. The Retrieval Augmentation Generation (RAG) method based on naive chunking is currently the most common implementation. Its core process is as follows: First, long texts are segmented into fixed lengths (e.g., 512 or 1024 tokens), allowing a small amount of overlap between adjacent text blocks (e.g., 100 tokens) to attempt to mitigate semantic breaks at the boundaries. When processing queries, the most relevant text blocks to the question are first retrieved using vector similarity calculations. These text blocks, along with the question, are then submitted to a large language model to generate the final answer.

[0005] 2. Strategies based on sliding windows or hierarchical mapping aim to alleviate the semantic fragmentation problem caused by simple chunking through more complex processing strategies. For example, a sliding window mechanism can be used to traverse the entire document to capture the context across segments, or a hierarchical tree-like summary structure of the document (i.e., hierarchical mapping) can be constructed first, and then a coarse-to-fine logic can be followed during the query, first locating the relevant chapters and then retrieving specific paragraphs.

[0006] 3. Methods that rely on the model's own long contextual capabilities: These methods directly utilize large language models that support extremely long contextual windows, such as Claude 2.1 (200K context) and GPT-4 Turbo (128K context), to input the entire or most of the document into the model for processing at once. Their advantage lies in the model's ability to fully perceive the global context.

[0007] 4. Pipeline methods based on traditional Natural Language Processing (NLP), which was the mainstream technical path before the era of large language models. It relies on a complex traditional NLP technology stack, which typically includes steps such as sentence segmentation, part-of-speech tagging, and named entity recognition (NER), and then performs relation extraction based on predefined rules or traditional machine learning models (such as conditional random fields, support vector machines, etc.).

[0008] Despite the powerful capabilities of Large Language Models (LLMs), existing technologies face a series of inherent technical bottlenecks when processing documents that are hundreds of pages long, such as contracts, technical manuals, and academic papers, thus limiting their accuracy and practicality. These bottlenecks are mainly reflected in the following aspects:

[0009] 1. Context window constraints and model illusion problem

[0010] All large models are limited by their fixed context windows (e.g., 4K, 8K, 32K tokens). When the length of the document to be processed far exceeds or even approaches this limit, the entire text cannot be input into the model at once for understanding. This causes generative models to frequently experience "hallucinations" due to the lack of a complete global context, i.e., fabricating facts or misinterpreting information. Although expanding the context window is an intuitive solution, relying solely on this method is not only extremely costly but also still suffers from physical limitations and model performance degradation issues (such as the "middle collapse" phenomenon), and is not a fundamental solution.

[0011] 2. Semantic fragmentation problem

[0012] To overcome the limitations of the context window, the mainstream approach is to segment long texts into multiple chunks. However, traditional fixed-length overlapping chunking strategies (e.g., 1024 tokens per chunk, with 200 overlapping tokens) are prone to interruption within sentences, paragraphs, or key semantic units, severely compromising information integrity. For example, if the first and second halves of a key clause are split into two chunks, the model cannot effectively understand and reason. Although the industry has attempted more refined strategies based on semantic boundaries (e.g., sentences, paragraphs) or recursive segmentation, effectively maintaining semantic coherence across chunks remains a core challenge for current technology.

[0013] 3. Processing efficiency and cost bottlenecks

[0014] Dividing a long document into N blocks typically requires N large-scale API calls. This not only leads to slow processing speed (especially high latency from serial calls), but also causes the economic cost to increase linearly with the document length, making it difficult to apply to large-scale enterprise scenarios. Although asynchronous or concurrent calls can improve speed to some extent, brute-force concurrency can easily trigger the API's rate limit, causing requests to be rejected. Therefore, it is necessary to design additional complex flow control mechanisms.

[0015] 4. Insufficient ability to handle complex document structures.

[0016] Professional long documents commonly contain structured elements with strong internal logic, such as tables, formulas, and code blocks. Simple block-splitting operations can easily destroy these structures (such as cutting a table in the middle), causing the complete loss of row and column relationships or syntactic semantics, rendering the extraction results invalid. Existing general-purpose large-scale models still have limited ability to understand and protect such non-continuous text structures, and there is a lack of specialized technologies to identify and maintain their integrity during processing.

[0017] 5. Challenges in Results Integration and Consistency

[0018] After independently extracting information from multiple blocks, the resulting fragments are scattered, redundant, and potentially conflicting. Effectively deduplicating and merging these local results into a globally consistent, accurate, and non-redundant final output presents a complex post-processing challenge. This typically requires building a dedicated pipeline that includes steps such as entity alignment, confidence weighting, and semantic deduplication, increasing system complexity and uncertainty.

[0019] Therefore, it is very important to design a high-precision, fast, cost-controllable, and robust system for extracting large model information and restoring structure from long text documents. Summary of the Invention

[0020] This invention aims to overcome the problems of low extraction accuracy, significant response delay, high economic cost, and insufficient practicality and stability in existing technologies for extracting information from long text documents. It provides a high-precision, fast, cost-controllable, and robust system for extracting large-scale model information and restoring the structure of long text documents.

[0021] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0022] A system for extracting large model information and restoring structure from long text documents, characterized in that it includes:

[0023] The document defines extraction points and text preprocessing modules to perform deep preprocessing and parsing of the input unstructured documents, extract text and coordinate information, and identify and protect special structures. At the same time, it adopts an intelligent dynamic block segmentation strategy based on semantic boundaries, combined with a dynamic overlap mechanism, to divide the input document into text blocks that maintain semantic integrity.

[0024] The high-concurrency processing and scheduling control module is used to process text blocks in batches and call large language models for distributed inference, outputting distributed results.

[0025] The extraction, reasoning, and result fusion module is used to perform semantic deduplication, entity alignment, and confidence fusion on the scattered results returned by the large language model, generating a globally consistent structured output.

[0026] Preferably, in the modules defining extraction points and text preprocessing, defining extraction points specifically includes the following process:

[0027] Set text content extraction prompts based on extraction points, and also set table content extraction prompts based on extraction points.

[0028] Preferably, in the definition of extraction points and text preprocessing modules, text preprocessing specifically includes the following processes:

[0029] OCR recognition and high-quality text extraction, specifically:

[0030] For scanned documents or image-based PDFs, an OCR engine is used for text recognition. In addition to extracting the text content, the coordinates and font layout information of each word block are also obtained simultaneously. For digital PDFs, the internal text stream is directly parsed, and the position and style data of the characters are also preserved.

[0031] Special structure identification and protection based on rules and machine learning, specifically:

[0032] Table protection: Locate all table areas in a document by analyzing the horizontal and vertical alignment of word blocks, detecting table lines, and utilizing a pre-trained table recognition model;

[0033] Formula and code block protection: Based on regular expressions, it identifies mathematical formulas and program code snippets in documents;

[0034] Professional terminology and keyword protection: Build a professional dictionary database related to the field, scan the text during the preprocessing stage, and mark key terms.

[0035] As a preferred approach, the intelligent segmentation strategy, which incorporates a dynamic overlap mechanism, is used in the definition of extraction points and text preprocessing modules. Specifically, the following processes are included:

[0036] Segmentation based on semantic boundaries, specifically:

[0037] Choose natural semantic boundaries as the preferred boundaries for block segmentation;

[0038] The dynamic overlap mechanism is as follows:

[0039] For long paragraphs that cannot be avoided and must be segmented, the system automatically calculates the overlapping area between the segments; the size of the overlapping area is dynamically adjusted, that is, the number of overlapping tokens in the overlapping area is automatically increased according to the semantic density of the preceding text.

[0040] For protected special structures, separate blocks are used.

[0041] Preferably, the high-concurrency processing and scheduling control module adopts an asynchronous batch inference mechanism based on a priority queue, an intelligent flow control strategy with an integrated circuit breaker mechanism, and a failover and retry strategy.

[0042] The asynchronous batch inference mechanism based on priority queues is as follows:

[0043] Request Packaging and Batch Processing: The system dynamically packages multiple independent processing requests generated after intelligent segmentation into a batch request;

[0044] Asynchronous non-blocking calls: All batch requests are sent in asynchronous non-blocking mode;

[0045] Priority queue scheduling: The system maintains a priority queue to manage all pending batch requests; the priority of requests is dynamically set according to business needs.

[0046] Prioritization based on processing latency: Requests with excessively long waiting times will have their priority automatically increased.

[0047] As a preferred embodiment, the intelligent flow control strategy with integrated circuit breaking mechanism is as follows:

[0048] Dynamic concurrency control: The system monitors the response status, response latency, and its own request queue depth of the large language model LLM API in real time, and dynamically adjusts the number of concurrent worker threads or the request sending rate based on the feedback of the metrics to achieve adaptive rate limiting.

[0049] Rate shaping using the token bucket algorithm: The system uses the token bucket algorithm and dynamically configures the rate limiting strategy based on the core parameters of the algorithm and the specific rate limiting strategies of different LLM providers.

[0050] Circuit Breaker Mechanism: When the system detects that the API error rate continues to exceed the preset threshold or the response time is abnormally prolonged, the circuit breaker mechanism will be automatically triggered. The circuit breaker will temporarily stop sending requests to the API endpoint and enter a "cooling-off period". During the "cooling-off period", the system will attempt to retry failed requests or return a downgraded result. After the cooling-off period ends, the circuit breaker will be half-open to send a small number of requests tentatively. If it is confirmed that the API service has been restored, the circuit breaker will be closed and normal calls will resume.

[0051] As a preferred embodiment, the failover and retry strategy is as follows:

[0052] Automatic retry: For requests that fail momentarily due to network fluctuations or server-side rate limiting, the system will automatically retry a limited number of times.

[0053] Preferably, the extraction reasoning and result fusion module includes a large model reasoning execution process, a result collection and preliminary analysis process, a confidence processing and quality filtering process, a semantic deduplication and cross-block entity alignment process, a restoration and integration process of special structure extraction results, and a global result fusion and output process.

[0054] The execution process of large-scale model inference is as follows:

[0055] Standardized API calls: Receive batch requests from the high-concurrency processing and scheduling control module, and initiate inference requests to the large language model through standardized interfaces;

[0056] Response reception and buffering: Asynchronously receive the raw response returned by the large language model; the system has a buffering mechanism to cope with network fluctuations or server response delays;

[0057] The results collection and preliminary analysis process is as follows:

[0058] Concurrent Results Summary: Collects the results returned by all asynchronous inference tasks using a high-performance concurrent queue;

[0059] JSON parsing and error handling: Parse the returned JSON string and extract structured results; mark and record non-standard formats, empty results, or parsing errors returned by large models.

[0060] Preferably, the confidence processing and quality filtering process is as follows:

[0061] Confidence score association: Assign a confidence score to each extracted result;

[0062] Threshold-based filtering: Set a configurable confidence threshold; results below the confidence threshold will be automatically filtered or marked as low-confidence items.

[0063] The semantic deduplication and cross-block entity alignment process is as follows:

[0064] Semantic deduplication: For redundant results describing the same entity extracted from different text blocks, a similarity calculation model based on semantic embedding is used to identify and deduplicate the content, retaining only the version with the highest confidence or the most complete information.

[0065] Entity parsing and association: Establishing entity associations across text blocks.

[0066] Preferably, the process of restoring and integrating the results of the special structure extraction is as follows:

[0067] Structure restoration: Restores the structured metadata contained in specially protected tables and formulas to standard table JSON or LaTeX format;

[0068] Context integration: The special structure to be restored is placed back into the original document context and fused with the results extracted from plain text to form a complete output;

[0069] The global result fusion and output are as follows:

[0070] Result aggregation: Aggregate all local results that have been parsed, filtered, deduplicated, correlated, and restored into a unified data structure;

[0071] Eventual consistency check: Performs an eventual consistency check, including checking whether required fields are complete and whether the numerical logic is consistent.

[0072] Structured output: Generates the final structured data file, which can be directly integrated and utilized by downstream business systems.

[0073] Compared with existing technologies, the advantages of this invention are: (1) Significantly improves extraction accuracy: Through intelligent dynamic segmentation and semantic boundary protection mechanism, the semantic fragmentation problem caused by traditional segmentation method is effectively avoided, ensuring the complete preservation of key information; combined with special structure protection and multi-dimensional result fusion, the model illusion phenomenon is greatly reduced, and the accuracy and consistency of information extraction are improved; (2) Significantly improves processing efficiency and response speed: The high-concurrency asynchronous scheduling mechanism and intelligent flow control strategy are adopted to significantly reduce API call latency and network overhead, support large-scale concurrent processing, and are suitable for enterprise-level massive document processing scenarios; (3) Significantly reduces economic costs: Through batch request processing, dynamic concurrency control and fault transfer mechanism, the number of API calls is effectively reduced. (3) Reduce the number of attempts and the cost of retrying failures to achieve a cost-controllable long text processing solution; (4) Enhance system robustness and stability: integrate circuit breaking mechanism, automatic retry and priority scheduling strategy, have good fault tolerance and service degradation mechanism, and ensure that the system can still run stably under high load or downstream service abnormality; (5) Support complex document structure processing: have the ability to identify and protect special structures such as tables, formulas, and code blocks, support lossless extraction of multi-level nested structures, and expand the applicability of the system in professional document processing; (6) Improve user experience and ease of use: provide a visual configuration interface and Prompt automatic optimization function, reduce the threshold for users, support flexible task customization and result visualization, and improve the operability and practicality of the system. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of an architecture for a large model information extraction and structure reconstruction system for long text documents according to the present invention;

[0075] Figure 2 This is a flowchart illustrating the high-concurrency processing and scheduling control module in this invention.

[0076] Figure 3 This is a flowchart illustrating the extraction reasoning and result fusion module in this invention.

[0077] Figure 4 A flowchart illustrating the application of the system for extracting large model information and restoring structure of long text documents, as provided in this embodiment of the invention, to extract key information from supply and sales contracts.

[0078] Figure 5 This is a schematic diagram showing the text extraction results of a text extraction process without LLM optimization in an embodiment of the present invention;

[0079] Figure 6 This is a schematic diagram showing the text extraction result after LLM-optimized prompt word process in an embodiment of the present invention. Detailed Implementation

[0080] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0081] This invention provides a system for extracting large model information and restoring structure from long text documents, the core of which is as follows:

[0082] First, the system performs in-depth preprocessing and parsing on input unstructured documents such as PDFs, extracting text and coordinate information through OCR recognition and layout analysis, and identifying special structures such as protected tables and formulas. Next, it adopts an intelligent dynamic segmentation strategy based on semantic boundaries, combined with a dynamic overlap mechanism, to divide long documents into text blocks that maintain semantic integrity. Then, it processes text blocks in batches through a high-concurrency asynchronous scheduling mechanism, calling a large language model for distributed inference. Finally, it performs semantic deduplication, entity alignment, and confidence fusion on the multi-block results returned by the model to generate globally consistent structured output.

[0083] Specifically, the system functional architecture proposed in this invention is as follows: Figure 1 As shown, the overall system mainly includes the following modules, as detailed below:

[0084] Define extraction points and text preprocessing modules:

[0085] The extraction point is defined by configuring the content to be extracted as needed. Two different prompt word processes are built-in for tables and body text. Regarding optimization, considering that some users may not be familiar with the large model, a new module is added to automatically optimize the prompt word process by calling the pre-defined large model.

[0086] Text preprocessing: This process forms the foundation of the entire system's processing flow. Its core task is to convert raw unstructured or semi-structured documents (such as scanned PDFs, digital PDFs, etc.) into high-quality, standardized text data suitable for subsequent intelligent segmentation and large-scale model inference. This process not only completes traditional text extraction and cleaning but also addresses the challenges of protecting special structures and maintaining semantic integrity in long documents through a series of refined processing strategies. The specific process is as follows:

[0087] 1. OCR recognition and high-quality text extraction

[0088] For scanned documents or image-based PDFs, a high-precision OCR engine is used for text recognition. In addition to extracting the text content, it also simultaneously acquires the coordinates, font, and other layout information of each word block. For digital PDFs, the internal text stream is directly parsed, preserving the precise position and style data of the characters. All this information is encapsulated in a unified Extract data class, providing a data foundation for subsequent structural analysis and intelligent segmentation.

[0089] 2. Special structure identification and protection based on rules and machine learning

[0090] This is a crucial step in the preprocessing stage, designed to identify and protect specialized content that is highly susceptible to corruption in simple chunking, which could lead to complete loss of semantics.

[0091] Table protection: By analyzing the horizontal and vertical alignment of word blocks, detecting table lines (solid or dashed), and utilizing a pre-trained table recognition model, all table areas in a document are precisely located. Once recognized, the entire table is treated as an independent, indivisible "semantic unit" or "protected object," and will be forcibly retained within the same text block during subsequent segmentation, preventing it from being cut off in the middle.

[0092] Formula and code block protection: Based on regular expressions, mathematical formulas and program code snippets in the document are identified. These are also marked as protected units to ensure their integrity.

[0093] Technical terminology and keyword protection: A domain-specific dictionary (such as legal clause names, medical terms, and engineering parameters) is built. During the preprocessing stage, the text is scanned, and these key terms are tagged. This measure not only helps protect important information but also provides semantic anchors for subsequent intelligent segmentation.

[0094] 3. Dynamically Overlapping Intelligent Blocking Strategy

[0095] Based on the completion of special structural protection, this system adopts a dynamic segmentation strategy to replace the traditional fixed-length segmentation.

[0096] Semantic boundary-based chunking: The preferred boundaries for chunking are natural semantic boundaries, such as paragraph ends, chapter titles, etc. This ensures that most chunks contain complete semantic paragraphs.

[0097] Dynamic overlap mechanism: For long paragraphs that must be segmented, the system calculates the overlap area between blocks. The size of this overlap area is not fixed but dynamically adjusted: it automatically increases the number of overlap tokens based on the semantic density of the preceding text (e.g., containing multiple technical terms or entities) to ensure that key contextual information is not lost due to segmentation. This dynamic mechanism maximizes the mitigation of semantic fragmentation while maintaining efficiency. For protected special structures already processed in the previous step, separate segmentation is used. Although this increases the number of large model calls and the overall system response time to some extent, it protects the integrity of the semantic structure and significantly improves extraction accuracy.

[0098] 4. Text Preprocessing and Normalization

[0099] The extracted raw text is cleaned, including removing meaningless garbled characters, standardizing spaces, and unifying encoding.

[0100] At the same time, it performs basic natural language processing tasks such as sentence boundary detection and word segmentation to prepare for the creation of a high-quality prompt word process.

[0101] Through the processing of this module, the original document is transformed into a series of protected text units rich in semantic and structural information, laying a solid data foundation for subsequent high-precision information extraction and effectively curbing the model illusion problem caused by format chaos or structural destruction from the source.

[0102] High-concurrency processing and scheduling control module: This module is the core of this invention, designed to efficiently, stably, and cost-effectively schedule a large number of calls to the Large Language Model (LLM) API. It systematically solves engineering challenges in long document processing, such as high request counts, strict rate limits, and high call costs, through an integrated asynchronous batch inference mechanism and intelligent flow control strategy. The specific process of this module is as follows: Figure 2 As shown, the details are as follows:

[0103] 1. Asynchronous batch inference mechanism based on priority queue

[0104] Request Packaging and Batch Processing: The system dynamically packages multiple independent processing requests (including text blocks and corresponding prompt word flows) generated after intelligent segmentation into a batch request. This batch processing method can significantly reduce the number of network round trips with the large model service, thereby reducing the overall request latency.

[0105] Asynchronous non-blocking calls: All batch requests are sent in an asynchronous non-blocking mode. The main processing thread is not blocked by a single API response and can continuously prepare and send subsequent requests, greatly improving the overall throughput and resource utilization of the system.

[0106] Priority queue scheduling: The system maintains a priority queue to manage all pending batch requests. Request priorities can be dynamically set according to business needs, for example: priority based on content criticality: text blocks containing important chapters or high-confidence keywords can be given higher priority and sent first to ensure the timeliness of extracting key information.

[0107] Prioritization based on processing latency: Requests with excessively long waiting times will have their priority automatically increased to prevent some requests from timing out due to continuous queue interruption.

[0108] 2. Intelligent flow control with integrated circuit breaker mechanism

[0109] Dynamic concurrency control: The system monitors the response status of the LLM API (such as HTTP 429 and 503 error codes), response latency, and its own request queue depth in real time. Based on these feedback metrics, the system dynamically adjusts the number of concurrent worker threads or the request sending rate to achieve adaptive rate limiting.

[0110] The token bucket algorithm implements rate shaping: The system uses the token bucket algorithm to finely "shape" the request sending rate. The core parameters of the algorithm (token generation rate, bucket capacity) can be dynamically configured according to the specific rate limiting policies of different LLM providers. This ensures that request traffic is sent smoothly and evenly, avoiding API-side restrictions triggered by excessive instantaneous traffic, thus guaranteeing continuous service availability.

[0111] Circuit Breaker Mechanism: When the system detects that the API error rate consistently exceeds a preset threshold or the response time is abnormally prolonged, the circuit breaker mechanism will be automatically triggered. The circuit breaker will temporarily stop sending requests to the API endpoint, entering a short "cooling-off period." During the cooling-off period, the system can attempt to retry failed requests or return a degraded result. After the cooling-off period ends, the circuit breaker will half-open to send a small number of tentative requests. If it is confirmed that the API service has recovered, the circuit breaker will close, and normal calls will resume. This mechanism effectively prevents system cascading failures caused by downstream service instability.

[0112] 3. Failover and Retry Strategies

[0113] Automatic retry: For requests that fail momentarily due to network fluctuations or server-side rate limiting, the system will automatically retry a limited number of times.

[0114] Through the design of this module, this invention achieves refined control over large-scale LLM calls with "high concurrency, low latency, high stability, and low cost," enabling it to handle enterprise-level processing tasks of massive long documents and breaking through the performance and stability bottlenecks of traditional serial or simple concurrent call methods.

[0115] Extraction, Inference, and Result Fusion Module: This module is the core execution layer for large-scale asynchronous inference tasks. It is responsible for efficiently and reliably completing large model calls and deeply processing and fusing the returned scattered results, ultimately generating high-quality, structured global output. Its design focuses on handling the challenges of result integrity, accuracy, and consistency in high-concurrency environments. The specific workflow of this module is as follows: Figure 3 As shown, the details are as follows:

[0116] 1. Large model inference execution

[0117] Standardized API calls: Receive batch requests from the scheduling module and initiate inference requests to large language models (such as GPT-4, Claude, etc.) through standardized interfaces. The request content includes the prompt word extraction process and corresponding preprocessed text blocks, designed after multiple experimental verifications and debugging.

[0118] Response Reception and Buffering: The system receives the raw response (usually in JSON format) from the asynchronous receiving model. A buffering mechanism is in place to handle network fluctuations or server response delays, ensuring that an exception in a single request does not block the overall processing flow.

[0119] 2. Results Collection and Preliminary Analysis

[0120] Concurrent Results Aggregation: A high-performance concurrent queue is used to collect the results returned by all asynchronous inference tasks. This queue features high throughput and persistence, ensuring no data is lost even under heavy requests.

[0121] JSON parsing and error handling: The returned JSON string is parsed to extract structured results. Strict exception handling is performed at this stage.

[0122] Mark and record non-standard formats, empty results, or parsing errors returned by the model.

[0123] Trigger an automatic retry mechanism or add the case to the queue for manual review to ensure the robustness of the processing flow.

[0124] 3. Confidence processing and quality filtering

[0125] Confidence score association: Assign a confidence score to each extraction result. This score can be derived from the model's own output (if supported), the logical consistency check of the extraction results, or the degree of matching with the domain dictionary.

[0126] Threshold-based filtering: Set a configurable confidence threshold. Results below the threshold will be automatically filtered or marked as low-confidence items, which can either be passed to subsequent modules for semantic verification or directly excluded, effectively controlling the downward propagation of noise.

[0127] 4. Semantic deduplication and cross-block entity alignment

[0128] Semantic deduplication: For redundant results describing the same entity extracted from different text blocks, a similarity calculation model based on semantic embedding (Sentence-BERT, etc.) is used for identification and deduplication. Only the version with the highest confidence or the most complete information is retained.

[0129] Entity parsing and association: Establishing entity relationships across text blocks. For example, "Party A's Name" extracted from "Chapter 1" needs to be aligned and merged with "Contract Signatory A" extracted from "Chapter 5," ultimately associating them under the same entity ID. This is a crucial step in building a globally consistent knowledge graph.

[0130] 5. Restoration and Integration of Extraction Results for Special Structures

[0131] Structure Restoration: For tables, formulas, and other content that are specially protected during the preprocessing stage, the extraction results typically contain structured metadata (such as row and column indices, formula numbers). This module is responsible for restoring this structured data to standard tabular JSON or LaTeX format to ensure its usability.

[0132] Context integration: The special structure that has been restored is placed back into its original document context and merged with other results extracted from plain text to form a complete output.

[0133] 6. Global Result Fusion and Output

[0134] Result aggregation: Aggregate all parsed, filtered, deduplicated, correlated, and restored local results into a unified data structure (usually JSON or XML).

[0135] Eventual consistency check: Performs an eventual consistency check, such as checking whether required fields are complete and whether the numerical logic is consistent (e.g., whether the total amount equals the sum of the individual items).

[0136] Structured output: Generates final structured data files that can be directly integrated and utilized by downstream business systems (such as databases, CRM, and BI tools). The output format supports flexible configuration to meet different integration needs.

[0137] Through the in-depth processing of this module, scattered, raw, and potentially conflicting model outputs are transformed into clean, complete, and globally consistent high-value structured data, marking the completion of the final transformation from unstructured long documents to trustworthy and usable data.

[0138] Based on the technical solution of this invention, the implementation process of this invention in practical applications will be illustrated through the following case scenarios. The specific application implementation scheme is as follows:

[0139] Application scenario: Extraction of key information from supply and sales contracts. The specific processing flow is as follows: Figure 4 As shown, the details are as follows:

[0140] 1. Input document:

[0141] Document type: PDF format supply and sales contract

[0142] Document length: 200 pages

[0143] Content features: Contains a large amount of structured and unstructured content, such as contract terms, technical specifications, price lists, signature and seal pages, appendices, etc., with a total of 225,405 characters, far exceeding the context window limit of current mainstream large language models (such as GPT-4Turbo) (usually 128K tokens).

[0144] 2. Document preprocessing:

[0145] Performs high-precision OCR recognition and text extraction on PDFs, preserving layout information such as word block coordinates, fonts, and typesetting.

[0146] Identify and protect key structures in contracts, such as forms, clause headings, monetary figures, dates, and legal terms.

[0147] The extracted text content is converted to Markdown format to retain some structural information (such as heading levels, lists, bold text, etc.) to facilitate understanding by the subsequent large model.

[0148] Users define 109 extraction points through the system interface, including:

[0149] Standard sampling points: contract number, name of Party A / Party B, total contract amount, signing date, delivery deadline, etc.

[0150] Complex semantic extraction points: objection handling clauses, acceptance criteria, liability for breach of contract, confidentiality agreements, ownership of intellectual property rights, etc.

[0151] 3. Intelligent block division:

[0152] A multi-level semantic segmentation strategy is adopted:

[0153] Prioritize segmentation by chapter title (e.g., "Chapter 1 General Provisions", "Chapter 2 Product Delivery"); second priority is segmentation by paragraph boundaries to ensure semantic integrity; for very long paragraphs (e.g., technical appendices), a dynamic overlap mechanism is used, with the overlapping area automatically adjusted according to semantic density; special structural protection: all tables, code blocks, formulas, etc. are treated as indivisible units and formed into separate blocks to ensure their structural integrity.

[0154] 4. High-concurrency scheduling:

[0155] The segmented text and the corresponding extraction prompt word process are dynamically packaged into batch tasks; a priority queue scheduling mechanism is used:

[0156] High priority: Blocks involving amounts, dates, and key terms;

[0157] Medium priority: General description content;

[0158] Low priority: appendices, references, etc.;

[0159] By combining the token bucket algorithm and the circuit breaker mechanism, the number of concurrent requests is dynamically controlled to avoid triggering API rate limiting.

[0160] 5. Large-scale model reasoning:

[0161] By using asynchronous batch calls, multiple text blocks are merged into a single batch request, significantly reducing the number of network round trips; GPT-4 is called to extract information, and each block returns a preliminary structured result (JSON format).

[0162] The inference component can additionally utilize LLM to automatically optimize user-provided prompts, even for prompts from inexperienced users, and in some cases, this module can still significantly improve accuracy. See below for specific comparison results. Figure 5 and Figure 6 . Figure 5 This shows the extraction results without the automatic prompt word optimization module. Figure 6 The extraction results are shown after incorporating the automatic prompt word optimization module. A comparison reveals that... Figure 5 There are problems such as incorrect sampling, missed sampling, and incomplete sampling. Figure 6 The extraction results are more complete and accurate, resulting in a significant improvement in overall performance. The system algorithm interface records the extraction confidence of each block for use in subsequent fusion stages.

[0163] 6. Results fusion and post-processing:

[0164] Semantic deduplication: Based on TF-IDF and cosine similarity algorithms, duplicate or highly similar terms in multiple blocks are merged;

[0165] Entity alignment: Identifying and merging the same entity (e.g., "Party A: a certain technology company") that is scattered in different blocks;

[0166] Confidence-weighted fusion: The results of the same extraction point in multiple blocks are weighted by confidence level to filter low-confidence segments;

[0167] 7. Output results:

[0168] Generate structured JSON output, including the content, confidence level, and occurrence position (page number, block ID) of each extraction point; provide a visual interface to display the extraction results, and support clicking to view original text fragments, improving interpretability and user experience.

[0169] Effect comparison:

[0170] (1) Sampling accuracy

[0171] Traditional methods suffer from severe semantic fragmentation, with key information scattered across different text blocks, causing the model to be unable to understand the complete context, resulting in an accuracy of only about 60%. This invention, through intelligent segmentation and semantic protection mechanisms, ensures the integrity of key information, improving accuracy to 85%, a relative improvement of about 25%.

[0172] (2) Processing time

[0173] Traditional methods, employing serial or simple concurrent calls, require over 300 seconds to process a 200-page document. This invention, through high-concurrency scheduling, batch request processing, and asynchronous inference mechanisms, reduces processing time to 40 seconds, significantly improving efficiency.

[0174] (3) Number of API calls

[0175] Traditional fixed-block methods require individual API calls for each text block, totaling 200 calls. This invention reduces unnecessary segmentation through intelligent blocking and, combined with batch processing technology, reduces the number of calls to 100, a 50% reduction.

[0176] (4) Table extraction support

[0177] Traditional methods are ineffective at handling structured content such as tables, often resulting in fragmented tables and information loss. This invention possesses table recognition and protection capabilities, can completely extract table content, and supports the processing of complex document structures.

[0178] (5) Semantic deduplication function

[0179] Traditional methods lack cross-block result fusion mechanisms, resulting in a large amount of redundant and conflicting information in the output. This invention introduces a multi-dimensional result fusion mechanism to achieve semantic-level deduplication, ensuring the consistency and integrity of the output results.

[0180] (6) System stability

[0181] Traditional methods are prone to triggering API rate limits in high-concurrency scenarios, leading to request rejections and poor system stability. This invention integrates intelligent flow control, circuit breaking mechanisms, and dynamic concurrency regulation, which can adapt to the status of downstream services and significantly improve system robustness and fault tolerance.

[0182] The above comparative results demonstrate that this invention significantly outperforms traditional fixed-block methods in terms of extraction accuracy, processing efficiency, economic cost, and system stability, making it particularly suitable for enterprise-level large-scale long document processing scenarios. This system performs exceptionally well in long text information extraction tasks, especially in handling complex and lengthy commercial contracts, maintaining high accuracy while significantly improving processing efficiency and system stability, thus possessing strong engineering implementation and commercialization value.

[0183] This invention proposes a method and system for efficient information extraction from large-scale, ultra-long texts in Intelligent Document Processing (IDP), effectively solving several core problems of large models in long text processing: 1) model illusion caused by ultra-long text input; 2) semantic fragmentation caused by traditional block-based methods; and 3) slow inference speed during ultra-long text extraction. Through deep optimization of the large-scale model processing mechanism, this invention significantly improves processing efficiency while maintaining high accuracy, providing a stable and reliable solution for processing hundreds-of-pages-long texts such as contracts and technical documents.

[0184] The innovative aspects of this invention are as follows:

[0185] 1. Semantic-aware intelligent segmentation mechanism

[0186] Employing a multi-level segmentation strategy (prioritizing sentence boundaries, followed by logical connectors, and finally protecting special structures), combined with terminology protection technology, it automatically identifies and maintains the integrity of domain-specific terminology. The built-in data module supports custom domain-specific keywords and retrieves them in real-time during the extraction process. Through a dynamic overlap compensation mechanism, it ensures the semantic coherence across blocks, effectively solving the semantic fragmentation problem caused by traditional fixed-length segmentation, thereby significantly improving the accuracy of long text processing.

[0187] 2. Dynamic Concurrency Control System

[0188] By monitoring key performance indicators such as response latency and error rate in real time, and dynamically allocating process tokens, intelligent concurrency optimization based on historical performance data is achieved. Combined with an adaptive rate control mechanism, it can automatically reduce and recover when encountering API limitations; and with the token bucket algorithm, it provides fine-grained management of request traffic, improving system throughput efficiency while ensuring service stability. An efficient asynchronous processing mechanism further reduces I / O wait time, significantly improving overall processing speed.

[0189] 3. Multi-dimensional result fusion mechanism

[0190] Cross-block entity association technology can merge related entities using string similarity to solve the problem of reorganizing scattered entities; confidence-weighted fusion algorithm can filter low-confidence segments; semantic deduplication engine achieves semantic-level deduplication based on semantic similarity detection algorithm and cosine similarity, ensuring the integrity and consistency of the final extraction results.

[0191] 4. Special structural protection system

[0192] It has the ability to automatically identify and isolate tables, formulas and code blocks, and adopts structured data lossless extraction technology to ensure content integrity; it supports the identification and protection of special structures with multiple nested layers, and achieves accurate processing of complex document structures.

[0193] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A system for extracting large model information and restoring structure from long text documents, characterized in that, include: The document defines extraction points and text preprocessing modules to perform deep preprocessing and parsing of the input unstructured documents, extract text and coordinate information, and identify and protect special structures. At the same time, it adopts an intelligent dynamic block segmentation strategy based on semantic boundaries, combined with a dynamic overlap mechanism, to divide the input document into text blocks that maintain semantic integrity. The high-concurrency processing and scheduling control module is used to process text blocks in batches and call large language models for distributed inference, outputting distributed results. The extraction, reasoning, and result fusion module is used to perform semantic deduplication, entity alignment, and confidence fusion on the scattered results returned by the large language model, generating a globally consistent structured output. In the modules for defining extraction points and text preprocessing, defining extraction points specifically includes the following process: Set text content extraction prompts based on extraction points, and also set table content extraction prompts based on extraction points; In the modules defining extraction points and text preprocessing, text preprocessing specifically includes the following steps: OCR recognition and high-quality text extraction, specifically: For scanned documents or image-based PDFs, an OCR engine is used for text recognition. In addition to extracting the text content, the coordinates and font layout information of each word block are also obtained simultaneously. For digital PDFs, the internal text stream is directly parsed, and the position and style data of the characters are also preserved. Special structure identification and protection based on rules and machine learning, specifically: Table protection: Locate all table areas in a document by analyzing the horizontal and vertical alignment of word blocks, detecting table lines, and utilizing a pre-trained table recognition model; Formula and code block protection: Based on regular expressions, it identifies mathematical formulas and program code snippets in documents; Professional terminology and keyword protection: Build a domain-related professional dictionary database, scan the text during the preprocessing stage, and mark key terms; In defining extraction points and text preprocessing modules, the intelligent block segmentation strategy, which incorporates a dynamic overlap mechanism, specifically includes the following processes: Segmentation based on semantic boundaries, specifically: Choose natural semantic boundaries as the preferred boundaries for block segmentation; The dynamic overlap mechanism is as follows: For long paragraphs that cannot be avoided and must be segmented, the system automatically calculates the overlapping area between the segments; the size of the overlapping area is dynamically adjusted, that is, the number of overlapping tokens in the overlapping area is automatically increased according to the semantic density of the preceding text. For protected special structures, separate blocks are used.

2. The system for large-scale model information extraction and structure reconstruction of long text documents according to claim 1, characterized in that, The high-concurrency processing and scheduling control module adopts an asynchronous batch inference mechanism based on priority queues, an intelligent flow control strategy with integrated circuit breaking mechanism, and a fault transfer and retry strategy. The asynchronous batch inference mechanism based on priority queues is as follows: Request Packaging and Batch Processing: The system dynamically packages multiple independent processing requests generated after intelligent segmentation into a batch request; Asynchronous non-blocking calls: All batch requests are sent in asynchronous non-blocking mode; Priority queue scheduling: The system maintains a priority queue to manage all pending batch requests; the priority of requests is dynamically set according to business needs. Prioritization based on processing latency: Requests with excessively long waiting times will have their priority automatically increased.

3. The system for large-scale model information extraction and structure reconstruction of long text documents according to claim 2, characterized in that, The intelligent flow control strategy with integrated circuit breaker mechanism is as follows: Dynamic concurrency control: The system monitors the response status, response latency, and its own request queue depth of the large language model LLM API in real time, and dynamically adjusts the number of concurrent worker threads or the request sending rate based on the feedback of the metrics to achieve adaptive rate limiting. Rate shaping using the token bucket algorithm: The system uses the token bucket algorithm and dynamically configures the rate limiting strategy based on the core parameters of the algorithm and the specific rate limiting strategies of different LLM providers. Circuit Breaker Mechanism: When the system detects that the API error rate continues to exceed the preset threshold or the response time is abnormally prolonged, the circuit breaker mechanism will be automatically triggered. The circuit breaker will temporarily stop sending requests to the API endpoint and enter a "cooling-off period". During the "cooling-off period", the system will attempt to retry failed requests or return a downgraded result. After the cooling-off period ends, the circuit breaker will be half-open to send a small number of requests tentatively. If it is confirmed that the API service has been restored, the circuit breaker will be closed and normal calls will resume.

4. The system for large-scale model information extraction and structure reconstruction of long text documents according to claim 3, characterized in that, The specific failover and retry strategy is as follows: Automatic retry: For requests that fail momentarily due to network fluctuations or server-side rate limiting, the system will automatically retry a limited number of times.

5. The system for large-scale model information extraction and structure reconstruction of long text documents according to claim 4, characterized in that, The extraction reasoning and result fusion module includes a large model reasoning execution process, a result collection and preliminary analysis process, a confidence processing and quality filtering process, a semantic deduplication and cross-block entity alignment process, a restoration and integration process of special structure extraction results, and a global result fusion and output process. The execution process of large-scale model inference is as follows: Standardized API calls: Receive batch requests from the high-concurrency processing and scheduling control module, and initiate inference requests to the large language model through standardized interfaces; Response reception and buffering: Asynchronously receive the raw response returned by the large language model; the system has a buffering mechanism to cope with network fluctuations or server response delays; The results collection and preliminary analysis process is as follows: Concurrent Results Summary: Collects the results returned by all asynchronous inference tasks using a high-performance concurrent queue; JSON parsing and error handling: Parse the returned JSON string and extract structured results; mark and record non-standard formats, empty results, or parsing errors returned by large models.

6. The system for large-scale model information extraction and structure reconstruction of long text documents according to claim 5, characterized in that, The confidence processing and quality filtering process is as follows: Confidence score association: Assign a confidence score to each extracted result; Threshold-based filtering: Set a configurable confidence threshold; results below the confidence threshold will be automatically filtered or marked as low-confidence items. The semantic deduplication and cross-block entity alignment process is as follows: Semantic deduplication: For redundant results that describe the same entity extracted from different text blocks, a similarity calculation model based on semantic embedding is used to identify and deduplicate the content, retaining only the version with the highest confidence or the most complete information. Entity parsing and association: Establishing entity associations across text blocks.

7. The system for large-scale model information extraction and structure reconstruction of long text documents according to claim 6, characterized in that, The process of restoring and integrating the extracted results of the special structure is as follows: Structure restoration: Restores the structured metadata contained in specially protected tables and formulas to standard table JSON or LaTeX format; Context integration: The special structure to be restored is placed back into the original document context and fused with the results extracted from plain text to form a complete output; The global result fusion and output are as follows: Result aggregation: Aggregate all local results that have been parsed, filtered, deduplicated, correlated, and restored into a unified data structure; Eventual consistency check: Performs an eventual consistency check, including checking whether required fields are complete and whether the numerical logic is consistent. Structured output: Generates the final structured data file, which can be directly integrated and utilized by downstream business systems.

Images