Large model information extraction and structure restoration system for long text document

Abstract

The invention belongs to the technical field of intelligent document processing, and particularly relates to a large model information extraction and structure restoration system for a long text document. The system comprises an extraction point defining and text preprocessing module which is used for carrying out deep preprocessing and analysis on an input unstructured document, extracting text and coordinate information and identifying and protecting a special structure; meanwhile, an intelligent dynamic blocking strategy based on semantic boundaries is adopted, a dynamic overlapping mechanism is combined, and an input document is divided into text blocks keeping semantic integrity; the high-concurrency processing and scheduling control module is used for processing the text blocks in batches, calling a large language model to carry out distributed reasoning and outputting a dispersion result; and the extraction reasoning and result fusion module is used for performing semantic deduplication, entity alignment and confidence fusion on the dispersion result returned by the large language model to generate globally consistent structured output. The method has the advantages of being high in precision, high in speed, controllable in cost and high in robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent document processing, and particularly relates to a large model information extraction and structure restoration system for long text documents. BACKGROUND

[0002] Intelligent Document Processing (IDP) is one of the core technologies of enterprise digital transformation, and its core task is to automatically convert unstructured or semi-structured document content (such as text, tables, key-value pairs, etc.) into structured data that can be operated. Among them, the technology of accurately extracting key information from documents is particularly important and has very high application value.

[0003] In the field of Intelligent Document Processing (IDP), for the information extraction task of long text, the existing technical solutions mainly include the following categories: 1. Retrieval Augmented Generation (RAG) method based on Naive Chunking, which is the most common implementation method at present. Its core process is: first, divide the long text into fixed length (for example, 512 or 1024 tokens), and allow a small amount of overlap (for example, 100 tokens) between adjacent text blocks to try to alleviate the semantic break at the boundary. When processing queries, first retrieve a number of text blocks most relevant to the question by vector similarity calculation, and then submit these text blocks and the question to a large language model to generate the final answer.

[0004] 2. Strategy based on sliding window or hierarchical mapping, which aims to alleviate the semantic fragmentation problem caused by simple chunking through more complex processing strategies. For example, use a sliding window mechanism to traverse the entire document to capture cross-fragment context, or first construct a tree-like hierarchical summary structure (i.e. hierarchical mapping) of the document, and when querying, follow the logic from coarse to fine, first locate to the relevant chapter and then retrieve specific paragraphs.

[0005] 3. Method relying on the long context ability of the model itself, which directly uses large language models such as Claude 2.1 (200K context), GPT-4 Turbo (128K context) that support super-long context window, and processes the entire or most of the document at a time. The advantage is that the model can fully perceive the global context.

[0006] 4. Pipeline methods based on traditional Natural Language Processing (NLP), which was the mainstream technical path before the era of large language models. It relies on a complex traditional NLP technology stack, which typically includes steps such as sentence segmentation, part-of-speech tagging, and named entity recognition (NER), and then performs relation extraction based on predefined rules or traditional machine learning models (such as conditional random fields, support vector machines, etc.).

[0007] Despite the powerful capabilities of Large Language Models (LLMs), existing technologies face a series of inherent technical bottlenecks when processing documents that are hundreds of pages long, such as contracts, technical manuals, and academic papers, thus limiting their accuracy and practicality. These bottlenecks are mainly reflected in the following aspects: 1. Context window limitations and model illusion problem All large models are limited by their fixed context windows (e.g., 4K, 8K, 32K tokens). When the length of the document to be processed far exceeds or even approaches this limit, the entire text cannot be input into the model at once for understanding. This causes generative models to frequently experience "hallucinations" due to the lack of a complete global context, i.e., fabricating facts or misinterpreting information. Although expanding the context window is an intuitive solution, relying solely on this method is not only extremely costly but also still suffers from physical limitations and model performance degradation issues (such as the "middle collapse" phenomenon), and is not a fundamental solution.

[0008] 2. Semantic fragmentation problem To overcome the limitations of the context window, the mainstream approach is to segment long texts into multiple chunks. However, traditional fixed-length overlapping chunking strategies (e.g., 1024 tokens per chunk, with 200 overlapping tokens) are prone to interruption within sentences, paragraphs, or key semantic units, severely compromising information integrity. For example, if the first and second halves of a key clause are split into two chunks, the model cannot effectively understand and reason. Although the industry has attempted more refined strategies based on semantic boundaries (e.g., sentences, paragraphs) or recursive segmentation, effectively maintaining semantic coherence across chunks remains a core challenge for current technology.

[0009] 3. Processing efficiency and cost bottlenecks Dividing a long document into N blocks typically requires N large-scale API calls. This not only leads to slow processing speed (especially high latency from serial calls), but also causes the economic cost to increase linearly with the document length, making it difficult to apply to large-scale enterprise scenarios. Although asynchronous or concurrent calls can improve speed to some extent, brute-force concurrency can easily trigger the API's rate limit, causing requests to be rejected. Therefore, it is necessary to design additional complex flow control mechanisms.

[0010] 4. Inadequate complex document structure processing capabilities Professional long documents commonly contain structured elements with strong internal logic such as tables, formulas, code blocks, etc. Simple chunking operations can easily destroy these structures (such as cutting off a table in the middle), resulting in complete loss of their row and column relationships or syntax semantics, making the extraction result invalid. Existing general large models still have limited understanding and protection capabilities for such non-continuous text structures, and lack specialized techniques for identifying and maintaining their integrity during processing.

[0011] 5. Result integration and consistency challenges After independent information extraction on multiple chunks, the results obtained are scattered, redundant, and may conflict with each other. How to effectively de-duplicate, merge, and aggregate these local results into a globally consistent, accurate, and non-redundant final output is a complex post-processing problem. This often requires the construction of a special pipeline including entity alignment, confidence weighting, and semantic de-duplication, increasing system complexity and uncertainty.

[0012] Therefore, it is very important to design a large model information extraction and structure restoration system for long text documents with high precision, fast speed, controllable cost, and strong robustness. SUMMARY

[0013] The present application is to overcome the problems of low extraction accuracy, significant response delay, high economic cost, and insufficient practicality and stability of existing long text information extraction tasks, and provides a large model information extraction and structure restoration system for long text documents with high precision, fast speed, controllable cost, and strong robustness.

[0014] To achieve the above invention purpose, the present application adopts the following technical solutions: The large model information extraction and structure restoration system for long text documents, characterized by comprising: Define extraction points and text preprocessing modules to perform deep preprocessing and analysis on input unstructured documents, extract text and coordinate information, and identify and protect special structures. At the same time, an intelligent dynamic chunking strategy based on semantic boundaries is adopted, combined with a dynamic overlapping mechanism, to divide the input document into text blocks that maintain semantic integrity; High-concurrency processing and scheduling control module for batch processing of text blocks and calling large language models for distributed reasoning to output scattered results; Extraction reasoning and result fusion module for performing semantic de-duplication, entity alignment, and confidence fusion on the scattered results returned by the large language model to generate globally consistent structured output.

[0015] As preferred, in the defining extraction point and text preprocessing module, the defining extraction point specifically includes the following processes: According to the extraction point, set the text content extraction prompt word (prompt), and set the table content extraction prompt word according to the extraction point.

[0016] As preferred, in the defining extraction point and text preprocessing module, the text preprocessing specifically includes the following processes: OCR recognition and high-quality text extraction, specifically: For scanned copies or image type PDFs, use OCR engine for character recognition. In addition to extracting text content, the coordinates of each word block and font format information are also obtained. For digital PDFs, the internal text stream is directly parsed, and the position and style data of the characters are also retained; Special structure recognition and protection based on rules and machine learning, specifically: Table protection: By analyzing the horizontal and vertical alignment relationship of word blocks, detecting table lines, and using pre-trained table recognition models, all table regions in the document are located; Formula and code block protection: Based on regular expressions, mathematical formulas and program code fragments in the document are identified; Professional term and keyword protection: Build a domain-related professional dictionary library, scan the text during the preprocessing stage, and mark the key terms.

[0017] As preferred, in the defining extraction point and text preprocessing module, the intelligent block strategy combined with the dynamic overlap mechanism, specifically includes the following processes: Semantic boundary-based block, specifically: Select natural semantic boundaries as the preferred boundary for block; Dynamic overlap mechanism, specifically: For long paragraphs that cannot be avoided and must be segmented, the system automatically calculates the overlap area between blocks and blocks; The size of the overlap area is dynamically adjusted, that is, the overlap area automatically increases the number of overlapping tokens according to the semantic density of the front text; For protected special structures, separate block is adopted.

[0018] As preferred, the high-concurrency processing and scheduling control module adopts an asynchronous batch reasoning mechanism based on a priority queue, an intelligent traffic control strategy integrated with a fuse mechanism, and a failover and retry strategy; The asynchronous batch reasoning mechanism based on a priority queue is as follows: Request packaging and batch processing: The system dynamically packages multiple independent processing requests generated after intelligent block to form a batch request; Asynchronous non-blocking call: all batch requests are sent in asynchronous non-blocking mode; Priority queue scheduling: the system maintains a priority queue to manage all pending batch requests; the priority of the request is dynamically set according to business needs; Priority based on processing delay: requests with excessive waiting time will automatically have their priority increased.

[0019] As a preferred, the intelligent flow control strategy of the integrated fuse mechanism is as follows: Dynamic concurrency control: the system monitors the response status, response delay of large language model LLM API and the request queue depth of itself in real time, and based on the index feedback, the system dynamically adjusts the number of concurrent worker threads or the request sending rate to realize adaptive flow control; Token bucket algorithm for rate shaping: the system uses token bucket algorithm and based on the core parameters of the algorithm, according to the specific rate limit strategy of different LLM suppliers, dynamic configuration is carried out.

[0020] Fuse mechanism: when the system detects that the API error rate continuously exceeds the preset threshold or the response time is abnormally prolonged, the fuse mechanism will be triggered; the fuse temporarily stops sending requests to the API endpoint and enters the "cooling period"; during the "cooling period", the system tries to retry the failed requests or returns the degraded results; after the cooling period ends, the fuse is half open to send a small amount of requests tentatively, and if it is confirmed that the API service has recovered, the fuse is closed and normal calling is resumed.

[0021] As a preferred, the failover and retry strategy is as follows: Automatic retry: for transient failure requests caused by network fluctuations or server throttling, the system automatically retries a limited number of times.

[0022] As a preferred, the extraction reasoning and result fusion module includes a large model reasoning execution process, a result collection and preliminary analysis process, a confidence processing and quality filtering process, a semantic deduplication and cross-block entity alignment process, a special structure extraction result restoration and integration process, and a global result fusion and output process. The large model reasoning execution process is as follows: Standardized API call: receiving batch requests from the high-concurrency processing and scheduling control module, initiating reasoning requests to large language models through standardized interfaces; Response receiving and buffering: asynchronously receiving the original response returned by the large language model; the system has a buffering mechanism to cope with network fluctuations or server return delays; The result collection and preliminary analysis process is as follows: Concurrent result aggregation: Collect all the results returned by the asynchronous inference tasks using a high-performance concurrent queue; JSON parsing and error handling: Parse the returned JSON string to extract structured extraction results; mark and record non-standard formats, empty results, or parsing errors returned by large models.

[0023] As a preferred, the confidence processing and quality filtering process are as follows: Confidence score association: Assign a confidence score to each extraction result; Threshold-based filtering: Set a configurable confidence threshold; results below the confidence threshold will be automatically filtered or marked as low-confidence items; The semantic deduplication and cross-block entity alignment process is as follows: Semantic deduplication: For redundant results extracted from different text blocks describing the same entity, use a semantic embedding-based similarity calculation model to identify and deduplicate the content, and only keep the version with the highest confidence or the most complete information; Entity resolution and association: Establish cross-text block entity association relationships.

[0024] As a preferred, the special structure extraction result restoration and integration process is as follows: Structure restoration: Restore the structured metadata contained in the special protected tables and formulas to standard table JSON or LaTeX format; Context integration: Reinsert the restored special structure into the original document context and integrate it with the results extracted from pure text to form a complete output; Global result fusion and output, as follows: Result aggregation: Aggregate all local results after parsing, filtering, deduplication, association, and restoration into a unified data structure; Final consistency check: Perform final consistency checks, including checking if the required fields are complete and the numerical logic is self-consistent; Structured output: Generate the final structured data file for direct integration and utilization by downstream business systems.

[0025] Compared with the prior art, the present application has the following beneficial effects: (1) significantly improving the extraction accuracy: through intelligent dynamic blocking and semantic boundary protection mechanism, the semantic fragmentation problem caused by traditional blocking method is effectively avoided, and the key information is ensured to be completely retained; combined with special structure protection and multi-dimensional result fusion, the model illusion phenomenon is greatly reduced, and the accuracy and consistency of information extraction are improved; (2) greatly improving the processing efficiency and response speed: using high-concurrency asynchronous scheduling mechanism and intelligent flow control strategy, the API call delay and network overhead are significantly reduced, supporting large-scale concurrent processing, suitable for enterprise-level massive document processing scenarios; (3) significantly reducing the economic cost: through batch request processing, dynamic concurrency control and failover mechanism, the number of API calls and failure retry cost are effectively reduced, realizing a cost-controllable long text processing scheme; (4) enhancing the robustness and stability of the system: integrating the fuse mechanism, automatic retry and priority scheduling strategy, the system has good fault tolerance ability and service degradation mechanism, ensuring that the system can still run stably under high load or abnormal downstream service; (5) supporting complex document structure processing: having the ability to identify and protect special structures such as tables, formulas and code blocks, supporting lossless extraction of multi-level nested structures, expanding the applicability of the system in professional document processing; (6) improving user experience and ease of use: providing a visual configuration interface and Prompt automatic optimization function, reducing the user threshold, supporting flexible task customization and result visualization, improving the operability and practicality of the system. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 It is a schematic diagram of a framework of the long text document large model information extraction and structure restoration system of the present application; Figure 2 It is a flowchart of the high-concurrency processing and scheduling control module in the present application; Figure 3 It is a flowchart of the extraction reasoning and result fusion module in the present application; Figure 4 It is a flowchart of the long text document large model information extraction and structure restoration system provided by the embodiment of the present application applied to supply and marketing contract key information extraction; Figure 5 It is a text extraction result display schematic diagram of the prompt word process without LLM optimization in the embodiment of the present application; Figure 6 It is a text extraction result display schematic diagram after the prompt word process of LLM optimization in the embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to more clearly illustrate the embodiments of the present application, the specific implementation of the present application will be described below with reference to the drawings. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings and other embodiments can also be obtained according to these drawings without creative labor for those skilled in the art.

[0028] The present application provides a large model information extraction and structure restoration system for long text documents, and the core scheme is as follows: Firstly, the system performs deep preprocessing and analysis on the input PDF and other unstructured documents, extracts text and coordinate information through OCR recognition and layout analysis, and identifies special structures such as protected tables and formulas; secondly, an intelligent dynamic blocking strategy based on semantic boundaries is adopted, combined with a dynamic overlapping mechanism, to divide the long document into text blocks that maintain semantic integrity; thirdly, the text blocks are processed in batches through a high-concurrency asynchronous scheduling mechanism, and a large language model is called for distributed reasoning; finally, the multiple block results returned by the model are subjected to semantic deduplication, entity alignment and confidence fusion to generate globally consistent structured output.

[0029] Specifically, the system function architecture proposed by the present application is as shown in Figure 1 The overall system mainly includes the following modules, specifically as follows: Definition of extraction point and text preprocessing module: Among them, the definition of extraction point: according to the need to configure the extracted content, at this time two different prompt word processes are built in for tables and text. For optimization, considering that some users do not have a high degree of understanding of large models, this place increases the function module, calling the set large model to automatically optimize the prompt word process.

[0030] Text preprocessing: this process is the basis of the whole system processing flow, and its core task is to convert the original unstructured or semi-structured documents (such as scanned PDF, digital PDF, etc.) into high-quality standardized text data for subsequent intelligent blocking and large model reasoning. This process not only completes the traditional text extraction and cleaning, but also solves the difficult problems of special structure protection and semantic integrity maintenance in long documents through a series of fine processing strategies, and the specific process is as follows: 1. OCR recognition and high-quality text extraction For scanned copies or image-type PDFs, a high-precision OCR engine is used for character recognition, which not only extracts text content, but also synchronously acquires layout information such as the coordinates and font of each word block. For digital PDFs, the internal text stream is directly parsed, and the precise position and style data of characters are also retained. All these information is encapsulated into a unified Extract data class, providing data basis for subsequent structure analysis and intelligent blocking.

[0031] 2. Special structure recognition and protection based on rules and machine learning This is a key step in the preprocessing stage, aiming to identify and protect professional content that is extremely vulnerable to being destroyed in simple chunking, resulting in complete loss of semantics.

[0032] Table protection: By analyzing the horizontal and vertical alignment of word blocks, detecting table lines (solid or dashed), and using a pre-trained table recognition model, all table regions in the document are accurately located. Once identified, the entire table is treated as an independent, indivisible "semantic unit" or "protected object" that will be forced to remain within the same text block during subsequent chunking, avoiding being cut in the middle.

[0033] Formula and code block protection: Based on regular expressions, mathematical formulas and program code snippets in the document are identified. These contents are also marked as protected units to ensure their integrity.

[0034] Professional term and keyword protection: A domain-related professional dictionary library (such as legal clause names, medical terms, engineering parameters) is constructed, and the text is scanned in the preprocessing stage to mark these key terms. This measure not only helps to protect important information, but also provides semantic anchors for subsequent intelligent chunking.

[0035] 3. Dynamic overlapping intelligent chunking strategy Based on the protection of special structures, the system adopts a dynamic chunking strategy to replace the traditional fixed-length chunking.

[0036] Chunking based on semantic boundaries: The preferred boundary for chunking is the natural semantic boundary, such as the end of a paragraph, chapter title, etc. This ensures that most blocks contain complete semantic paragraphs.

[0037] Dynamic overlapping mechanism: For long paragraphs that cannot be avoided and must be divided, the system calculates the overlapping area between blocks. The size of this overlapping area is not fixed, but dynamically adjusted: it automatically increases the number of overlapping tokens according to the semantic density of the front text (such as containing multiple professional terms or entities) to ensure that critical contextual information is not lost due to chunking. This dynamic mechanism maximizes the alleviation of semantic fragmentation while ensuring efficiency. For protected special structures that have already been processed in the previous step, separate chunking is adopted. Although this will increase the number of large model calls and increase the response time of the entire system, it can protect the integrity of the semantic structure and significantly improve the accuracy of extraction.

[0038] 4. Text preprocessing and normalization The extracted raw text is cleaned, including removing meaningless random codes, standardizing spaces, and unifying encodings.

[0039] At the same time, basic natural language processing tasks such as sentence boundary detection and word segmentation are performed to provide preparation for creating high-quality prompt words.

[0040] Through the processing of this module, the original document is transformed into a series of text units rich in semantic and structural information and protected, laying a solid data foundation for subsequent high-precision information extraction and effectively preventing model hallucination problems caused by format chaos or structural damage from the source.

[0041] High-concurrency processing and scheduling control module: This module is the core module of the present application, aiming to efficiently, stably and at low cost schedule a large number of calls to the Large Language Model (LLM) API. It systematically solves the engineering problems of many requests, rate limits, high call costs, etc. in long document processing through an integrated asynchronous batch inference mechanism and intelligent traffic control strategy. The module process is specifically as shown in Figure 2 , and specifically as follows: 1. Asynchronous batch inference mechanism based on priority queue Request packaging and batch processing: The system dynamically packages multiple independent processing requests (including text blocks and corresponding prompt word processes) generated after intelligent chunking to form a batch request. This batch processing method can significantly reduce the number of network round trips to the large model service, thereby reducing the overall request delay.

[0042] Asynchronous non-blocking call: All batch requests are sent in asynchronous non-blocking mode. The main processing thread will not be blocked by a single API response, and can continuously prepare and send subsequent requests, greatly improving the overall throughput and resource utilization of the system.

[0043] Priority queue scheduling: The system maintains a priority queue to manage all batch requests to be processed. The priority of the request can be dynamically set according to business needs, for example: priority based on content criticality: text blocks containing important chapters or high-confidence key terms can be given higher priority to ensure the timeliness of key information extraction.

[0044] Priority based on processing delay: Requests with excessive waiting time will automatically have their priority increased to prevent some requests from being queued for too long.

[0045] 2. Intelligent traffic control with integrated fuse mechanism Dynamic concurrency control: The system monitors the response status (such as HTTP 429, 503 error codes), response delay, and request queue depth of the LLM API in real time. Based on these feedback indicators, the system dynamically adjusts the number of concurrent worker threads or the request sending rate to achieve adaptive traffic limiting.

[0046] The token bucket algorithm implements rate shaping: The system uses the token bucket algorithm to finely "shape" the request sending rate. The core parameters of the algorithm (token generation rate, bucket capacity) can be dynamically configured according to the specific rate limiting policies of different LLM providers. This ensures that request traffic is sent smoothly and evenly, avoiding API-side restrictions triggered by excessive instantaneous traffic, thus guaranteeing continuous service availability.

[0047] Circuit Breaker Mechanism: When the system detects that the API error rate consistently exceeds a preset threshold or the response time is abnormally prolonged, the circuit breaker mechanism will be automatically triggered. The circuit breaker will temporarily stop sending requests to the API endpoint, entering a short "cooling-off period." During the cooling-off period, the system can attempt to retry failed requests or return a degraded result. After the cooling-off period ends, the circuit breaker will half-open to send a small number of tentative requests. If it is confirmed that the API service has recovered, the circuit breaker will close, and normal calls will resume. This mechanism effectively prevents system cascading failures caused by downstream service instability.

[0048] 3. Failover and Retry Strategies Automatic retry: For requests that fail momentarily due to network fluctuations or server-side rate limiting, the system will automatically retry a limited number of times.

[0049] Through the design of this module, this invention achieves refined control over large-scale LLM calls with "high concurrency, low latency, high stability, and low cost," enabling it to handle enterprise-level processing tasks of massive long documents and breaking through the performance and stability bottlenecks of traditional serial or simple concurrent call methods.

[0050] Extraction, Inference, and Result Fusion Module: This module is the core execution layer for large-scale asynchronous inference tasks. It is responsible for efficiently and reliably completing large model calls and deeply processing and fusing the returned scattered results, ultimately generating high-quality, structured global output. Its design focuses on handling the challenges of result integrity, accuracy, and consistency in high-concurrency environments. The specific workflow of this module is as follows: Figure 3 As shown, the details are as follows: 1. Large model inference execution Standardized API calls: Receive batch requests from the scheduling module and initiate inference requests to large language models (such as GPT-4, Claude, etc.) through standardized interfaces. The request content includes the prompt word extraction process and corresponding preprocessed text blocks, designed after multiple experimental verifications and debugging.

[0051] Response Reception and Buffering: The system receives the raw response (usually in JSON format) from the asynchronous receiving model. A buffering mechanism is in place to handle network fluctuations or server response delays, ensuring that an exception in a single request does not block the overall processing flow.

[0052] 2. Results Collection and Preliminary Analysis Concurrent Results Aggregation: A high-performance concurrent queue is used to collect the results returned by all asynchronous inference tasks. This queue features high throughput and persistence, ensuring no data is lost even under heavy requests.

[0053] JSON parsing and error handling: The returned JSON string is parsed to extract structured results. Strict exception handling is performed at this stage. Mark and record non-standard formats, empty results, or parsing errors returned by the model.

[0054] Trigger an automatic retry mechanism or add the case to the queue for manual review to ensure the robustness of the processing flow.

[0055] 3. Confidence processing and quality filtering Confidence score association: Assign a confidence score to each extraction result. This score can be derived from the model's own output (if supported), the logical consistency check of the extraction results, or the degree of matching with the domain dictionary.

[0056] Threshold-based filtering: Set a configurable confidence threshold. Results below the threshold will be automatically filtered or marked as low-confidence items, which can either be passed to subsequent modules for semantic verification or directly excluded, effectively controlling the downward propagation of noise.

[0057] 4. Semantic deduplication and cross-block entity alignment Semantic deduplication: For redundant results describing the same entity extracted from different text blocks, a similarity calculation model based on semantic embedding (Sentence-BERT, etc.) is used for identification and deduplication. Only the version with the highest confidence or the most complete information is retained.

[0058] Entity parsing and association: Establishing entity relationships across text blocks. For example, "Party A's Name" extracted from "Chapter 1" needs to be aligned and merged with "Contract Signatory A" extracted from "Chapter 5," ultimately associating them under the same entity ID. This is a crucial step in building a globally consistent knowledge graph.

[0059] 5. Restoration and Integration of Results Extracted from Special Structures Structure Restoration: For tables, formulas, and other content that are specially protected during the preprocessing stage, the extraction results typically contain structured metadata (such as row and column indices, formula numbers). This module is responsible for restoring this structured data to standard tabular JSON or LaTeX format to ensure its usability.

[0060] Context integration: The special structure that has been restored is placed back into its original document context and merged with other results extracted from plain text to form a complete output.

[0061] 6. Global Result Fusion and Output Result aggregation: Aggregate all parsed, filtered, deduplicated, correlated, and restored local results into a unified data structure (usually JSON or XML).

[0062] Eventual consistency check: Performs an eventual consistency check, such as checking whether required fields are complete and whether the numerical logic is consistent (e.g., whether the total amount equals the sum of the individual items).

[0063] Structured output: Generates final structured data files that can be directly integrated and utilized by downstream business systems (such as databases, CRM, and BI tools). The output format supports flexible configuration to meet different integration needs.

[0064] Through the in-depth processing of this module, scattered, raw, and potentially conflicting model outputs are transformed into clean, complete, and globally consistent high-value structured data, marking the completion of the final transformation from unstructured long documents to trustworthy and usable data.

[0065] Based on the technical solution of this invention, the implementation process of this invention in practical applications will be illustrated through the following case scenarios. The specific application implementation scheme is as follows: Application scenario: Extraction of key information from supply and sales contracts. The specific processing flow is as follows: Figure 4 As shown, the details are as follows: 1. Input document: Document type: PDF format supply and sales contract Document length: 200 pages Content features: Contains a large amount of structured and unstructured content, such as contract terms, technical specifications, price lists, signature and seal pages, appendices, etc., with a total of 225,405 characters, far exceeding the context window limit of current mainstream large language models (such as GPT-4Turbo) (usually 128K tokens).

[0066] 2. Document preprocessing: Performs high-precision OCR recognition and text extraction on PDFs, preserving layout information such as word block coordinates, fonts, and typesetting.

[0067] Identify and protect key structures in contracts, such as forms, clause headings, monetary figures, dates, and legal terms.

[0068] The extracted text content is converted to Markdown format to retain some structural information (such as heading levels, lists, bold text, etc.) to facilitate understanding by the subsequent large model.

[0069] Users define 109 extraction points through the system interface, including: Standard sampling points: contract number, name of Party A / Party B, total contract amount, signing date, delivery deadline, etc. Complex semantic extraction points: objection handling clauses, acceptance criteria, liability for breach of contract, confidentiality agreements, ownership of intellectual property rights, etc.

[0070] 3. Intelligent block division: A multi-level semantic segmentation strategy is adopted: Prioritize segmentation by chapter title (e.g., "Chapter 1 General Provisions", "Chapter 2 Product Delivery"); second priority is segmentation by paragraph boundaries to ensure semantic integrity; for very long paragraphs (e.g., technical appendices), a dynamic overlap mechanism is used, with the overlapping area automatically adjusted according to semantic density; special structural protection: all tables, code blocks, formulas, etc. are treated as indivisible units and formed into separate blocks to ensure their structural integrity.

[0071] 4. High-concurrency scheduling: The segmented text and the corresponding extraction prompt word process are dynamically packaged into batch tasks; a priority queue scheduling mechanism is used: High priority: Blocks involving amounts, dates, and key terms; Medium priority: General description content; Low priority: appendices, references, etc.; By combining the token bucket algorithm and the circuit breaker mechanism, the number of concurrent requests is dynamically controlled to avoid triggering API rate limiting.

[0072] 5. Large-scale model reasoning: By using asynchronous batch calls, multiple text blocks are merged into a single batch request, significantly reducing the number of network round trips; GPT-4 is called to extract information, and each block returns a preliminary structured result (JSON format). The inference component can additionally utilize LLM to automatically optimize user-provided prompts, even for prompts from inexperienced users, and in some cases, this module can still significantly improve accuracy. See below for specific comparison results. Figure 5 and Figure 6 . Figure 5 This shows the extraction results without the automatic prompt word optimization module. Figure 6 The extraction results are shown after incorporating the automatic prompt word optimization module. A comparison reveals that... Figure 5 There are problems such as incorrect sampling, missed sampling, and incomplete sampling. Figure 6 The extraction results are more complete and accurate, resulting in a significant improvement in overall performance. The system algorithm interface records the extraction confidence of each block for use in subsequent fusion stages.

[0073] 6. Results fusion and post-processing: Semantic deduplication: Based on TF-IDF and cosine similarity algorithms, duplicate or highly similar terms in multiple blocks are merged; Entity alignment: Identifying and merging the same entity (e.g., "Party A: a certain technology company") that is scattered in different blocks; Confidence-weighted fusion: The results of the same extraction point in multiple blocks are weighted by confidence level to filter low-confidence segments; 7. Output results: Generate structured JSON output, including the content, confidence level, and occurrence position (page number, block ID) of each extraction point; provide a visual interface to display the extraction results, and support clicking to view original text fragments, improving interpretability and user experience.

[0074] Effect comparison: (1) Sampling accuracy Traditional methods suffer from severe semantic fragmentation, with key information scattered across different text blocks, causing the model to be unable to understand the complete context, resulting in an accuracy of only about 60%. This invention, through intelligent segmentation and semantic protection mechanisms, ensures the integrity of key information, improving accuracy to 85%, a relative improvement of about 25%.

[0075] (2) Processing time Traditional methods, employing serial or simple concurrent calls, require over 300 seconds to process a 200-page document. This invention, through high-concurrency scheduling, batch request processing, and asynchronous inference mechanisms, reduces processing time to 40 seconds, significantly improving efficiency.

[0076] (3) Number of API calls Traditional fixed-block methods require individual API calls for each text block, totaling 200 calls. This invention reduces unnecessary segmentation through intelligent blocking and, combined with batch processing technology, reduces the number of calls to 100, a 50% reduction.

[0077] (4) Table extraction support Traditional methods are ineffective at handling structured content such as tables, often resulting in fragmented tables and information loss. This invention possesses table recognition and protection capabilities, can completely extract table content, and supports the processing of complex document structures.

[0078] (5) Semantic deduplication function Traditional methods lack cross-block result fusion mechanisms, resulting in a large amount of redundant and conflicting information in the output. This invention introduces a multi-dimensional result fusion mechanism to achieve semantic-level deduplication, ensuring the consistency and integrity of the output results.

[0079] (6) System stability Traditional methods are prone to triggering API rate limits in high-concurrency scenarios, leading to request rejections and poor system stability. This invention integrates intelligent flow control, circuit breaking mechanisms, and dynamic concurrency regulation, which can adapt to the status of downstream services and significantly improve system robustness and fault tolerance.

[0080] The above comparative results demonstrate that this invention significantly outperforms traditional fixed-block methods in terms of extraction accuracy, processing efficiency, economic cost, and system stability, making it particularly suitable for enterprise-level large-scale long document processing scenarios. This system performs exceptionally well in long text information extraction tasks, especially in handling complex and lengthy commercial contracts, maintaining high accuracy while significantly improving processing efficiency and system stability, thus possessing strong engineering implementation and commercialization value.

[0081] This invention proposes a method and system for efficient information extraction from large-scale, ultra-long texts in Intelligent Document Processing (IDP), effectively solving several core problems of large models in long text processing: 1) model illusion caused by ultra-long text input; 2) semantic fragmentation caused by traditional block-based methods; and 3) slow inference speed during ultra-long text extraction. Through deep optimization of the large-scale model processing mechanism, this invention significantly improves processing efficiency while maintaining high accuracy, providing a stable and reliable solution for processing hundreds-of-pages-long texts such as contracts and technical documents.

[0082] The innovative aspects of this invention are as follows: 1. Semantic-aware intelligent segmentation mechanism Employing a multi-level segmentation strategy (prioritizing sentence boundaries, followed by logical connectors, and finally protecting special structures), combined with terminology protection technology, it automatically identifies and maintains the integrity of domain-specific terminology. The built-in data module supports custom domain-specific keywords and retrieves them in real-time during the extraction process. Through a dynamic overlap compensation mechanism, it ensures the semantic coherence across blocks, effectively solving the semantic fragmentation problem caused by traditional fixed-length segmentation, thereby significantly improving the accuracy of long text processing.

[0083] 2. Dynamic Concurrency Control System By monitoring key performance indicators such as response latency and error rate in real time, and dynamically allocating process tokens, intelligent concurrency optimization based on historical performance data is achieved. Combined with an adaptive rate control mechanism, it can automatically reduce and recover when encountering API limitations; and with the token bucket algorithm, it provides fine-grained management of request traffic, improving system throughput efficiency while ensuring service stability. An efficient asynchronous processing mechanism further reduces I / O wait time, significantly improving overall processing speed.

[0084] 3. Multi-dimensional result fusion mechanism Cross-block entity association technology can merge related entities using string similarity to solve the problem of reorganizing scattered entities; confidence-weighted fusion algorithm can filter low-confidence segments; semantic deduplication engine achieves semantic-level deduplication based on semantic similarity detection algorithm and cosine similarity, ensuring the integrity and consistency of the final extraction results.

[0085] 4. Special structural protection system It has the ability to automatically identify and isolate tables, formulas and code blocks, and adopts structured data lossless extraction technology to ensure content integrity; it supports the identification and protection of special structures with multiple nested layers, and achieves accurate processing of complex document structures.

[0086] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A large model information extraction and structure restoration system for long text documents, characterized by, The application relates to a large language model processing method and device. The application comprises the following steps: Defining an extraction point and a text preprocessing module are used for deep preprocessing and analysis of an input unstructured document, and text and coordinate information are extracted and special structures are identified; meanwhile, an intelligent dynamic block strategy based on semantic boundaries is adopted, and a dynamic overlapping mechanism is combined to divide the input document into text blocks that maintain semantic integrity; A high-concurrency processing and scheduling control module is used for batch processing of the text blocks and calling a large language model for distributed reasoning to output scattered results; 2.The system for large model information extraction and structure restoration of long text documents according to claim 1, wherein, An extraction reasoning and result fusion module is used for semantic deduplication, entity alignment and confidence fusion of the scattered results returned by the large language model to generate a globally consistent structured output. In the defining extraction point and text preprocessing module, the defining extraction point specifically comprises the following steps: 3.The system for large model information extraction and structure restoration of long text documents according to claim 2, characterized in that, Text content extraction prompt words are set according to the extraction point, and table content extraction prompt words are set according to the extraction point. In the defining extraction point and text preprocessing module, the text preprocessing specifically comprises the following steps: OCR recognition and high-quality text extraction, specifically: For scanned copies or image-type PDFs, an OCR engine is used for character recognition, and in addition to extracting text content, the coordinates, font style information of each word block are also acquired; for digital PDFs, internal text streams are directly parsed, and the position and style data of characters are also retained; Special structure identification and protection based on rules and machine learning, specifically: Table protection: all table regions in the document are located by analyzing the horizontal and vertical alignment relationship of word blocks, detecting table lines and using a pre-trained table identification model; Formula and code block protection: mathematical formulas and program code segments in the document are identified based on regular expressions; 4.The system for large model information extraction and structure restoration of long text documents according to claim 3, wherein, Professional term and keyword protection: a domain-related professional dictionary library is constructed, and the text is scanned in the preprocessing stage, and key terms are marked. In the defining extraction point and text preprocessing module, the intelligent block strategy combined with the dynamic overlapping mechanism specifically comprises the following steps: Semantic boundary-based block, specifically: Natural semantic boundaries are selected as the preferred boundaries for block division; Dynamic overlapping mechanism, specifically: For long paragraphs that cannot be avoided and must be divided, the system automatically calculates the overlapping area between blocks and blocks; the size of the overlapping area is dynamically adjusted, that is, the number of overlapping tokens in the overlapping area is automatically increased according to the semantic density of the front text; 5.The system for large model information extraction and structure restoration of long text documents according to claim 4, characterized in that, For protected special structures, a separate block is adopted. The high-concurrency processing and scheduling control module adopts an asynchronous batch reasoning mechanism based on a priority queue, an intelligent traffic control strategy integrated with a fuse mechanism, and a fault transfer and retry strategy; The asynchronous batch reasoning mechanism based on the priority queue is specifically as follows: Request packaging and batch processing: the system dynamically packages multiple independent processing requests generated after intelligent block division to form a batch request; Asynchronous non-blocking call: all batch requests are sent in an asynchronous non-blocking mode; Priority queue scheduling: the system maintains a priority queue for managing all batch requests to be processed; the priority of the request is dynamically set according to business requirements; Priority based on processing delay: requests with excessive latency are automatically promoted in priority. 6.The system for large model information extraction and structure restoration of long text documents according to claim 5, wherein, The intelligent flow control strategy integrated with the fuse mechanism is as follows: Dynamic concurrency control: the system monitors the response state, response delay of the large language model LLM API and the request queue depth of the system in real time, and dynamically adjusts the number of concurrent worker threads or the request sending rate based on the index feedback, to achieve adaptive flow control; Token bucket algorithm for rate shaping: the system uses the token bucket algorithm and dynamically configures based on the core parameters of the algorithm according to the specific rate limiting strategy of different LLM providers; Fuse mechanism: when the system detects that the API error rate continuously exceeds the preset threshold or the response time abnormally prolongs, the fuse mechanism is triggered; the fuse temporarily stops sending requests to the API endpoint and enters the "cooling period"; during the "cooling period", the system tries to retry the failed requests or returns a degraded result; after the cooling period ends, the fuse is half open to send a small amount of requests tentatively, and if it is confirmed that the API service has recovered, the fuse is closed to restore normal calling.

7. The large model information extraction and structure restoration system for long text documents according to claim 6, characterized by, The failover and retry strategy is as follows: Automatic retry: for transient failed requests caused by network fluctuations or server flow control, the system automatically retries a limited number of times. 8.The system for long text document's large model information extraction and structure restoration according to claim 7, wherein, The extraction reasoning and result fusion module includes a large model reasoning execution process, a result collection and preliminary analysis process, a confidence processing and quality filtering process, a semantic deduplication and cross-block entity alignment process, a special structure extraction result restoration and integration process, and a global result fusion and output process. The large model reasoning execution process is as follows: Standardized API calling: receiving batch requests from the high-concurrency processing and scheduling control module, initiating reasoning requests to the large language model through standardized interfaces; Response receiving and buffering: asynchronously receiving the original response returned by the large language model; the system has a buffering mechanism to cope with network fluctuations or server return delays; The result collection and preliminary analysis process is as follows: Concurrent result aggregation: using a high-performance concurrent queue to collect the results returned by all asynchronous reasoning tasks; JSON parsing and error handling: parsing the returned JSON string to extract structured extraction results; marking and recording non-standard formats, empty results or parsing errors returned by the large model. 9.The system for long text document big model information extraction and structure restoration according to claim 8, wherein, The confidence processing and quality filtering process is as follows: Confidence score association: assigning a confidence score to each extraction result; Threshold-based filtering: setting a configurable confidence threshold; results below the confidence threshold will be automatically filtered or marked as low-confidence items; The semantic deduplication and cross-block entity alignment process is as follows: Semantic deduplication: for redundant results describing the same entity extracted from different text blocks, a similarity calculation model based on semantic embedding is used to identify and deduplicate the content, retaining only the version with the highest confidence or the most complete information; Entity analysis and association: establishing cross-text block entity association relationships. 10.The system for large model information extraction and structure restoration of long text documents according to claim 9, wherein, The special structure extraction result restoration and integration process is as follows: Structure restoration: restore the special-protected tables and structured metadata contained in the formula into standard table JSON or LaTeX format; Context integration: re-integrate the restored special structure into the original document context and merge with the results extracted from the plain text to form a complete output; Global result fusion and output, as follows: Result aggregation: aggregate all local results after parsing, filtering, deduplication, association and restoration into a unified data structure; Final consistency check: perform final consistency check, including checking whether the mandatory fields are complete and the numerical logic is self-consistent; Structured output: generate the final structured data file for direct integration and utilization by downstream business systems.

Images