A structure adaptive dual-mode long document intelligent slicing method

Abstract

The application discloses a kind of structural adaptive dual-mode long document intelligent slicing method, the present application is aggregated from bottom to top by adaptive computing power scheduling semantics, using multi-agent relay abstract strategy, construct three-layer semantic index tree of super-long basic text, atomic abstract, global high-order index abstract, provide metadata support for subsequent context injection of slice, finally through the way of cutting with end punctuation backtracking and context injection, generate final physical slice, effectively realize the adaptive perception and accurate processing of strong structured long document, weak structured long document, can automatically balance processing efficiency and cutting precision, avoid the semantic break and context loss caused by traditional mechanical cutting, and by recording full-link traceability information and hierarchical abstract enhancement, significantly improve the semantic integrity of final physical slice and downstream retrieval recall rate, reduce illusion risk, ensure the high quality and interpretability of RAG knowledge base construction.

Description

Technical Field

[0001] This invention relates to the field of long document processing technology, and specifically to a structure-adaptive dual-mode intelligent slicing method for long documents. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, Retrieval Augmentation (RAG) has become a core paradigm for solving the model "illusion" problem and processing private domain knowledge. In RAG systems, document slicing is a key step in transforming unstructured documents into searchable units in a vector database. The quality of its slicing directly determines the recall rate of downstream knowledge retrieval and the accuracy of answer generation. Currently, existing document slicing technologies mainly rely on mechanical rule-based slicing when processing long documents such as financial research reports, legal documents, and technical manuals. This includes rigid granularity of slicing, mainstream fixed character count slicing, or simple punctuation slicing.

[0003] Traditional segmentation methods are prone to forcibly truncating long sentences or logical paragraphs, resulting in fragmented semantics, semantic breaks, or loss of context, leading to low segmentation accuracy. At the same time, due to the lack of complete contextual information during retrieval, semantic integrity is compromised, and downstream retrieval recall is often low, posing a risk of illusion and ultimately compromising the quality of the RAG knowledge base.

[0004] Therefore, it is necessary to invent a structurally adaptive dual-mode intelligent slicing method for long documents to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a structure-adaptive dual-mode intelligent slicing method for long documents. By constructing a three-layer semantic index tree consisting of ultra-long base text, atomic summaries, and global high-order index summaries, metadata support is provided for context injection of subsequent slicing. Through a slicing method with sentence-end punctuation backtracking and context injection, the final physical slices are generated. This effectively achieves adaptive perception and accurate processing of both strongly structured and weakly structured long documents. By recording full-link traceability information and hierarchical summary enhancement, the high quality and interpretability of the RAG knowledge base construction are ensured, thus addressing the aforementioned shortcomings in the technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a structure-adaptive dual-mode intelligent slicing method for long documents, comprising the following steps:

[0007] Step 1: Receive the PDF document to be processed, extract the text content and insert page number anchor marks according to the physical pages to build the original text stream;

[0008] Step 2: Perform basic repair of the original text stream to obtain a text file;

[0009] Step 3: Sample several pages evenly from the text file, perform parallel analysis using a large language model, and determine the pattern routing;

[0010] Step 4: Perform intelligent noise cleaning and directory blocking on the text file to obtain denoised text;

[0011] Step 5: Perform dual-mode differential preprocessing on the denoised text to obtain ultra-long basic text blocks;

[0012] Step 6: Perform hierarchical restructuring on the excessively long basic text blocks to generate restructured hierarchical text;

[0013] Step 7: Generate the final physical slice based on the reorganized hierarchical text.

[0014] The aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 1, receives the PDF document to be processed, extracts the text content, inserts page number anchor marks according to physical pages, and constructs the original text stream. The specific steps are as follows:

[0015] 1.1. Call the built-in PDF parsing engine, load the PDF document to be processed in binary read-only mode, establish the memory mapping of the file object model, and prepare for industry-level traversal;

[0016] 1.2 Perform a page-by-page linear scan of the PDF document to be processed, obtain the physical page number index of the current page in real time during the traversal, and generate a uniquely identified source anchor string;

[0017] The source tracing anchor string is set to: >>>Page n<<<;

[0018] 1.3. Forcefully inject the source anchor string into the starting position of the extracted text on the current page, and add a standard line break at the end of the page to block the unnatural adhesion of cross-page text;

[0019] 1.4 After all pages have been traversed, all processed single-page text in the buffer is concatenated into a continuous global string according to the physical page number order, thus constructing an original text stream with full-link traceability capability.

[0020] The aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 2, performs initial repair of the original text stream to obtain a text file. The specific steps are as follows:

[0021] 2.1 Construct a non-semantic line break feature library to identify unnatural line breaks caused by PDF layout, while strictly preserving semantically separated line breaks;

[0022] Among them, the non-semantic line break feature library includes paired symbol closure detection, numerical unit continuity detection, and line-end non-sentence-end punctuation detection;

[0023] 2.2. Traverse the current line of the original text stream and obtain its next adjacent line. Call the regular expression matching function to match and determine the connection between the current line and the next line with the non-semantic line break feature library to obtain the text file.

[0024] The regular expression matching function's decision logic is as follows:

[0025] ;

[0026] in, For the currently processed line of text, The next adjacent row; A Boolean function for the current line; A Boolean function for two adjacent rows; A Boolean function for the current line; ( ) represents the merge function; For logical OR.

[0027] In the aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 2.1, the pair symbol closure detection is used to detect whether there are unclosed punctuation marks or parentheses at the end of a line. The specific rule is: count the number of symbols on the left and right sides of the current text line. If the number of symbols on the left is greater than the number of symbols on the right, then the end of the line is determined to be unclosed and belongs to non-semantic line break.

[0028] Numerical unit continuity detection is used to prevent numerical values ​​from being forcibly split from the following units, time, or numerical parts by newline characters. The specific rules are as follows: use a regular expression matching function to determine whether the end of the current line is an Arabic numeral, and then use a regular expression matching function to check whether the beginning of the next line is a specific unit symbol or a continuation of numbers.

[0029] The line-end-not-sentence-end punctuation detection is used to identify situations where pause punctuation marks in the middle of a sentence are cut off by a newline character. The specific rule is: detect whether the character at the end of the current line belongs to the preset set of sentence punctuation marks. This line-end-not-sentence-end punctuation detection strictly excludes sentence-end punctuation marks to ensure that paragraphs with clear semantic end marks are not merged.

[0030] In step 2.2, the matching result determination rules are as follows:

[0031] If the combination of the current line and the next line matches any item in the non-semantic line break feature library, the line break character between the two lines will be deleted, and the content of the next line will be appended to the end of the previous line to form a new merged text line. After the merge occurs, the system will record the merge operation and continue to use the merged new text line as the basis for a new round of feature judgment with subsequent lines until no new merge rules are triggered in the whole round.

[0032] If the non-semantic line break feature library is not matched, the current line break character is retained, the line index is moved down, and the next pair of text lines is processed until the full text scan is completed. Finally, a text file with formatting errors corrected and paragraph structure preserved is output.

[0033] The aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 3, uniformly samples several pages from the text file, performs parallel analysis using a large language model, and determines the mode routing. The specific steps are as follows:

[0034] 3.1 Perform equal-step sampling on the text file, according to the preset window step size. This process splits a text file into a sequence of consecutive page groups. Extract each group The plain text of multiple consecutive pages contained within is concatenated in physical order to construct a structured detection prompt word containing the content of consecutive pages;

[0035] in, , The total number of groups, ;

[0036] 3.2 Grouping the pages into sequences The input is fed into a parallel inference engine based on the SGLang framework, and a batch concurrency mechanism is used to extract data each time. The first group is simultaneously sent to the large language model, which, based on the structure detection cue word instruction, identifies whether there are first-level and second-level headings in the sampled group, and finally outputs the first group. Structural feature vectors of each group ;

[0037] in, , Indicates whether a first-level heading exists. Indicates whether a second-level heading exists;

[0038] 3.3 Calculate the structured cumulative confidence score of the text file. The specific formula is as follows:

[0039] ;

[0040] in, The weight of a first-level heading, specifically... , The weight of the second-level heading is as follows: ;

[0041] 3.4 Define the dynamic routing decision function An early stopping mechanism is introduced to perform route determination and determine the mode route. The specific formula is as follows:

[0042] ;

[0043] in, To set a preset threshold, the confidence level is accumulated during the traversal process. Exceeding the preset threshold , specific The system immediately terminates subsequent detection, determines that the document is strongly structured, and activates the title processing mode. That is, for strongly structured documents with explicit hierarchical identifiers, the extracted title features are used as physical anchors to perform top-down physical segmentation and hierarchical tree reconstruction flow branches.

[0044] If all have been traversed If the document still fails to meet the standard after being grouped, it is judged as weakly structured and the untitled processing mode is activated. That is, for weakly structured documents that lack explicit hierarchical identifiers, the semantic connectivity between adjacent paragraphs is analyzed using a large language model, and the flow branches are iteratively stitched together from bottom to top.

[0045] The aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 4, performs intelligent noise cleaning and directory blocking on the text file to obtain denoised text. The specific steps are as follows:

[0046] 4.1 The large language model performs directory index feature recognition on the sampled page based on prompt words, determines whether the current page is a directory page, and removes directory pages;

[0047] The prompts include numeric guide lines, dotted lines, and page number indexes. If the large language model detects a prompt, it determines that the current page is a directory page. The system directly removes all content of that page from the text file and simultaneously updates the physical index mapping of the remaining pages.

[0048] 4.2. Process the page sequence after removing the table of contents in batches, input the page text into a large language model for layout structure analysis, and output the suspected header of each page. and suspected footer ;

[0049] 4.3 Perform global word frequency statistics and voting on all identified suspected headers and footers to determine the header noise features with the highest recurrence rate. Batch remove these features across the entire text, followed by local cleaning. Directly call the independent suspected header and footer identification records of each page and perform a second scan on the globally cleaned pages to accurately remove page-specific non-common noise elements that are not covered by global features, thus obtaining denoised text.

[0050] In the aforementioned structure-adaptive dual-mode long document intelligent slicing method, step 5 involves obtaining ultra-long basic text blocks, which include semantically coherent basic text blocks or structurally standardized basic text blocks. The specific steps for obtaining these blocks are as follows:

[0051] 5.1 If the mode routing is set to the untitled processing mode, the output denoised text will be segmented into initial paragraph sequences based on newline characters. It then initiates a multi-round parallel semantic iterative merging process, in which adjacent text paragraph pairs are traversed in each round of iteration. The decision on whether to merge sentences is made using a dual-validation model of "hard constraints at the end of the sentence and soft semantic discrimination". The specific formula for the merging decision function is as follows:

[0052] ;

[0053] in, ;1 , , For the set of sentence terminators, if paragraph If the sentence ends with a punctuation mark from the set of sentence terminators, then the indicator function... Returning to 0 forces a block on merging to prevent erroneous concatenation of different semantic paragraphs;

[0054] For indicator functions, This is a semantic connectivity discrimination function based on a large language model. The specific discrimination result includes yes or no, where yes indicates that it can be merged and no indicates that it cannot be merged. During the discrimination, the system iterates repeatedly until no new merging is generated, and outputs semantically coherent basic text blocks.

[0055] Sentence-end hard constraints refer to using a pre-defined set of sentence-end terminators for low-cost rule pre-interception; if a rule is hit, it will be directly blocked and merged.

[0056] Semantic soft discrimination refers to using a large language model to perform in-depth analysis of the semantic context connectivity of adjacent paragraphs when allowing a rule to be passed.

[0057] 5.2 If the mode is routed to a mode with a title, execute a physical segmentation strategy based on title features and output a basic text block with standardized structure.

[0058] The aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 5.2, if the mode routing is to the heading processing mode, executes a physical segmentation strategy based on heading features, and outputs structurally standardized basic text blocks. The specific steps are as follows:

[0059] 5.2.1 Coarse-grained pre-segmentation: Based on the context window constraint of the large language model, and utilizing a length threshold and sentence-end punctuation backtracking mechanism, long documents are pre-segmented into a set of temporary fragments. ;

[0060] 5.2.2 Title Feature Extraction: [The remaining text appears to be incomplete and requires further context.] Parallel input is used to a large language model, extracting the text of first-level and second-level headings to generate an ordered heading sequence. ;

[0061] 5.2.3 Flexible Anchoring and Physical Segmentation: An flexible string matching algorithm is used, ignoring spaces and differences between full-width and half-width characters, to locate the title in the original document stream. Physical starting index Based on this, the document is divided into independent sub-blocks, each starting with a title. ;

[0062] 5.2.4 For each independent sub-block Detect its header text and locate the corresponding title. Physically remove the title text from the independent sub-block so that only the plain text content remains, thereby eliminating semantic redundancy and interference from the title text during subsequent summary extraction.

[0063] 5.2.5 Fragmentation and Fractionation: Detects micro-text blocks whose length is less than a preset value after segmentation, merges them back into the preceding text blocks, and outputs a basic text block with a standardized structure.

[0064] The default value is 50 characters.

[0065] The aforementioned structure-adaptive dual-mode long document intelligent slicing method performs hierarchical restructuring on the ultra-long basic text blocks in step 6 to generate restructured hierarchical text. The specific steps are as follows:

[0066] 6.1 For extremely long basic text blocks, a multi-agent relay summarization strategy is adopted, using the summary of the previous segment as context to guide the generation of the summary of the current segment, ultimately producing an atomic summary for each basic text block. ;

[0067] Among them, the extra-long basic text block is set to have more than 6000 characters;

[0068] 6.2. Separate atomic summaries Input large language models in batches, analyze the topic flow and semantic tightness between adjacent summaries, and output a set of logically merged intervals. ;

[0069] in, ; The global index ID of the starting atomic digest within the logical merge interval; The global index ID for terminating atomic digests within the logical merging interval; each batch is set to a group of 20 atomic digests;

[0070] 6.3. Assume the input atom digest The model outputs a set of continuous and mutually exclusive index ranges. Based on this interval set, the system physically aggregates the text blocks corresponding to the atomic summaries within the index range into a second-level heading area;

[0071] In step 6.3, the system uses a cross-batch status rollback mechanism to bring the unclosed interval information at the end of the previous batch into the next batch, so as to ensure the integrity of the long logic region across batches.

[0072] 6.4 For each physically aggregated second-level heading region, merge the atomic summaries of all ultra-long basic text blocks within it, and call the large language model again to generate a summary, thus generating a global high-order index summary for that region.

[0073] 6.5. Structure the generated global high-order index summary with each ultra-long basic text block and its atomic summary in the logical area of ​​the corresponding second-level heading, and output the reorganized hierarchical text with a two-level summary nesting structure;

[0074] The reorganized hierarchical text is set to JSON format.

[0075] The aforementioned structure-adaptive dual-mode long document intelligent slicing method, in step 7, generates the final physical slices based on the reorganized hierarchical text. The specific steps are as follows:

[0076] 7.1 Input the reorganized hierarchical text into the segmenter, starting from the current segmentation point. Calculate its optimal physical cutoff point. The specific formula is as follows:

[0077] ;

[0078] in, Set the target slice length to 1000 characters. To create an overlapping window, specifically set to 100 characters. To define the scope of the backtracking, the specific limit is 150 characters;

[0079] in, It is a function with maximum value. For the currently processed very long basic text block at the position The character at that location, Given the set of punctuation marks at the end of a sentence, if no punctuation mark is found within the backtracking interval, the sentence is forcibly truncated to... ;

[0080] 7.2 Generating the starting point for the next slice The specific formula is as follows:

[0081] ;

[0082] in, This will be used as the step coordinate for the next iteration of the sliding window. This is used to repeatedly execute step 7.1 to continuously generate a sequence of local slices with overlapping regions until the sliding window covers the entire text of the currently processed ultra-long base text block;

[0083] 7.3. For those belonging to the first The second-level region, the first Slices of basic blocks The system retrieves the global high-order index summary or atomic summary of the region, normalizes it, and outputs a context-enhanced final physical slice. The specific formula is as follows:

[0084] ;

[0085] in, For string concatenation; Present an atomic summary of this basic block;

[0086] () is the semantic prefixation function.

[0087] Compared with the prior art, the beneficial effects of the present invention are:

[0088] This invention eliminates non-semantic line break noise caused by PDF typesetting and establishes a physical coordinate system by constructing a micro-repair mechanism based on the original text stream with full-link traceability and a non-semantic line break feature library. Simultaneously, it utilizes a large language model for structural feature detection and dynamic routing, adaptively matching differentiated processing modes for untitled or titled processing modes, achieving adaptive traffic distribution of processing strategies. Furthermore, it performs two-stage intelligent noise cleaning and full-link directory blocking processing, and adopts anti-interference elastic anchoring or multi-round semantic connectivity merging strategies based on the dynamic routing results, enabling the output of structurally standardized basic text blocks that highly conform to human logical cognition. Finally, through bottom-up semantic aggregation with adaptive computing power scheduling, it utilizes multiple... The intelligent agent relay summarization strategy constructs a three-layer semantic index tree consisting of ultra-long base text, atomic summaries, and global high-order index summaries. This provides metadata support for context injection of subsequent slices. Furthermore, through a segmentation method with sentence-end punctuation backtracking and context injection, the final physical slices are generated. This effectively achieves adaptive perception and accurate processing of both strongly structured and weakly structured long documents, automatically balancing processing efficiency and segmentation accuracy. It avoids semantic breaks and context loss caused by traditional mechanical segmentation. By recording full-link traceability information and hierarchical summarization enhancement, it significantly improves the semantic integrity of the final physical slices and the downstream retrieval recall rate, reduces the risk of illusion, and ensures the high quality and interpretability of the RAG knowledge base construction. Attached Figure Description

[0089] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0090] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0091] This invention provides the following: Figure 1 The method for intelligently slicing long documents in a structure-adaptive dual-mode approach, as shown, includes the following steps:

[0092] Step 1: Receive the PDF document to be processed, extract the text content, insert page number anchors according to the physical pages, and construct the original text stream. The specific steps are as follows:

[0093] 1.1. Call the built-in PDF parsing engine, load the PDF document to be processed in binary read-only mode, establish the memory mapping of the file object model, and prepare for industry-level traversal;

[0094] The PDF parsing engine is built on the PyPDF2 or PDFPlumber library. When using it, the path validity and file integrity of the PDF document to be processed are first verified.

[0095] 1.2 Perform a page-by-page linear scan of the PDF document to be processed, obtain the physical page number index of the current page in real time during the traversal, and generate a uniquely identified source anchor string;

[0096] The source tracing anchor string is set to: >>>Page n<<<;

[0097] 1.3. Forcefully inject the source anchor string into the starting position of the extracted text on the current page, and add a standard line break at the end of the page to block the unnatural adhesion of cross-page text;

[0098] The addition of standard line breaks ensures that each page of text has an independent physical location identifier and maintains physical isolation in semantics.

[0099] 1.4 After all pages have been traversed, all processed single-page text in the buffer is concatenated into a continuous global string according to the physical page number order, thus constructing an original text stream with full-link traceability capability;

[0100] The system specifies a standard character encoding format (utf-8), and subsequently serializes the original text stream into an intermediate text file (.txt).

[0101] In step 1, by forcibly injecting source anchor strings and standard line breaks at the beginning and end of the text on a single page, the unnatural adhesion of text at the beginning and end of cross-pages (such as confusion between numbers at the end of the page and the text on the next page) commonly seen in traditional PDF extraction techniques is effectively prevented. At the same time, this processing method establishes a physical coordinate system in the plain text, providing a reliable data benchmark for subsequent intelligent noise reduction (such as global positioning of headers and footers) and accurate source tracing of the final slice.

[0102] Step 2: Perform initial repair of the original text stream to obtain a text file. The specific steps are as follows:

[0103] 2.1 Construct a non-semantic line break feature library to identify unnatural line breaks caused by PDF layout, while strictly preserving semantically separated line breaks;

[0104] Among them, the non-semantic line break feature library includes paired symbol closure detection, numerical unit continuity detection, and line-end non-sentence-end punctuation detection;

[0105] The paired symbol closure detection is used to detect whether there are unclosed punctuation marks or parentheses at the end of a line. The specific rules are as follows: count the number of symbols on the left and right sides of the current text line. If the number of symbols on the left is greater than the number of symbols on the right, then the end of the line is determined to be unclosed, which is a non-semantic line break. The set of paired symbols is defined as follows: Chinese parentheses “(” and “)”, English parentheses “(” and “)”, square brackets “[” and “]”, solid square brackets “

” and “

[0106] The unit continuity check is used to prevent numerical values ​​from being forcibly separated from immediately following units, time, or numerical portions by newline characters. The specific rules are as follows:

[0107] First, use a regular expression matching function to check if the end of the current line is an Arabic numeral (including integers and floating-point numbers with decimals). Then, use a regular expression matching function to check if the beginning of the next line is a specific unit symbol or a continuation of numbers.

[0108] Among them, the protected unit symbols include: area units (㎡, m²), weight units (kg, ton, t), energy and electricity units (degree, kW, MW, GW), time units (year, month, day, day, hour, minute, second), currency units (ten thousand, hundred million), and percentage symbols (%).

[0109] The line-end non-sentence-end punctuation detection is used to identify situations where pause punctuation marks in the middle of a sentence are cut off by a newline character. The specific rules are as follows: detect whether the character at the end of the current line belongs to the preset set of sentence punctuation marks. This line-end non-sentence-end punctuation detection strictly excludes sentence-end punctuation marks (such as period, question mark, exclamation mark) to ensure that paragraphs with clear semantic end marks are not merged.

[0110] The set of punctuation marks in a sentence includes: comma (,), pause mark (、), semicolon (;), colon (:), dash (—), connector (–), and ellipsis (…);

[0111] 2.2. Traverse the current line of the original text stream and obtain its next adjacent line. Call the regular expression matching function to match and determine the connection between the current line and the next line with the non-semantic line break feature library to obtain the text file.

[0112] The regular expression matching function's decision logic is as follows:

[0113] ;

[0114] in, For the currently processed line of text, The next adjacent row; This is a Boolean function for the current line, used to detect whether the number of symbols on the left side of the line is greater than the number of symbols on the right side, and to identify unclosed symbols. A Boolean function for two adjacent rows, used to detect... Whether the end of the line is a number, and Is the first line a protected unit or a consecutive value? This is a Boolean function for the current line, used to detect whether the last character of the line belongs to the set of connectors in the sentence; ( ) represents the merge function; The logical OR operation means that if any one of the three conditions is met, the merge will be performed.

[0115] The matching result determination rules are as follows:

[0116] If the combination of the current line and the next line matches any item in the non-semantic line break feature library (i.e., the paired symbols are not closed, the numerical unit is split, or the line ends with a non-sentence-ending punctuation mark), the line break character (\n) between the two lines will be deleted, and the content of the next line will be appended to the end of the previous line to form a new merged text line. After the merge occurs, the system will record the merge operation and continue to use the merged new text line as the basis for a new round of feature judgment with subsequent lines until no new merge rules are triggered in the entire round.

[0117] If the non-semantic line break feature library is not hit, the current line break character is retained, the line index is moved down, and the next pair of text lines is processed until the full text scan is completed. Finally, a text file with formatting errors corrected and paragraph structure preserved is output.

[0118] In step 2, by constructing a non-semantic line break feature library and performing continuous regular expression discrimination, most of the mechanical line breaks caused by the physical layout of PDF are repaired in advance with extremely low computational overhead. This maximizes the restoration and protection of the integrity of the original semantics of long documents and effectively avoids noise interference and computational waste caused by basic layout fragments to the subsequent understanding of large language models.

[0119] Step 3: Uniformly sample several pages from the text file, perform parallel analysis using a large language model, and determine the pattern routing. The specific steps are as follows:

[0120] 3.1 Perform equal-step sampling on the text file, according to the preset window step size. This process splits a text file into a sequence of consecutive page groups. Extract each group The plain text of multiple consecutive pages contained within is concatenated in physical order to construct a structured detection prompt word containing the content of consecutive pages;

[0121] in, , The total number of groups, ;

[0122] In step 3.1, after this step, the large language model can obtain the complete cross-page context within a single inference window, thereby accurately capturing cross-page titles or continuous layout features that may be truncated due to physical pagination, ensuring that the large language model can capture the complete features of cross-page titles or continuous layout.

[0123] 3.2 Grouping the pages into sequences The input is fed into a parallel inference engine based on the SGLang framework, and a batch concurrency mechanism is used to extract data each time. The first group is simultaneously sent to the large language model, which, based on the structure detection cue word instruction, identifies whether there are first-level and second-level headings in the sampled group, and finally outputs the first group. Structural feature vectors of each group ;

[0124] Among them, a batch concurrency mechanism is adopted to send the grouped data to the Qwen3 series (such as Qwen3-4B-Instruct) large language model. This large language model serves as the core semantic analysis unit. Based on the structure detection prompt word instruction, it uses its pre-trained language representation ability to identify whether there are first-level and second-level headings in the sampled group.

[0125] in, , Indicates whether a first-level heading exists. Indicates whether a second-level heading exists;

[0126] 3.3 Calculate the structured cumulative confidence score of the text file. The specific formula is as follows:

[0127] ;

[0128] in, The weight of a first-level heading, specifically... , The weight of the second-level heading is as follows: ;

[0129] 3.4 Define the dynamic routing decision function An early stopping mechanism is introduced to perform route determination and determine the mode route. The specific formula is as follows:

[0130] ;

[0131] in, To set a preset threshold, the confidence level is accumulated during the traversal process. Exceeding the preset threshold , specific The system immediately terminates subsequent checks, determines the document to be strongly structured, and activates the heading processing mode; if all documents have been traversed... If the grouping still fails to meet the standard, it is judged as weakly structured and the untitled processing mode is activated.

[0132] Among them, the title processing mode is defined as follows: For strongly structured documents with explicit hierarchical identifiers (such as numerical serial numbers, chapter identifiers, etc.), the extracted title features will be used as physical anchors to perform top-down physical segmentation and hierarchical tree reconstruction flow branches.

[0133] The untitled processing mode is defined as follows: For weakly structured documents that lack explicit hierarchical identifiers, a large language model will be used to analyze the semantic connectivity between adjacent paragraphs and perform bottom-up iterative stitching of flow branches.

[0134] In step 3, by introducing a structure detection mechanism based on equal-step sampling and a dynamic routing decision function, and combining it with the early stopping mechanism of cumulative confidence of the large language model, the system can quickly detect the global skeleton type of long documents with extremely low computing power. This avoids the semantic damage caused by using a single rigid segmentation for all documents, realizes adaptive diversion of processing strategies, and greatly improves the system's operating efficiency and cross-domain document compatibility.

[0135] Step 4: Perform intelligent noise cleaning and directory blocking on the text file to obtain denoised text. The specific steps are as follows:

[0136] 4.1 The large language model performs directory index feature recognition on the sampled page based on prompt words, determines whether the current page is a directory page, and removes directory pages;

[0137] The prompts include numeric guide lines, dotted lines, and page number indexes. If the large language model detects a prompt, it determines that the current page is a table of contents page. The system directly removes all content of that page from the text file and updates the physical index mapping of the remaining pages simultaneously to prevent the table of contents from interfering with the semantic connectivity of the main text.

[0138] 4.2. Process the page sequence after removing the table of contents in batches, input the page text into a large language model for layout structure analysis, and output the suspected header of each page. and suspected footer ;

[0139] The system does not immediately delete suspected headers from each page. and suspected footer Instead, it stores all extracted suspected header and footer text fragments into a candidate noise buffer;

[0140] 4.3 Perform global word frequency statistics and voting on all identified suspected headers and footers to determine the header noise features with the highest recurrence rate. Batch remove these features across the entire text, followed by local cleaning. Directly call the independent suspected header and footer identification records of each page and perform a second scan on the globally cleaned pages to accurately remove page-specific non-common noise elements that are not covered by global features, thus obtaining denoised text.

[0141] In step 4, by introducing a collaborative denoising mechanism of index feature recognition and global word frequency statistics and voting, we can accurately and batch remove high-frequency headers and footers that run throughout the entire text, while effectively avoiding the risk of accidentally deleting normal content in the main text by traditional mechanical regular expression matching. This maximizes the restoration of the document's purity and prevents non-text noise from interfering with subsequent RAG semantic connectivity analysis and knowledge retrieval.

[0142] Step 5: Perform dual-mode differential preprocessing on the denoised text to obtain ultra-long basic text blocks;

[0143] Among them, the ultra-long basic text blocks include semantically coherent basic text blocks or structurally standardized basic text blocks, and the specific acquisition steps are as follows;

[0144] 5.1 If the mode routing is set to the untitled processing mode, the output denoised text will be segmented into initial paragraph sequences based on newline characters. It then initiates a multi-round parallel semantic iterative merging process, in which adjacent text paragraph pairs are traversed in each round of iteration. The decision on whether to merge sentences is made using a dual-validation model of "hard constraints at the end of the sentence and soft semantic discrimination". The specific formula for the merging decision function is as follows:

[0145] ;

[0146] in, 1 , , For the set of sentence terminators, if paragraph If the sentence ends with a punctuation mark from the set of sentence terminators, then the indicator function... Returning to 0 forces a block on merging to prevent erroneous concatenation of different semantic paragraphs;

[0147] For indicator functions, This is a semantic connectivity discrimination function based on a large language model. The specific discrimination result includes yes or no, where yes indicates that it can be merged and no indicates that it cannot be merged. During the discrimination, the system iterates repeatedly until no new merging is generated, and outputs semantically coherent basic text blocks.

[0148] Among them, hard constraints at the end of a sentence refer to using a pre-set set of sentence terminators to perform low-cost rule pre-interception; if a rule is hit, it will be directly blocked and merged.

[0149] Semantic soft discrimination refers to using a large language model to perform in-depth analysis of the semantic context connectivity of adjacent paragraphs when allowing a rule to be passed.

[0150] In step 5.1, a rule-pre-interception mechanism with zero inference cost is adopted. Before calling the large language model, the sentence-end features of the preceding paragraph are detected first. If the sentence-end hits the preset set of terminators, it is directly determined that it cannot be merged and the subsequent large language model inference steps are skipped. The large language model semantic discrimination is only triggered for fuzzy boundaries that cannot be determined by the rules, thereby significantly reducing the computational cost while ensuring accuracy.

[0151] 5.2 If the mode routing leads to a mode with headings, execute a physical segmentation strategy based on heading features to output basic text blocks with standardized structure. The specific steps are as follows:

[0152] 5.2.1 Coarse-grained pre-segmentation: Based on the context window limitation of the large language model, long documents are pre-segmented into temporary fragment sets using a length threshold (e.g., 6000 characters) and a sentence-end punctuation backtracking mechanism. ;

[0153] 5.2.2 Title Feature Extraction: [The remaining text appears to be incomplete and requires further context.] Parallel input to a large language model, extracting the primary and secondary heading text contained therein, and generating an ordered heading sequence. ;

[0154] 5.2.3 Flexible Anchoring and Physical Segmentation: An flexible string matching algorithm is used, ignoring spaces and differences between full-width and half-width characters, to locate the title in the original document stream. Physical starting index Based on this, the document is divided into independent sub-blocks, each starting with a title. ;

[0155] The elastic string matching algorithm logic is as follows: It extracts the title data from the large language model. The text identifies all whitespace characters and dynamically inserts whitespace tolerance characters (such as \s* in a regular expression) to construct a matching pattern. This matching pattern is then used to locate the title in the denoised text. Physical starting index Based on this, the document is divided into independent sub-blocks, each starting with a title. ;

[0156] 5.2.4 For each independent sub-block Detect its header text and locate the corresponding title. Physically remove the title text from the independent sub-block so that only the plain text content is retained in the independent sub-block, thereby eliminating semantic redundancy and interference of the title text in subsequent summary extraction;

[0157] 5.2.5 Fragmentation and Fractionation: Detects micro-text blocks whose length is less than a preset value after segmentation, merges them back into the preceding text blocks, and outputs a basic text block with a standardized structure.

[0158] The default value is 50 characters.

[0159] In step 5, by implementing a dual-mode differentiated text shaping strategy, the system can intelligently adapt to the structural characteristics of the document. For weakly structured documents without titles, a stitching mechanism that combines zero-cost rule pre-processing with large-model semantic verification is adopted. For strongly structured documents with titles, an elastic matching algorithm is used to solve the common "format whitespace drift" problem in PDF extraction. This effectively solves the contextual fragmentation caused by traditional mechanical segmentation based on fixed characters, and outputs basic text blocks with a highly standardized structure that conforms to human logical cognition.

[0160] Step 6: Perform hierarchical restructuring on excessively long basic text blocks to generate restructured hierarchical text. The specific steps are as follows:

[0161] 6.1 For extremely long basic text blocks, a multi-agent relay summarization strategy is adopted, using the summary of the previous segment as context to guide the generation of the summary of the current segment, ultimately producing an atomic summary for each basic text block. ;

[0162] Among them, the extra-long basic text block is set to have more than 6000 characters;

[0163] The multi-agent relay summarization strategy in this method is a staged, progressive summarization generation method designed for ultra-long basic text blocks. Its core is to solve the context window limitation, illusion, and forgetting problems when large models process ultra-long basic text blocks through relay-style context passing. The specific working principle is as follows:

[0164] When processing the first segment, a summary of that segment is generated directly;

[0165] When processing the second and subsequent segments, the summary generated from the previous segment is used as context input to guide the large language model to understand the preceding logic and then generate a summary for the current segment.

[0166] By following this process segment by segment, an atomic summary of the extremely long basic text block is finally produced, which is a fine-grained summary of the extremely long basic text block that retains complete contextual relationships.

[0167] 6.2. Separate atomic summaries Input large language models in batches, analyze the topic flow and semantic tightness between adjacent summaries, and output a set of logically merged intervals. ;

[0168] in, , used to represent a logically merged interval for the large language model decision; The global index ID of the starting atomic digest within the logical merge interval; The global index ID for terminating atomic digests within the logical merging interval; each batch is set to a group of 20 atomic digests;

[0169] 6.3. Assume the input atom digest The model outputs a set of continuous and mutually exclusive index ranges. Based on this interval set, the system physically aggregates the text blocks corresponding to the atomic summaries within the index range into a second-level heading area;

[0170] During this process, the system adopts a cross-batch status rollback mechanism to carry the unclosed interval information at the end of the previous batch into the next batch, ensuring the integrity of the long logical region across batches.

[0171] The specific logic of the cross-batch state re-carry mechanism is as follows: when processing the current batch, extract all atomic digest IDs in the last logical region divided by the large language model, and bring them into the header of the inference sequence of the next batch for joint re-judgment to ensure the integrity of the long logical region across batches.

[0172] 6.4 For each physically aggregated second-level heading region, merge the atomic summaries of all ultra-long basic text blocks within it, and call the large language model again to generate a summary, thus generating a global high-order index summary for that region.

[0173] 6.5. Structure the generated global high-order index summary with each ultra-long basic text block and its atomic summary in the logical area of ​​the corresponding second-level heading, and output the reorganized hierarchical text with a two-level summary nesting structure;

[0174] Among them, the reorganized hierarchical text is set to JSON format, which can be called by downstream dynamic slicing;

[0175] In step 6, by introducing a multi-agent relay summarization strategy and a cross-batch logical region aggregation algorithm based on a large language model, the context window limitation and illusion forgetting problem in processing ultra-long basic text blocks are effectively overcome. By constructing a three-layer semantic index tree of "ultra-long basic text - atomic summary - global high-order index summary", metadata support is provided for the context injection of subsequent slices.

[0176] Step 7: Based on the reorganized hierarchical text, generate the final physical slices. The specific steps are as follows;

[0177] 7.1 Input the reorganized hierarchical text into the segmenter, starting from the current segmentation point. Calculate its optimal physical cutoff point. The specific formula is as follows:

[0178] ;

[0179] in, Set the target slice length to 1000 characters. To create an overlapping window, specifically set to 100 characters. To define the scope of the backtracking, the specific limit is 150 characters;

[0180] in, It is a function with maximum value. For the currently processed very long basic text block at the position The character at that location, Given the set of punctuation marks at the end of a sentence, if no punctuation mark is found within the backtracking interval, the sentence is forcibly truncated to... ;

[0181] 7.2 Generating the starting point for the next slice The specific formula is as follows:

[0182] ;

[0183] In step 7.2, This will be used as the step coordinate for the next iteration of the sliding window. This is used to repeatedly execute step 7.1 to continuously generate a sequence of local slices with overlapping regions until the sliding window covers the entire text of the currently processed ultra-long base text block;

[0184] 7.3. For those belonging to the first The second-level region, the first Slices of basic blocks The system retrieves the global high-order index summary or atomic summary of the region, normalizes it, and outputs a context-enhanced final physical slice. The specific formula is as follows:

[0185] ;

[0186] in, For string concatenation; Present an atomic summary of this basic block;

[0187] () is a semantic prefixing function, which detects the last character of the atomic summary. If it is not a punctuation mark, it automatically completes the sentence terminator (such as "."), thereby building a semantic isolation barrier between the summary and the main text and preventing unnatural adhesion between the content at the end of the summary and the first sentence of the main text, which could lead to semantic ambiguity.

[0188] In step 7.3, end-to-end traceability information is recorded in the slice metadata to construct a context-enhanced slice set with self-explanatory and accurate traceability capabilities;

[0189] The end-to-end traceability information is set as follows: Zone X --- Block Y --- Block Z or in ID form X---Y---Z;

[0190] In step 7, by using a segmentation method with overlapping windows and sentence-end punctuation backtracking, atomic summaries are injected into the header of the generated final physical slice, and the full-link traceability information is recorded in its associated metadata at the same time. This solves the common problem of loss of context fragment information in RAG retrieval, so that each slice still has complete semantic background and physical coordinates after being separated from the original document, which significantly improves the recall rate and generation accuracy of downstream knowledge retrieval.

[0191] A structure-adaptive dual-mode intelligent long document slicing system, the specific operation of which is based on the aforementioned structure-adaptive dual-mode intelligent long document slicing method, is characterized by including: a document initialization module, a basic repair module, a structure routing module, an intelligent noise reduction module, a dual-mode preprocessing module, a hierarchical aggregation module, and a dynamic slicing module.

[0192] The document initialization module is used to establish the physical coordinate system of the document, receive the PDF document to be processed and extract the text content, insert page number anchor marks according to the physical pages, and construct the original text stream with full-link traceability information;

[0193] The basic repair module is used to perform rule-based micro-governance. It uses a preset regular expression rule library to detect and merge non-semantic line breaks caused by typesetting in the original text stream, and repair basic errors such as unclosed pairs of symbols and broken numerical units.

[0194] The intelligent noise reduction module is used to purify the layout noise of the document. After the pattern routing is established, it first uses a large language model to identify and remove the table of contents page. Then, it combines global word frequency statistics and voting mechanism to build a high-frequency noise library to perform the first round of common feature cleaning of the whole text. Finally, based on the independent recognition results of the large language model on a single page, it performs a second round of page-by-page targeted cleaning on the page after the first round of cleaning to remove non-common noise elements unique to that page.

[0195] The dual-mode preprocessing module is used to perform differentiated text shaping. When entering the "no-title processing mode", it uses a multi-round parallel semantic connectivity judgment mechanism to merge erroneously truncated paragraphs. When entering the "title-based processing mode", it uses a title-based physical segmentation strategy to remove the title text. Finally, it outputs an ultra-long basic text block.

[0196] The hierarchical aggregation module is used to construct a semantic index tree for documents, generate atomic summaries in parallel for ultra-long basic text blocks, analyze the semantic correlation between adjacent summaries using a large language model to cluster and generate second-level heading logical regions, and further generate a global high-order index summary for that region.

[0197] The dynamic slicing module is used to generate the final physical slice with enhanced context. Based on the reorganized hierarchical text, it uses a greedy sliding window algorithm with overlapping windows and sentence-end punctuation backtracking to generate the final physical slice. The high-level summary and hierarchical position information of the region are injected into the slice header to form a standardized slice set with self-interpretation capabilities.

[0198] Verification experiment:

[0199] This experiment uses an 80-page PDF document titled "Feasibility Analysis Report of XX Technology Company's 2024 Semi-Annual R&D Projects" as an example to demonstrate the entire process of this invention, from the original unstructured document to the generation of high-quality semantic segments. The specific experimental process is as follows:

[0200] (1) The input data agent receives the PDF document:

[0201] Document characteristics: It contains numerous multi-page technical parameter tables, headers with the word "Confidential", a detailed table of contents, and the main text contains a multi-level heading structure (such as "I. Project Background", "1.1 Technical Route").

[0202] (2) Processing flow, specifically including:

[0203] Step A, Document Initialization and Anchor Injection:

[0204] Parsing and Anchor Points: The system starts a traversal loop, parsing the PDF content page by page. Taking page 5 as an example, the system automatically inserts a physical anchor string before extracting the text of that page: ">>>Page 5<<<...Technical specifications are as follows", thus giving the plain text a physical positioning attribute.

[0205] Page cross-page blocking: After the system finishes retrieving each page (e.g., page 5), it forcibly appends a standard line break character, thereby physically isolating the text at the end of the current page from the beginning of the next page (page 6), preventing semantic overlap caused by page cross-page (e.g., preventing the page number at the end of the page from being confused with the text value on the next page).

[0206] Step B, rule-based initial repair:

[0207] Sentence segmentation repair: The system scanned the text fragment: "The total budget for this project is 5 million yuan, which will be used for server procurement."

[0208] Rule determination: The line ends with the number "500" and the next line begins with the unit "ten thousand yuan", which matches the "numerical value and unit continuity detection rule".

[0209] Merge execution: Automatically removes the newline character in the middle and restores it to "...5 million yuan, used for...", ensuring the integrity of the numerical semantics;

[0210] Step C, Structural Feature Detection and Pattern Routing:

[0211] Sampling analysis: The system samples all content of the document page by page.

[0212] LLM discrimination: Parallel calls to large model analysis to identify explicit title features such as "I. Project Background" and "II. Technical Solution";

[0213] Routing decision: Based on the weighted formula, if the cumulative score of the first-level heading weight or the second-level heading weight exceeds a preset threshold... (e.g., 0.2), the system determines that the document is a "document with a title" and automatically activates the "document with a title processing mode";

[0214] Step D, Intelligent Noise Cleaning and Directory Blocking:

[0215] Table of contents removal: The model identifies pages 2-3 as containing a large number of page number guide lines in the form of "..........4", and determines them as table of contents pages. The entire page is removed to prevent keywords in the table of contents from interfering with the search.

[0216] Noise removal: Through a global voting mechanism, it was found that "XX Technology Company - Confidential Document" appeared at the top of 98% of the pages, which was identified as header noise; the system performed a global hard cleanup, deleting this text line from all pages in batches;

[0217] Step E involves preprocessing to differentiate title patterns:

[0218] Segmentation and recognition: First, the long document is divided into temporary segments according to its length, and the LLM is input in parallel to extract the title structure (such as "1.1 Technical Architecture").

[0219] Physical segmentation: Based on the identified title, the elastic string matching algorithm is used to locate the position of the title in the original text, and the document is physically segmented to form independent basic text blocks starting with "1.1 Technical Architecture";

[0220] Title stripping: To avoid the title text in the body slice from affecting the subsequent summary generation, the system physically removes the title text "1.1 Technical Architecture" at the top of the text block, and only retains the body content under the title;

[0221] Step F, bottom-up semantic hierarchy aggregation:

[0222] Atomic Summary: Generates an atomic summary (H2Outline) from the "Technical Architecture" text block (approximately 4000 characters) with the title removed: "This section details the backend design using a microservice architecture, including Spring Cloud component selection and containerized deployment solutions."

[0223] Region aggregation: Model analysis revealed that four consecutive content blocks were highly semantically related, and they were merged into a single logical region;

[0224] Higher-order summary: A higher-order index summary is generated for this merged region using the atomic summaries of these four sections: "[Technical Solution Section] This section focuses on describing the overall technical implementation path of the system, covering the backend microservice architecture design and core database selection scheme."

[0225] Step G, Context Injection and Dynamic Slice Generation:

[0226] Slicing strategy: Generate chapter summary slices: The system first creates an independent slice containing a high-order index summary of the region, labeled "Chapter Summary", which is used to quickly locate the entire topic area during retrieval.

[0227] Generate text slices: Perform sliding window segmentation on the "Technical Architecture" text (e.g., target length 1000 words, overlap 100 words).

[0228] Context injection: Automatically inject the corresponding atomic digest as a prefix into the header of several smaller blocks of text that have been split from a large block;

[0229] Final output: Slice 1 (Summary): "[Technical Solution Section] This section focuses on describing the overall system... (High-level summary)"; Slice 2 (Text): "This section elaborates on the adoption of a microservice architecture... (Atomic summary) [Prefix]... Specifically, Nacos was selected as the registry center... (Text fragment)."

[0230] In summary, this invention provides a reliable data benchmark for intelligent noise cleaning and accurate source tracing of final physical slices by constructing a micro-repair mechanism that integrates the original text stream with full-link traceability and a non-semantic line break feature library.

[0231] By utilizing a large language model for structural feature detection and dynamic routing, and adaptively matching the differentiated processing modes of untitled or titled processing modes, adaptive routing of processing strategies is achieved, which greatly improves the system's operating efficiency and cross-domain document compatibility.

[0232] Furthermore, by performing two-stage intelligent noise cleaning and full-link directory blocking processing, and by adopting anti-interference elastic anchoring or multi-round semantic connectivity merging strategies based on dynamic routing results, the context fragmentation caused by traditional mechanical segmentation based on fixed characters is effectively solved, and basic text blocks with highly standardized structures that conform to human logical cognition can be output.

[0233] Subsequently, through bottom-up semantic aggregation with adaptive computing power scheduling and a multi-agent relay summarization strategy, a three-layer semantic index tree was constructed, consisting of ultra-long basic text, atomic summaries, and global high-order index summaries. This provides metadata support for context injection of subsequent slices. Finally, through a segmentation method with sentence-end punctuation backtracking and context injection, the final physical slices were generated. This effectively achieved adaptive perception and accurate processing of both strongly structured and weakly structured long documents, automatically balancing processing efficiency and segmentation accuracy. It avoided semantic breaks and context loss caused by traditional mechanical segmentation. Furthermore, by recording full-link traceability information and hierarchical summarization enhancement, it significantly improved the semantic integrity of the final physical slices and the downstream retrieval recall rate, reduced the risk of illusion, and ensured the high quality and interpretability of the RAG knowledge base construction.

[0234] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used above are only some embodiments described in this invention. Obviously, those skilled in the art can obtain other drawings based on these drawings.

[0235] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A structurally adaptive dual-mode intelligent slicing method for long documents, characterized in that: Includes the following steps: Step 1: Receive the PDF document to be processed, extract the text content and insert page number anchor marks according to the physical pages to build the original text stream; Step 2: Perform basic repair of the original text stream to obtain a text file; Step 3: Uniformly sample several pages from the text file, perform parallel analysis using a large language model, and determine the pattern routing. The specific steps are as follows: 3.1 Perform equal-step sampling on the text file, according to the preset window step size. The text file is divided into a sequence of consecutive page groups. Extract each group The plain text of multiple consecutive pages contained within is concatenated in physical order to construct a structured detection prompt word containing the content of consecutive pages; in, , The total number of groups, ; 3.2 Grouping the pages into sequences The input is fed into a parallel inference engine based on the SGLang framework, and a batch concurrency mechanism is used to extract data each time. The first group is simultaneously sent to the large language model, which, based on the structure detection cue word instruction, identifies whether there are first-level and second-level headings in the sampled group, and finally outputs the first group. Structural feature vectors of each group ; in, , Indicates whether a first-level heading exists. Indicates whether a second-level heading exists; 3.3 Calculate the structured cumulative confidence score of the text file. The specific formula is as follows: ； in, As for the weight of the first-level heading, the specific , The weight of the second-level heading is as follows: ; 3.4 Define the dynamic routing decision function An early stopping mechanism is introduced to perform route determination and determine the mode route. The specific formula is as follows: ； in, To set a preset threshold, the confidence level is accumulated during the traversal process. Exceeding the preset threshold , specific The system immediately terminates subsequent detection, determines that the document is strongly structured, and activates the title processing mode. That is, for strongly structured documents with explicit hierarchical identifiers, the extracted title features are used as physical anchors to perform top-down physical segmentation and hierarchical tree reconstruction flow branches. If all have been traversed If the standard is still not met after grouping, it is judged as weakly structured and the untitled processing mode is activated. That is, for weakly structured documents that lack explicit hierarchical identification, the semantic connectivity between adjacent paragraphs is analyzed using a large language model, and the flow branches are iteratively stitched from bottom to top. Step 4: Perform intelligent noise cleaning and directory blocking on the text file to obtain denoised text; Step 5: Perform dual-mode differential preprocessing on the denoised text to obtain ultra-long basic text blocks; Step 6: Perform hierarchical restructuring on excessively long basic text blocks to generate restructured hierarchical text; Step 7: Generate the final physical slice based on the reorganized hierarchical text.

2. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 1, characterized in that: In step 1, the PDF document to be processed is received, the text content is extracted, and page number anchor marks are inserted according to the physical pages to construct the original text stream. The specific steps are as follows: 1.

1. Call the built-in PDF parsing engine, load the PDF document to be processed in binary read-only mode, establish the memory mapping of the file object model, and prepare for industry-level traversal; 1.2 Perform a page-by-page linear scan of the PDF document to be processed, obtain the physical page number index of the current page in real time during the traversal, and generate a uniquely identified source anchor string; The source tracing anchor string is set to: >>>Page n<<<; 1.

3. Forcefully inject the source anchor string into the starting position of the extracted text on the current page, and add a standard line break at the end of the page to block the unnatural adhesion of cross-page text; 1.4 After all pages have been traversed, all processed single-page text in the buffer is concatenated into a continuous global string according to the physical page number order, thus constructing an original text stream with full-link traceability capability.

3. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 1, characterized in that: In step 2, the original text stream is initially repaired to obtain a text file. The specific steps are as follows: 2.1 Construct a non-semantic line break feature library to identify unnatural line breaks caused by PDF layout, while strictly preserving semantically separated line breaks; Among them, the non-semantic line break feature library includes paired symbol closure detection, numerical unit continuity detection, and line-end non-sentence-end punctuation detection; 2.

2. Traverse the current line of the original text stream and obtain its next adjacent line. Call the regular expression matching function to match and determine the connection between the current line and the next line with the non-semantic line break feature library to obtain the text file. The regular expression matching function's decision logic is as follows: ； in, For the currently processed line of text, The next adjacent row; A Boolean function for the current line; A Boolean function for two adjacent rows; A Boolean function for the current line; ( ) represents the merge function; For logical OR.

4. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 3, characterized in that: In step 2.1, the paired symbol closure detection is used to detect whether there are unclosed punctuation marks or parentheses at the end of a line. The specific rule is: count the number of symbols on the left and right sides of the current text line. If the number of symbols on the left is greater than the number of symbols on the right, then the end of the line is determined to be unclosed and belongs to non-semantic line break. Numerical unit continuity detection is used to prevent numerical values ​​from being forcibly split from the following units, time, or numerical parts by newline characters. The specific rules are as follows: use a regular expression matching function to determine whether the end of the current line is an Arabic numeral, and then use a regular expression matching function to check whether the beginning of the next line is a specific unit symbol or a continuation of numbers. The line-end-not-sentence-end punctuation detection is used to identify situations where pause punctuation marks in the middle of a sentence are cut off by a newline character. The specific rule is: detect whether the character at the end of the current line belongs to the preset set of sentence punctuation marks. This line-end-not-sentence-end punctuation detection strictly excludes sentence-end punctuation marks to ensure that paragraphs with clear semantic end marks are not merged. In step 2.2, the matching result determination rules are as follows: If the combination of the current line and the next line matches any item in the non-semantic line break feature library, the line break character between the two lines will be deleted, and the content of the next line will be appended to the end of the previous line to form a new merged text line. After the merge occurs, the system will record the merge operation and continue to use the merged new text line as the basis for a new round of feature judgment with subsequent lines until no new merge rules are triggered in the whole round. If the non-semantic line break feature library is not matched, the current line break character is retained, the line index is moved down, and the next pair of text lines is processed until the full text scan is completed. Finally, a text file with formatting errors corrected and paragraph structure preserved is output.

5. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 1, characterized in that: In step 4, the text file undergoes intelligent noise cleaning and directory blocking to obtain denoised text. The specific steps are as follows: 4.1 The large language model identifies the directory index features of the sampled page based on the prompt words, determines whether the current page is a directory page, and removes directory pages; The prompts include numeric guide lines, dotted lines, and page number indexes. If the large language model detects a prompt, it determines that the current page is a directory page. The system directly removes all content of that page from the text file and simultaneously updates the physical index mapping of the remaining pages. 4.

2. Process the page sequence after removing the table of contents in batches, input the page text into a large language model for layout structure analysis, and output the suspected header of each page. and suspected footer ; 4.3 Perform global word frequency statistics and voting on all identified suspected headers and footers to determine the header noise features with the highest recurrence rate. Batch remove these features across the entire text, followed by local cleaning. Directly call the independent suspected header and footer identification records of each page and perform a second scan on the globally cleaned pages to accurately remove page-specific non-common noise elements that are not covered by global features, thus obtaining denoised text.

6. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 1, characterized in that: In step 5, the ultra-long basic text block includes semantically coherent basic text blocks or structurally standardized basic text blocks, and the specific steps for obtaining them are as follows: 5.1 If the mode routing is set to the untitled processing mode, the output denoised text will be segmented into initial paragraph sequences based on newline characters. It then initiates a multi-round parallel semantic iterative merging process, in which adjacent text paragraph pairs are traversed in each round of iteration. The decision on whether to merge sentences is made using a dual-validation model of "hard constraints at the end of the sentence and soft semantic discrimination". The specific formula for the merging decision function is as follows: ； in, ;1 , , For the set of sentence terminators, if paragraph If the sentence ends with a punctuation mark from the set of sentence terminators, then the indicator function... Returning to 0 forces a block on merging to prevent erroneous concatenation of different semantic paragraphs; For indicator functions, This is a semantic connectivity discrimination function based on a large language model. The specific discrimination result includes yes or no, where yes indicates that it can be merged and no indicates that it cannot be merged. During the discrimination, the system iterates repeatedly until no new merging is generated, and outputs semantically coherent basic text blocks. Sentence-end hard constraints refer to using a pre-defined set of sentence-end terminators for low-cost rule pre-interception; if a rule is hit, it will be directly blocked and merged. Semantic soft discrimination refers to using a large language model to perform in-depth analysis of the semantic context connectivity of adjacent paragraphs when allowing a rule to be passed. 5.2 If the mode is routed to a mode with a title, execute a physical segmentation strategy based on title features and output a basic text block with standardized structure.

7. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 6, characterized in that: In step 5.2, if the mode is routed to a mode with a title, a physical segmentation strategy based on title features is executed to output a basic text block with standardized structure. The specific steps are as follows: 5.2.1 Coarse-grained pre-segmentation: Based on the context window constraint of the large language model, and utilizing a length threshold and sentence-end punctuation backtracking mechanism, long documents are pre-segmented into a set of temporary fragments. ; 5.2.2 Title Feature Extraction: [The text abruptly ends here, likely due to an incomplete sentence or a Parallel input is used to a large language model, extracting the text of first-level and second-level headings to generate an ordered heading sequence. ; 5.2.3 Flexible Anchoring and Physical Segmentation: An flexible string matching algorithm is used, ignoring spaces and differences between full-width and half-width characters, to locate the title in the original document stream. Physical starting index Based on this, the document is divided into independent sub-blocks, each starting with a title. ; 5.2.4 For each independent sub-block Detect its header text and locate the corresponding title. Physically remove the title text from the independent sub-block so that only the plain text content remains, thereby eliminating semantic redundancy and interference from the title text during subsequent summary extraction. 5.2.5 Fragmentation and Fractionation: Detects micro-text blocks whose length is less than a preset value after segmentation, merges them back into the preceding text blocks, and outputs a basic text block with a standardized structure. The default value is 50 characters.

8. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 1, characterized in that: In step 6, the excessively long basic text blocks are uniformly reorganized into hierarchical text to generate reorganized hierarchical text. The specific steps are as follows: 6.1 For extremely long basic text blocks, a multi-agent relay summarization strategy is adopted, using the summary of the previous segment as context to guide the generation of the summary of the current segment, ultimately producing an atomic summary for each basic text block. ; Among them, the extra-long basic text block is set to have more than 6000 characters; 6.

2. Separate atomic summaries Input large language models in batches, analyze the topical shifts and semantic coherence between adjacent summaries, and output a set of logically merged intervals. ; in, ; The global index ID for the starting atomic digest within the logical merge interval; The global index ID for terminating atomic digests within the logical merging interval; each batch is set to a group of 20 atomic digests; 6.

3. Assume the input atom digest The model outputs a set of continuous and mutually exclusive index ranges. Based on this interval set, the system physically aggregates the text blocks corresponding to the atomic summaries within the index range into a second-level heading area; In step 6.3, the system uses a cross-batch status rollback mechanism to bring the unclosed interval information at the end of the previous batch into the next batch, so as to ensure the integrity of the long logic region across batches. 6.4 For each physically aggregated second-level heading region, merge the atomic summaries of all ultra-long basic text blocks within it, and call the large language model again to generate a summary, thus generating a global high-order index summary for that region. 6.

5. Structure the generated global high-order index summary with each ultra-long basic text block and its atomic summary in the logical area of ​​the corresponding second-level heading, and output the reorganized hierarchical text with a two-level summary nesting structure; The reorganized hierarchical text is set to JSON format.

9. The structurally adaptive dual-mode intelligent slicing method for long documents according to claim 1, characterized in that: In step 7, based on the reorganized hierarchical text, the final physical slice is generated. The specific steps are as follows: 7.1 Input the reorganized hierarchical text into the segmenter, starting from the current segmentation point. Calculate its optimal physical cutoff point. The specific formula is as follows: ； in, Set the target slice length to 1000 characters. To create an overlapping window, specifically set to 100 characters. To define the scope of the backtracking, the specific limit is 150 characters; in, It is a function with maximum value. For the currently processed very long basic text block at the position The character at that location, Given the set of punctuation marks at the end of a sentence, if no punctuation mark is found within the backtracking interval, the sentence is forcibly truncated to... ; 7.2 Generating the starting point for the next slice The specific formula is as follows: ； in, This will be used as the step coordinate for the next iteration of the sliding window. This is used to repeatedly execute step 7.1 to continuously generate a sequence of local slices with overlapping regions until the sliding window covers the entire text of the currently processed ultra-long base text block; 7.

3. For those belonging to the first The second-level region, the first Slices of basic blocks The system retrieves the global high-order index summary or atomic summary of the region, normalizes it, and outputs a context-enhanced final physical slice. The specific formula is as follows: ； in, For string concatenation; Present an atomic summary of this basic block; () is the semantic prefixation function.

Images