RAG-based text block dynamic segmentation method and system

Abstract

The present invention provides a method and system for dynamic text block segmentation based on RAG. The method comprises parsing a document's multi-level directory structure and segmenting text blocks based on directory nodes; switching to a hybrid segmentation mode based on rules and semantics for documents without or with incomplete directories; performing latent Dirichlet allocation topic modeling on continuous text streams and calculating the topic distribution vector for each paragraph in real time; performing adaptive segmentation on detected topic boundaries, inserting hard segmentation markers at topic mutation points, and employing soft segmentation for gradually changing topic areas; setting the initial window size of the sliding window based on the document type and monitoring the semantic density within the window in real time; and hierarchical block reorganization. The system comprises a segmentation mode module, a marker confirmation module, and a block reorganization module. The present invention improves the accuracy of RAG retrieval, reduces memory usage, and supports streaming throughput.

Description

Technical Field

[0001] The present invention relates to the technical field of text data processing, and in particular to a RAG-based text block dynamic segmentation method and system. Background Art

[0002] In Retrieval-Augmented Generation (RAG) systems, the quality of text segmentation directly impacts the accuracy of knowledge retrieval and the reliability of generated content. Traditional text segmentation methods employ fixed-length windows or static segmentation strategies based on punctuation, which are difficult to adapt to the semantic structural characteristics of texts in different domains. Especially when processing long texts containing nested semantics, such as technical documents and legal texts, existing segmentation methods can easily lead to the disconnection of key context, causing semantic drift during the retrieval phase. Furthermore, when faced with unstructured data in multimodal knowledge bases, static segmentation strategies are unable to dynamically adjust text granularity, resulting in a poor match between retrieval results and generation requirements, severely restricting the practical application of RAG systems. The existing technology has the following disadvantages: (1) it is easy to split a topic into two blocks, making each block semantically incomplete; (2) the content in a block is incoherent and the information is fragmented, which is not conducive to subsequent retrieval and generation.

[0003] Prior art 1, application number: CN202411426611.2 discloses a text segmentation method, device, storage medium and electronic device, the method comprising: obtaining a text to be segmented; determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented; performing data segmentation processing on the text to be segmented using the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented; performing data segmentation processing on each initial data block using at least a method of dynamically adjusting the segmentation block length to obtain target data blocks corresponding to the text to be segmented. Although for different text types, the method of dynamically adjusting the length of the segmentation block or the method of dynamically adjusting the length of the segmentation block and the length of the repeated block are used to segment each initial data block, it can ensure that the content of each block of the original initial data block is complete, and the segmentation result is less likely to be ambiguous; however, there is a lack of semantic coherence guarantee: it only relies on the preset rules of the text type and does not consider the semantic breaks caused by topic drift; the directory structure has poor adaptability: the dynamic mapping between the directory hierarchy and the segmentation granularity is not established, resulting in the loss of context association during cross-chapter retrieval; the conflict handling mechanism is missing: when the preset rules conflict with the semantic boundaries, the retrieval consistency cannot be maintained through the compensation mechanism.

[0004] Prior art 2, application number: CN 202410571033.5 discloses a text segmentation method, device, and medium for a large language model. The method includes: obtaining the text to be segmented; selecting a matching segmentation method to segment the text based on the text type, so that each segmented text block does not exceed the input capacity limit of the large language model; storing each text block in a vector library to provide a reference for the large language model to perform specific learning tasks. Although the impact of the large language model on text segmentation is taken into account, an adaptive text segmentation method is provided based on the application characteristics and needs of the large language model; however, the response to multi-granularity requirements is insufficient: segmentation is adjusted only by a single dimension of text type, which cannot meet the retrieval requirements of different granularities within the same document; multimodal structure processing is missing: the segmentation characteristics of non-continuous text such as directory levels, tables, and footnotes are not considered; the dynamic adjustment dimension is single: the number of characters is used as the adjustment basis only, and adaptive optimization is not combined with semantic density.

[0005] Prior art three, application number: CN202411805410.3 discloses a text segmentation method and device for retrieval-enhanced generation, including: constructing a text tree for the original text, wherein the text tree uses the original text attributes as the root node, titles of each level as child nodes, and non-title paragraphs as leaf nodes; segmenting the text blocks according to the text tree, so that each segmented text block starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and a directory structure consisting of titles of all levels and original text attributes corresponding to all parent nodes obtained step by step upward. Although it is possible to segment the text in a way that is more suitable for the tree structure, reducing the loss of text information during the segmentation process, thereby improving the recall effect of the RAG system; however, there are static block constraints: relying on a preset word limit to force segmentation, destroying natural semantic units; lack of semantic drift response: no topic continuity monitoring mechanism has been established, resulting in a break in cross-block semantic associations during retrieval; and lack of a dynamic optimization closed loop: the segmentation strategy is rigidified, and it is impossible to optimize subsequent segmentation decisions based on retrieval feedback.

[0006] Currently, the existing technologies 1, 2, and 3 have the problem of incomplete semantics, incoherent content, and fragmented information caused by traditional fixed text segmentation techniques. In order to meet the complex needs of segmenting long and short texts, dynamic text segmentation based on the combination of structure and semantics is required. Therefore, the present invention provides a method and system for dynamic text segmentation based on RAG. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention provides a method for dynamic segmentation of text blocks based on RAG, comprising the following steps:

[0008] Parse the document's multi-level directory structure and segment text blocks based on directory nodes; detect paragraph separators and sentence end symbols as secondary segmentation boundaries; for documents with a complete directory, generate a tree-like text block structure that strictly corresponds to the directory hierarchy; for documents without a directory or with an incomplete directory, switch to a hybrid rule-based and semantic segmentation mode, retaining the existing directory nodes as the primary segmentation points and enabling semantic segmentation for supplementary segmentation in areas where the directory is missing;

[0009] Perform latent Dirichlet allocation topic modeling on continuous text streams, calculate the topic distribution vector of each text segment in real time, calculate the topic similarity of adjacent windows using a sliding window, and identify the topic boundary when the topic similarity exceeds a threshold. Adaptive segmentation is performed on the detected topic boundary, inserting hard segmentation markers at topic mutation points and using soft segmentation for gradually changing topic areas.

[0010] The initial window size of the sliding window is set according to the document type, and the semantic density within the window is monitored in real time; hierarchical block reorganization is carried out; when the rule segmentation and semantic segmentation results conflict: priority is given to preserving the integrity of the directory structure, and a bidirectional index anchor is inserted in the conflicting area to maintain the context association during retrieval.

[0011] Optionally, the multi-level directory structure includes chapter, section or article numbers, the paragraph separator includes a blank line or an indent, and the sentence end symbol includes a period, a question mark or an exclamation mark.

[0012] Optionally, the process of parsing the document's multi-level directory structure includes the following steps:

[0013] Identify hierarchical markup features in the document and extract explicit chapter, section, or article numbering systems as the basis for primary divisions. Automatically generate independent text containers for each numbered node, forming an initial tree-like skeleton structure. Perform paragraph granularity analysis on each container unit in the skeleton structure: detect visual barriers formed by blank lines as paragraph separators, and identify text alignment boundaries generated by indentation changes as secondary division clues.

[0014] Within the divided paragraph units, sentence-level parsing is performed, constructing the minimum semantic unit boundary using sentence-end punctuation triples. List item detection mode is automatically activated for paragraphs containing numbered sequences, identifying substructures formed by consecutive numbers. When missing areas in the table of contents are detected, a compensation mechanism is automatically activated, retaining existing table of contents nodes as fixed anchors and constructing virtual nodes within the missing areas using semantic continuity analysis.

[0015] The initial segmentation scheme derived from the directory is aligned with the symbol analysis results, and areas where conflicts still exist after alignment are marked with abnormal signs, and then the sliding window analysis process is entered.

[0016] Optionally, when the deviation between the directory segmentation boundary and the natural paragraph boundary exceeds the preset tolerance, the following processing solutions are selected based on the context density;

[0017] High-density terminology area: keep directory boundaries first;

[0018] Narrative text area: Use natural paragraph boundaries;

[0019] Transition Zone: Inserts collapsible virtual segment markers.

[0020] Optionally, the process of constructing the minimum semantic unit boundary includes the following steps:

[0021] The paragraph units obtained by using paragraph delimiters are used as the input infrastructure for the minimum semantic unit boundary. The text stream within each container unit is used as the initial processing object. Within a given paragraph unit, the text character sequence is scanned to locate the position coordinates of the sentence-end punctuation triples. The existence of the sentence-end punctuation triples is verified based on the paragraph boundary constraints. The text intervals between the triples are generated into primary semantic blocks, the length of which is limited by the results of the granularity analysis.

[0022] For primary semantic blocks containing number prefixes, the list item detection mode is triggered. The similarity between the substructure formed by the consecutive numbering identified by the trigger condition and the established hierarchical markup features is judged. The consecutive numbered items automatically form a substructure tree, and the root node of the substructure tree establishes a parent-child relationship with the paragraph container node.

[0023] When the sequence of primary semantic blocks is interrupted, the gap area compensation mechanism is activated, using the parsed normal structure as the anchor reference system. By analyzing the verb tense patterns and entity reference relationships of the semantic blocks on both sides of the interrupted area, the potential structural continuity is deduced, and the generated virtual nodes inherit the same hierarchical attributes as the adjacent real nodes.

[0024] Optionally, the process of performing adaptive segmentation on the detected topic boundaries includes the following steps:

[0025] Based on the text content within the sliding window, the topic distribution vector of the current window is calculated. The topic distribution vector represents the probability weight of each topic within the window. If the text contains a directory structure, the window is initialized to align with the directory node boundary. If there is no directory structure, the window size is generated according to the initial setting.

[0026] A symmetric probability distribution difference metric is used to compare topic distribution vectors of adjacent windows. If the similarity is lower than a preset threshold, it is determined to be a topic mutation. If the similarity is higher than the threshold but shows a continuous downward trend, it is determined to be a topic gradual change area. The window size is dynamically adjusted based on the detection results.

[0027] Insert segmentation markers at mutation points and expandable segmentation markers at gradient areas, and associate context across boundaries during retrieval. If a segmentation point conflicts with a directory node, prioritize preserving directory integrity and insert a bidirectional index anchor at the conflicting location. The structured output of the segmentation results includes: type marker (hard / soft segmentation), original directory location (if any), associated topic distribution vector, and hierarchical integration.

[0028] Optionally, the topic similarity calculation process includes the following steps:

[0029] Get the topic distribution vectors of adjacent windows as input, where each topic distribution vector contains the probability weight value of each topic in the window; if it is the first calculation, initialize the reference set containing all known topics; if it is not the first time, inherit the valid topic set confirmed by the previous calculation;

[0030] Normalize the topic distribution vectors of adjacent windows to ensure that the sum of all components is 1; calculate the symmetric difference between the two distribution vectors, which satisfies the three basic properties of the distance function: non-negativity, symmetry, and triangle inequality;

[0031] The distribution difference value is mapped to a similarity score through a monotonically decreasing function, and an adaptively adjustable conversion threshold is set. The default value range is based on historical statistical results, and the normalized similarity score is output, with a fixed value range of [0,1]. Based on the characteristics of the current document type, the preset judgment threshold is loaded; the final score of the topic similarity of adjacent windows is output, and the final score is compared with the judgment threshold to produce a sudden or gradual judgment result.

[0032] Optionally, the process of outputting normalized similarity scores includes the following steps:

[0033] Receive symmetric difference values ​​to represent the difference between the two topic distributions; verify that the input value meets three basic mathematical constraints; record the current difference value in the historical statistical sequence; establish a conversion relationship from difference value to similarity, dynamically determine the initial morphological parameters of the conversion curve based on the characteristics of historical data, and set the boundary conditions of the conversion curve: the minimum difference corresponds to the highest similarity, and the theoretical maximum difference corresponds to zero similarity;

[0034] Substitute the current symmetric difference value into the mapping function to obtain the initial similarity, and calibrate the stability of the initial result based on the recent data fluctuation characteristics; determine the reference baseline based on the historical data distribution characteristics, adjust the calibrated topic similarity to the standard measurement interval, and use the upper and lower limits to truncate the result to within the specified range;

[0035] A dynamic baseline is established by comprehensively considering the historical similarity level and current document characteristics, the threshold is adjusted considering the topic diversity index and context consistency factors, and a boundary protection mechanism is applied to prevent the threshold from drifting beyond a reasonable range.

[0036] Optionally, the hierarchical block reorganization includes:

[0037] First-level block: inherits the directory structure segmentation node;

[0038] Secondary block: semantic segmentation boundary applied within directory nodes;

[0039] Three-level segmentation: dynamic granularity adjustment of sliding windows is implemented for complex paragraphs.

[0040] The present invention provides a RAG-based text block dynamic segmentation system, comprising:

[0041] The segmentation mode module is responsible for parsing the multi-level directory structure of the document and segmenting the text blocks based on the directory nodes. It detects paragraph delimiters and sentence end symbols as secondary segmentation boundaries. For documents with a complete directory, it generates a tree-like text block structure that strictly corresponds to the directory hierarchy. For documents without a directory or an incomplete directory, it switches to a hybrid segmentation mode that uses rules and semantics, retaining the existing directory nodes as the primary segmentation points and using semantic segmentation to supplement the segmentation in areas where the directory is missing.

[0042] The tag confirmation module is responsible for performing latent Dirichlet allocation topic modeling on the continuous text stream and calculating the topic distribution vector of each text segment in real time. It calculates the topic similarity of adjacent windows through a sliding window. When the topic similarity exceeds a threshold, it is determined to be a topic boundary and performs adaptive segmentation on the detected topic boundary. It inserts hard segmentation markers at topic mutation points and uses soft segmentation for gradually changing topic areas.

[0043] The block reorganization module is responsible for setting the initial window size of the sliding window according to the document type, monitoring the semantic density within the window in real time, and hierarchical block reorganization. When the rule segmentation and semantic segmentation results conflict: prioritize preserving the integrity of the directory structure, inserting bidirectional index anchors in the conflicting area, and maintaining the context association during retrieval.

[0044] The present invention forms an orthogonal segmentation dimension through the explicit hierarchy of the catalog and the implicit semantic boundary of the LDA topic model, and realizes the adaptive switching of hard / soft segmentation strategies in combination with the dynamic topic similarity calculation of the sliding window. The directory node serves as the main segmentation anchor to ensure the logical integrity of the document, and the semantic segmentation supplements the boundary identification of the unstructured area at the fine-grained level. The two establish a topological mapping relationship through a bidirectional index anchor to ensure the compatibility of structural navigation and semantic retrieval during retrieval. The three-level block hierarchy realizes progressive processing from macro-catalog inheritance (level one), meso-semantic segmentation (level two) to micro-sliding window granularity control (level three). After the window size is initially set according to the document type, it is dynamically adjusted by triggering real-time semantic density monitoring: the window is expanded for high-density professional terminology areas to maintain conceptual integrity, and the window is reduced for low-density narrative areas to improve segmentation accuracy. When a boundary conflict occurs between rule segmentation (catalog priority) and semantic segmentation (topic mutation point), a hybrid storage strategy with a tree-like text block structure as the trunk and semantic segmentation as branches is adopted, and the context-related path of the conflicting area is recorded through a bidirectional anchor. During the retrieval phase, structural traversal (directory tree jump) or semantic penetration (topic vector retrieval) can be selected as needed, achieving cross-level context reconstruction with O(1) time complexity. The incremental calculation of LDA topic distribution and the streaming caching mechanism of the sliding window enable the system to complete real-time segmentation without global document loading, making it particularly suitable for real-time log streams or progressive parsing of long documents. The coordinated adjustment of topic similarity thresholds and window sizes allows for FPGA hardware offload.

[0045] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 Flowchart of the RAG-based text block dynamic segmentation method in Example 1 of the present invention;

[0049] Figure 2 This is a schematic diagram of a method for dynamically segmenting text blocks based on RAG in Example 1 of the present invention;

[0050] Figure 3 This is a schematic diagram of the principle of semantic segmentation in Example 1 of the present invention;

[0051] Figure 4 This is a diagram showing the process of parsing a multi-level directory structure of a document in Example 2 of the present invention;

[0052] Figure 5 FIG. 4 is a process diagram for performing adaptive segmentation on detected topic boundaries in Example 4 of the present invention;

[0053] Figure 6 This is a block diagram of the RAG-based text block dynamic segmentation system in Example 7 of the present invention. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0055] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present application. The singular forms "a", "the" and "the" used in the embodiments of the present application are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0056] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0057] Example 1: Figure 1 As shown, the embodiment of the present invention provides a method for dynamic segmentation of text blocks based on RAG, comprising the following steps:

[0058] S100: Parse the document's multi-level directory structure and segment text blocks based on directory nodes; detect paragraph separators and sentence end symbols as secondary segmentation boundaries; for documents with a complete directory, generate a tree-like text block structure that strictly corresponds to the directory hierarchy; for documents without a directory or with an incomplete directory, switch to a hybrid rule-based and semantic segmentation mode, retaining the existing directory nodes as the primary segmentation points and using semantic segmentation for supplementary segmentation in areas where the directory is missing;

[0059] The multi-level directory structure includes chapter / section / article numbers, paragraph separators include blank lines and indents, and sentence end symbols include periods / question marks / exclamation marks;

[0060] S200: Performs latent Dirichlet allocation topic modeling on the continuous text stream, calculates the topic distribution vector of each text segment in real time, calculates the topic similarity of adjacent windows using a sliding window, and identifies a topic boundary when the topic similarity exceeds a threshold. The detected topic boundary is then adaptively segmented, inserting hard segmentation markers at topic mutation points and applying soft segmentation to areas with gradually changing topics.

[0061] S300: The initial window size of the sliding window is set according to the document type, and the semantic density within the window is monitored in real time; hierarchical segmentation and reorganization are performed; when the rule segmentation and semantic segmentation results conflict, the integrity of the directory structure is prioritized, and a bidirectional index anchor is inserted in the conflicting area to maintain the context association during retrieval;

[0062] Among them, hierarchical block reorganization includes:

[0063] First-level block: inherits the directory structure segmentation node;

[0064] Secondary block: semantic segmentation boundary applied within directory nodes;

[0065] Three-level segmentation: dynamic granularity adjustment of sliding windows is implemented for complex paragraphs.

[0066] The working principle and beneficial effects of the above technical solution are as follows: this embodiment first parses the multi-level directory structure of the document, and segments the text blocks based on the directory nodes; detects paragraph separators and sentence end symbols as secondary segmentation boundaries; for documents with complete directories, generates a tree-like text block structure that strictly corresponds to the directory hierarchy; for documents without directories or incomplete directories, switches to a mixed segmentation mode of rules and semantics, retains the existing directory nodes as the main segmentation points, and enables semantic segmentation for supplementary segmentation in the directory-missing areas; wherein, the multi-level directory structure includes chapter / section / article numbers, paragraph separators include blank lines and indentations, and sentence end symbols include periods / question marks / exclamation marks; secondly, performs latent Dirichlet allocation topic modeling on the continuous text stream, calculates the topic distribution vector of each paragraph of text in real time, and The moving window calculates the topic similarity of adjacent windows. When the topic similarity exceeds the threshold, it is determined to be a topic boundary. Adaptive segmentation is performed on the detected topic boundary, hard segmentation markers are inserted at the topic mutation point, and soft segmentation is used for the gradual topic area. Finally, the initial window size of the sliding window is set according to the document type, and the semantic density in the window is monitored in real time. Hierarchical block reorganization; when the rule segmentation and semantic segmentation results conflict: give priority to preserving the integrity of the directory structure, insert a bidirectional index anchor in the conflicting area, and maintain the context association during retrieval; Among them, hierarchical block reorganization includes: first-level block: inherit the directory structure segmentation node; second-level block: semantic segmentation boundary applied inside the directory node; third-level block: dynamic granularity adjustment of the sliding window for complex paragraphs (the principle is referred to in the attached Figure 2 , the principle of semantic segmentation is shown in the attached Figure 3The above scheme forms orthogonal segmentation dimensions through the explicit hierarchy of the table of contents and the implicit semantic boundaries of the LDA topic model. This approach, combined with dynamic topic similarity calculation using a sliding window, enables adaptive switching between hard and soft segmentation strategies. The table of contents nodes serve as the primary segmentation anchor to ensure the logical integrity of the document, while semantic segmentation complements the boundary identification of unstructured regions at a fine-grained level. The two establish a topological mapping relationship through bidirectional index anchors, ensuring the compatibility of structural navigation and semantic retrieval during retrieval. A three-level segmentation hierarchy implements a progressive process from macro-level table of contents inheritance (level 1), meso-level semantic segmentation (level 2), to micro-level sliding window granularity control (level 3). After the window size is initially set based on the document type, it is dynamically adjusted through real-time semantic density monitoring: expanding the window for high-density terminology areas to maintain conceptual integrity, while reducing the window for low-density narrative areas to improve segmentation accuracy. When boundary conflicts arise between rule-based segmentation (table of contents priority) and semantic segmentation (topic mutation points), a hybrid storage strategy is adopted, with a tree-like text block structure as the trunk and semantic segmentation as the branches. Bidirectional anchors are used to record the contextual association paths of the conflicting areas. During the retrieval phase, structural traversal (directory tree jump) or semantic penetration (topic vector retrieval) can be selected as needed, achieving cross-level context reconstruction with O(1) time complexity. The incremental calculation of LDA topic distribution and the streaming caching mechanism of the sliding window enable the system to complete real-time segmentation without global document loading, making it particularly suitable for real-time log streams or progressive parsing of long documents. The coordinated adjustment of topic similarity thresholds and window sizes allows for FPGA hardware offload.

[0067] In summary, compared with the traditional fixed-granularity blocking method, this embodiment improves the accuracy of RAG retrieval and reduces memory usage on the standard test set, while supporting streaming throughput.

[0068] This embodiment utilizes dynamic segmentation technology to enable the RAG system to more accurately capture semantic units in text, thereby improving the quality of knowledge retrieval and generation. This is a key step in optimizing complex document processing. This system achieves precise retrieval by avoiding the fragmentation of key information caused by fixed segmentation (e.g., a sentence containing an entity and a relationship in half); generational coherence by ensuring that the context used by the model when generating answers contains complete semantic units; and domain adaptability by adjusting segmentation strategies to adapt to different text types (technical documents, social media conversations).

[0069] This embodiment primarily describes a data processing strategy and method designed for segmenting text in Word, PDF, and Markdown files when building a knowledge base in RAG (Retrieval Enhanced Generation) projects. The core concept is to adjust the block size based on the structure or semantics of the content, making each block more semantically complete. This allows for dynamic segmentation of text blocks and improves retrieval performance. Furthermore, this technology is versatile and can be flexibly extended to the field of Large Language Models (LLMs).

[0070] Example 2: Figure 4 As shown, based on Example 1, the process of parsing the multi-level directory structure of a document provided by the embodiment of the present invention includes the following steps:

[0071] S101: Identify hierarchical markup features in the document and extract explicit chapter, section, or paragraph numbering systems as the basis for primary divisions. Automatically generate independent text containers for each numbered node to form an initial tree-like skeleton structure. Perform paragraph granularity analysis on each container unit in the skeleton structure. Detect visual barriers formed by blank lines as paragraph separators, and identify text alignment boundaries generated by indentation changes as secondary division clues.

[0072] S102: Within the divided paragraph units, sentence-level parsing is performed to construct the minimum semantic unit boundary using the sentence-end punctuation triples (period / question mark / exclamation mark). For paragraphs containing numbered sequences, list item detection mode is automatically activated to identify substructures formed by consecutive numbers. When a missing area in the table of contents is detected, a compensation mechanism is automatically activated, retaining the existing table of contents nodes as fixed anchor points and constructing virtual nodes in the missing area using semantic continuity analysis.

[0073] S103: Positionally aligning the initial segmentation scheme derived from the catalog with the symbol analysis results, marking any areas where conflicts remain after alignment with anomalies, and then transitioning to a sliding window analysis process. If the deviation between the catalog segmentation boundary and the natural paragraph boundary exceeds a preset tolerance, the following processing options are selected based on context density.

[0074] High-density terminology area: keep directory boundaries first;

[0075] Narrative text area: Use natural paragraph boundaries;

[0076] Transition Zone: Inserts collapsible virtual segment markers.

[0077] The working principle and beneficial effects of the above technical solution are as follows: This embodiment first identifies the hierarchical markup features in the document and extracts the explicit chapter, section or article numbering system as the basis for trunk division; each numbering node automatically generates an independent text container to form an initial tree-like skeleton structure; for each container unit in the skeleton structure, paragraph granularity analysis is performed: the visual isolation zone formed by blank lines is detected as a paragraph separation mark, and the text alignment boundary generated by indentation changes is identified as a secondary division clue; secondly, within the divided paragraph units, sentence-level parsing is implemented, and the minimum semantic unit boundary is constructed through the sentence-end punctuation triple (period / question mark / exclamation mark); the list item detection mode is automatically activated for paragraphs containing numbering sequences to identify the substructure formed by continuous numbering; when a missing area in the directory is detected, a compensation mechanism is automatically activated, the existing directory node is retained as a fixed anchor point, and a virtual node is constructed in the vacant interval using semantic continuity analysis; finally, the initial segmentation scheme derived from the directory is aligned with the symbol analysis results, and areas that still have conflicts after alignment are marked with abnormal signs, and the sliding window analysis process is switched to. The hierarchical system formed by the processing flow of this embodiment has a downward compatibility feature: the segmentation results of the upper-level directory nodes directly constrain the scope boundaries of the lower-level processing, and the correction signal obtained from the lower-level analysis adjusts the partitioning strategy of the upper-level container through the feedback channel, forming a closed-loop optimization system; the output text block structure maintains topological isomorphism with the original directory, while embedding verified fine-grained partitioning marks.

[0078] Example 3: Based on Example 2, the process of constructing the minimum semantic unit boundary provided by the embodiment of the present invention includes the following steps:

[0079] S1021: Using paragraph delimiters to obtain paragraph units as the input infrastructure for the minimum semantic unit boundary; using the text stream within each container unit as the initial processing object; scanning the text character sequence within a given paragraph unit to locate the position coordinates of sentence-end punctuation triples, with the existence of sentence-end punctuation triples verified based on paragraph boundary constraints; generating primary semantic blocks between the text triples, with the length of the primary semantic blocks limited by the granularity analysis results;

[0080] S1022: For a primary semantic block containing a number prefix, a list item detection mode is triggered. The substructure formed by the consecutive numbers identified by the trigger condition is similar to the established hierarchical markup features. The consecutive numbered items are automatically formed into a substructure tree. A parent-child relationship is established between the root node of the substructure tree and the paragraph container node.

[0081] S1023: When the sequence of primary semantic blocks is interrupted, the gap area compensation mechanism is activated, and the parsed normal structure is used as the anchor reference system. By analyzing the verb tense patterns and entity reference relationships of the semantic blocks on both sides of the interrupted area, the potential structural continuity is derived, and the generated virtual nodes inherit the same hierarchical attributes as the adjacent real nodes.

[0082] The working principle and beneficial effects of the above technical solution are as follows: this embodiment first uses the paragraph unit obtained by the paragraph separation identifier as the input basic structure of the minimum semantic unit boundary; the text flow within each container unit is used as the initial processing object; within a given paragraph unit, the text character sequence is scanned to locate the position coordinates of the sentence-end punctuation triple, and the existence verification of the sentence-end punctuation triple is based on the paragraph boundary constraint; the text interval between the triples generates a primary semantic block, and the length of the primary semantic block is limited by the granularity analysis results; secondly, for the primary semantic block containing a number prefix, the list item detection mode is triggered, and the similarity between the substructure formed by the consecutive numbering identified by the trigger condition and the established hierarchical mark feature is judged. The consecutive numbered items automatically form a substructure tree, and the root node of the substructure tree establishes a parent-child relationship with the paragraph container node; finally, when the sequence of the primary semantic block is interrupted, the vacant area compensation mechanism is activated, using the parsed normal structure as the anchor reference system. By analyzing the verb tense pattern and entity reference relationship of the semantic blocks on both sides of the interrupted area, the potential structural continuity is derived, and the generated virtual node inherits the same hierarchical attributes as the adjacent real nodes. Each step of the above scheme automatically constructs a document object model with multi-layered semantic associations based on a structured input data stream. By coupling syntactic unit segmentation under paragraph granularity constraints with a dynamic compensation mechanism, a unified representation system is established that combines the document's surface layout features with its deeper semantic associations. This forms a complete mapping chain from physical layout to logical structure. The spatial constraints provided by the paragraph container and the primary semantic blocks generated by sentence-end punctuation detection form the basic segmentation framework. List item pattern recognition establishes a fine-grained subtree structure within this framework, ultimately achieving closed-loop repair of broken structures through gap compensation. These three processing stages jointly ensure the stable conversion of the document's discrete symbol sequence into a continuous semantic topology, maintaining information integrity across all levels. The length constraints of the primary semantic blocks and the hierarchical inheritance of the list item subtree form a vertical protection mechanism, while verb tense and entity reference analysis provide horizontal continuity assurance, eliminating the semantic fragmentation caused by traditional methods that rely solely on layout features. This dual protection ensures that the output structure simultaneously meets layout boundary constraints and semantic coherence requirements. An adaptive error control system is established. The initial segmentation based on paragraph containers provides an error propagation boundary for processing. The similarity judgment of list item detection realizes local structure verification. Virtual nodes are generated in vacant areas to form global structure compensation. The three-level fault-tolerant mechanism constitutes a progressive error control network, which controls the local feature recognition error within the range of a single subtree and avoids the risk of structural collapse.

[0083] Example 4: Figure 5 As shown, based on Example 1, the process of performing adaptive segmentation on the detected topic boundary provided by the embodiment of the present invention includes the following steps:

[0084] S201: Based on the text content in the sliding window, the topic distribution vector of the current window is calculated. The topic distribution vector represents the probability weight of each topic in the window. If the text contains a directory structure, the window is initialized to align with the directory node boundary. If there is no directory structure, the window size is generated according to the initial setting.

[0085] S202: Using a symmetric probability distribution difference metric, compare the topic distribution vectors of adjacent windows; if the topic similarity is lower than a preset threshold, it is determined to be a sudden topic change; if the topic similarity is higher than the threshold but shows a continuous downward trend, it is determined to be a gradual topic change area; the window size is dynamically adjusted based on the detection results;

[0086] S203: Insert segmentation markers at mutation points and expandable segmentation markers in gradient areas, and associate contexts across boundaries during retrieval; if a segmentation point conflicts with a directory node, prioritize preserving directory integrity and insert a bidirectional index anchor at the conflicting position; structured output of segmentation results: type marker (hard / soft segmentation), original directory location (if any), associated topic distribution vector, and hierarchical integration.

[0087] The working principle and beneficial effects of the above technical solution are as follows: First, based on the text content in the sliding window, this embodiment calculates the topic distribution vector of the current window, which represents the probability weight of each topic in the window; if the text contains a directory structure, the window is initialized to align with the directory node boundary; if there is no directory structure, the window size is generated according to the initial setting; secondly, a symmetric probability distribution difference metric is used to compare the topic distribution vectors of adjacent windows; if the similarity is lower than the preset threshold, it is determined to be a topic mutation; if the similarity is higher than the threshold but shows a continuous downward trend, it is determined to be a topic gradient area; the window size is dynamically adjusted according to the detection results; finally, a segmentation mark is inserted at the mutation point, and an expandable segmentation mark is inserted in the gradient area to associate context across boundaries during retrieval; if the segmentation point conflicts with the directory node, the directory integrity is prioritized and a bidirectional index anchor is inserted at the conflicting position; the structured output of the segmentation result: type mark (hard / soft segmentation), original directory position (if any), associated topic distribution vector and hierarchical integration. The above solution ensures that the rule segmentation and semantic segmentation are spatially aligned by initializing the window boundary and the topic distribution vector through the directory node. When a sudden topic change is detected, hard segmentation markers forcibly interrupt the current semantic unit, while soft segmentation in transitional regions maintains long-range dependencies through expandable markers. This combination achieves a balance between fine-grained segmentation of local sudden changes and global coherence preservation. Window resizing is driven by feedback from a symmetric probability difference metric, automatically expanding the window in high-density regions to reduce over-segmentation and contracting the window in low-density regions to capture fine-grained topic transitions. This mechanism, combined with the bidirectional index anchors used in directory conflicts, creates a redundant design, ensuring that even when directory integrity is prioritized, fragmented semantic fragments can still be linked through anchors. The output structured metadata (type markers / directory location / topic vectors) forms a three-level indexing system: hard segmentation markers directly map physical storage boundaries, soft segmentation markers support probabilistic boundary expansion during retrieval, and topic distribution vectors provide term-level semantic routing for dense retrieval. The resulting hierarchical integration enables document collections to meet the dual requirements of precise retrieval (based on the directory hierarchy) and semantic retrieval (based on topic distribution). Initial window alignment to the directory reduces inefficient computation, dynamic adjustment strategies reduce repeated modeling overhead, and a conflict anchor mechanism avoids data redundancy at the storage level. The final output solution achieves Pareto optimality in terms of computational complexity (O(n) sliding window detection), storage efficiency (hard segmentation to compress the original text), and retrieval performance (soft segmentation to support approximate nearest neighbors).

[0088] Example 5: Based on Example 4, the subject similarity calculation process provided in this embodiment of the present invention includes the following steps:

[0089] S2021: Obtain topic distribution vectors of adjacent windows as input, where each topic distribution vector contains the probability weight value of each topic in the window; if this is the first calculation, initialize a reference set containing all known topics; if not, inherit the valid topic set confirmed by the previous calculation;

[0090] S2022: Normalize the topic distribution vectors of adjacent windows to ensure that the sum of all components is 1; calculate the symmetric difference between the two distribution vectors, which satisfies the three basic properties of the distance function: non-negativity, symmetry, and triangle inequality;

[0091] S2023: Map the distribution difference value to a similarity score through a monotonically decreasing function, set an adaptively adjustable conversion threshold, and output a normalized similarity score with a default value range based on historical statistical results. The value range is fixed to [0,1]. Based on the characteristics of the current document type, load the preset judgment threshold. Output the final score of the topic similarity of adjacent windows, compare the final score with the judgment threshold, and generate a sudden or gradual judgment result.

[0092] The working principle and beneficial effects of the above technical solution are as follows: this embodiment first obtains the topic distribution vectors of adjacent windows as input, where each topic distribution vector contains the probability weight value of each topic in the window; if it is the first calculation, initialize the reference set containing all known topics; if it is not the first time, inherit the valid topic set confirmed by the previous calculation; secondly, the topic distribution vectors of adjacent windows are normalized to ensure that the sum of each component is 1; calculate the symmetric difference value between the two distribution vectors, and the symmetric difference value satisfies the three basic properties of the distance function: non-negativity, symmetry and triangle inequality; finally, the distribution difference value is mapped to a similarity score through a monotonically decreasing function, and an adaptively adjustable conversion threshold is set. The default value range is based on historical statistical results, and the normalized similarity score is output, and the value range is fixed to [0,1]; based on the characteristics of the current document type, the preset judgment threshold is loaded; the final score of the topic similarity of the adjacent windows is output, and the final score is compared with the judgment threshold to produce a sudden or gradual judgment result. This approach eliminates probability bias caused by window size differences through normalization, ensuring comparability of topic vectors across text units of varying granularity. The mathematical property of symmetric difference values ​​ensures metric consistency across document comparisons, resolving the issue of traditional similarity calculations not converging in the probability distribution space. A two-tiered adjustment mechanism based on historical statistics and document type features is implemented: a global default value maintains the baseline judgment criteria, while local document feature compensation ensures domain-specific sensitivity. A monotonic function mapping transforms nonlinear distribution differences into directly comparable similarity scalars. Initialization with a reference set for the first computation and inheritance of valid topic sets from non-first computations enable progressive expansion of the topic space, preventing topic dimensionality explosion while supporting the dynamic absorption of unregistered topics, forming a stable incremental learning framework. The comparison of the final score with the decision threshold not only outputs a discrete decision (sudden / gradual) but also provides feedback to update the historical statistics database, enabling continuous adaptive decision threshold parameters to effectively address the dynamic characteristics of different corpora and topic evolution patterns.

[0093] Example 6: Based on Example 5, the process of outputting normalized similarity scores provided in this example includes the following steps:

[0094] S2031: Receive symmetric difference values ​​to represent the difference between the two topic distributions; verify that the input values ​​meet three basic mathematical constraints: non-negativity, symmetry, and triangular inequality; record the current difference values ​​in the historical statistical sequence; establish a conversion relationship from difference values ​​to similarity, dynamically determine the initial morphological parameters of the conversion curve based on the characteristics of the historical data, and set the boundary conditions of the conversion curve: the minimum difference corresponds to the highest similarity, and the theoretical maximum difference corresponds to zero similarity;

[0095] S2032: Substitute the current symmetric difference value into the mapping function to obtain the initial similarity, and perform stability calibration on the initial result based on the recent data fluctuation characteristics; determine the reference baseline based on the historical data distribution characteristics, adjust the calibrated topic similarity to the standard measurement interval, and use the upper and lower limits to truncate the result to within the specified range;

[0096] S2033: Establish a dynamic baseline based on historical similarity levels and current document characteristics, adjust the threshold based on topic diversity indicators and contextual consistency factors, and apply a boundary protection mechanism to prevent the threshold from drifting beyond a reasonable range.

[0097] The working principle and beneficial effects of the above technical solution are as follows: this embodiment first receives a symmetric difference value to characterize the difference between the two topic distributions; verifies that the input value meets three basic mathematical constraints: non-negativity, symmetry and triangular inequality; records the current difference value into the historical statistical sequence; establishes a conversion relationship from the difference value to the similarity, dynamically determines the initial morphological parameters of the conversion curve according to the historical data characteristics, and sets the boundary conditions of the conversion curve: the minimum difference corresponds to the highest similarity, and the theoretical maximum difference corresponds to zero similarity; secondly, substitutes the current symmetric difference value into the mapping function to obtain the initial similarity, and calibrates the stability of the initial result according to the recent data fluctuation characteristics; determines a reference baseline based on the historical data distribution characteristics, adjusts the calibrated topic similarity to the standard measurement interval, and cuts the result into the specified range through upper and lower limits; finally, establishes a dynamic baseline based on the historical similarity level and the current document characteristics, adjusts the threshold considering the topic diversity index and context consistency factors, and applies a boundary protection mechanism to prevent the threshold from drifting beyond a reasonable range. The initial difference values ​​in this scheme are verified for non-negativity, symmetry, and triangular inequality, ensuring that the mathematical foundation for subsequent calculations complies with metric space requirements. Historical sequence records also provide data support for dynamic parameter adjustment. Preset boundary conditions ensure the theoretical completeness of the output range. A dynamic parameter determination mechanism based on historical data features makes the similarity conversion curve adaptive, maintaining the mathematical property of monotonically decreasing while adjusting sensitivity based on changes in data distribution. The fluctuation calibration step suppresses outlier interference and enhances computational robustness. Scale normalization is achieved through a two-layer adjustment mechanism: first, calibration based on a baseline based on historical distribution, and then, through interval truncation constraints, ensuring that the output is strictly confined to the range [0, 1]. This process maintains comparability across documents. The dynamic baseline integrates historical statistical patterns and current document features, and weighted adjustment based on topic diversity and contextual coherence makes the judgment threshold context-aware. A boundary protection mechanism maintains the validity and stability of the threshold.

[0098] Example 7: Figure 6 As shown, based on Examples 1 to 6, the RAG-based text block dynamic segmentation system provided by the embodiment of the present invention includes:

[0099] The segmentation mode module is responsible for parsing the multi-level directory structure of the document and segmenting the text blocks based on the directory nodes. It detects paragraph delimiters and sentence end symbols as secondary segmentation boundaries. For documents with a complete directory, it generates a tree-like text block structure that strictly corresponds to the directory hierarchy. For documents without a directory or an incomplete directory, it switches to a hybrid segmentation mode that uses rules and semantics, retaining the existing directory nodes as the primary segmentation points and using semantic segmentation to supplement the segmentation in areas where the directory is missing.

[0100] The multi-level directory structure includes chapter / section / article numbers, paragraph separators include blank lines and indents, and sentence end symbols include periods / question marks / exclamation marks;

[0101] The tag confirmation module is responsible for performing latent Dirichlet allocation topic modeling on the continuous text stream and calculating the topic distribution vector of each text segment in real time. It calculates the topic similarity of adjacent windows through a sliding window. When the topic similarity exceeds a threshold, it is determined to be a topic boundary and performs adaptive segmentation on the detected topic boundary. It inserts hard segmentation markers at topic mutation points and uses soft segmentation for gradually changing topic areas.

[0102] The block reorganization module is responsible for setting the initial window size of the sliding window according to the document type and monitoring the semantic density within the window in real time; hierarchical block reorganization; when the rule segmentation and semantic segmentation results conflict, it prioritizes preserving the integrity of the directory structure and inserting bidirectional index anchors in the conflicting area to maintain the context of the search;

[0103] Among them, hierarchical block reorganization includes:

[0104] First-level block: inherits the directory structure segmentation node;

[0105] Secondary block: semantic segmentation boundary applied within directory nodes;

[0106] Three-level segmentation: dynamic granularity adjustment of sliding windows is implemented for complex paragraphs.

[0107] The working principle and beneficial effects of the above technical solution are as follows: the segmentation mode module of this embodiment parses the multi-level directory structure of the document and segments the text blocks based on the directory nodes; detects paragraph separators and sentence end symbols as secondary segmentation boundaries; for documents with complete directories, generates a tree-like text block structure that strictly corresponds to the directory hierarchy; for documents without directories or incomplete directories, switches to a mixed segmentation mode of rules and semantics, retains the existing directory nodes as the main segmentation points, and enables semantic segmentation for supplementary segmentation in the directory missing areas; the tag confirmation module performs latent Dirichlet allocation topic modeling on the continuous text stream, Calculate the topic distribution vector of each text segment; calculate the topic similarity of adjacent windows through a sliding window, and determine it as a topic boundary when the topic similarity exceeds a threshold, and perform adaptive segmentation on the detected topic boundary; insert hard segmentation marks at topic mutation points, and use soft segmentation for gradual topic areas; the initial window size of the sliding window of the block reorganization module is set according to the document type, and the semantic density in the window is monitored in real time; hierarchical block reorganization; when the rule segmentation and semantic segmentation results conflict: give priority to preserving the integrity of the directory structure, insert a bidirectional index anchor in the conflicting area, and maintain the context association during retrieval. This embodiment uses the segmentation mode module to achieve precise alignment of the document directory structure and text content, ensuring that the inherent hierarchical relationship such as chapter / section / article is not lost during the segmentation process; a tree-like text structure can be generated for documents with complete directories, and the existing structural framework is maintained for documents with missing directories through a mixed segmentation of rules and semantics. Initial segmentation is performed using a combination of formatting features (blank lines / indents) and linguistic features (end-of-sentence symbols). Combined with LDA topic modeling in the tag confirmation module, topic continuity is dynamically detected through a sliding window. A dual judgment mechanism (threshold determination and gradient analysis) improves the accuracy of segmentation boundaries. The segmentation and reorganization module implements a three-level segmentation system: the first level maintains macrostructure, the second level ensures topic coherence, and the third level processes micro-semantic units. This mechanism supports multi-scale information organization, from the chapter level to within paragraphs. When rule-based segmentation (directory-oriented) and semantic segmentation (topic-oriented) are inconsistent, a bidirectional index anchor technique is used to preserve the search context, balancing the conflicting requirements between structural integrity and semantic rationality.

[0108] In summary, this embodiment achieves the coordination and unification of hierarchical structure preservation, topic boundary identification, and multi-granularity segmentation requirements during document parsing, and provides text block input with reasonable structure and semantic coherence for retrieval enhancement generation tasks.

[0109] This embodiment is a method for processing text data in chunks. Its main application scenarios include: processing complex structured documents, multi-round dialogue and interactive systems, and specialized scenarios in vertical fields. Typical scenario examples are shown in Table 1:

[0110] Table 1

[0111]

[0112]

[0113] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the present invention's equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A method for dynamic segmentation of text blocks based on RAG, characterized in that: The following steps are involved: Parse the document's multi-level directory structure and segment text blocks based on directory nodes; detect paragraph separators and sentence end symbols as secondary segmentation boundaries; For documents with complete directories, a tree-like text block structure that strictly corresponds to the directory hierarchy is generated. For documents without a directory or with an incomplete directory, a hybrid segmentation mode of rules and semantics is used, retaining the existing directory nodes as the primary segmentation points and enabling semantic segmentation for supplementary segmentation in areas where the directory is missing. Perform latent Dirichlet allocation topic modeling on continuous text streams, calculate the topic distribution vector of each text segment in real time, calculate the topic similarity of adjacent windows using a sliding window, and identify the topic boundary when the topic similarity exceeds a threshold. Adaptive segmentation is performed on the detected topic boundary, inserting hard segmentation markers at topic mutation points and using soft segmentation for gradually changing topic areas. The initial window size of the sliding window is set according to the document type, and the semantic density within the window is monitored in real time; hierarchical block reorganization is carried out; when the rule segmentation and semantic segmentation results conflict: priority is given to preserving the integrity of the directory structure, and a bidirectional index anchor is inserted in the conflicting area to maintain the context association during retrieval.

2. The RAG-based text block dynamic segmentation method according to claim 1, characterized in that: in, The multi-level directory structure includes chapter, section or article numbers, paragraph separators include blank lines or indents, and sentence end symbols include periods, question marks or exclamation marks.

3. The RAG-based text block dynamic segmentation method according to claim 1, characterized in that: The process of parsing the multi-level directory structure of a document includes the following steps: Identify hierarchical markup features in the document and extract explicit chapter, section, or article numbering systems as the basis for primary divisions. Automatically generate independent text containers for each numbered node, forming an initial tree-like skeleton structure. Perform paragraph granularity analysis on each container unit in the skeleton structure: detect visual barriers formed by blank lines as paragraph separators, and identify text alignment boundaries generated by indentation changes as secondary division clues. Within the divided paragraph units, sentence-level parsing is performed, constructing the minimum semantic unit boundary using sentence-end punctuation triples. List item detection mode is automatically activated for paragraphs containing numbered sequences, identifying substructures formed by consecutive numbers. When missing areas in the table of contents are detected, a compensation mechanism is automatically activated, retaining existing table of contents nodes as fixed anchors and constructing virtual nodes within the missing areas using semantic continuity analysis. The initial segmentation scheme derived from the directory is aligned with the symbol analysis results, and areas where conflicts still exist after alignment are marked with abnormal signs, and then the sliding window analysis process is entered.

4. The RAG-based text block dynamic segmentation method according to claim 3, characterized in that: When the deviation between the table of contents segmentation boundary and the natural paragraph boundary exceeds the preset tolerance, the following processing solutions are selected based on the context density; High-density terminology area: keep directory boundaries first; Narrative text area: Use natural paragraph boundaries; Transition Zone: Inserts collapsible virtual segment markers.

5. The RAG-based text block dynamic segmentation method according to claim 3, characterized in that: The process of constructing the minimum semantic unit boundary includes the following steps: The paragraph unit obtained by using the paragraph separation mark is used as the input basic structure of the minimum semantic unit boundary; the text flow in each container unit is used as the initial processing object; Within a given paragraph unit, the text character sequence is scanned to locate the position coordinates of the sentence-end punctuation triple. The existence of the sentence-end punctuation triple is verified based on the paragraph boundary constraint. The text interval between the triples is generated into a primary semantic block, the length of which is limited by the granularity analysis results. For primary semantic blocks containing number prefixes, the list item detection mode is triggered. The similarity between the substructure formed by the consecutive numbering identified by the trigger condition and the established hierarchical markup features is judged. The consecutive numbered items automatically form a substructure tree, and the root node of the substructure tree establishes a parent-child relationship with the paragraph container node. When the sequence of primary semantic blocks is interrupted, the gap area compensation mechanism is activated, using the parsed normal structure as the anchor reference system. By analyzing the verb tense patterns and entity reference relationships of the semantic blocks on both sides of the interrupted area, the potential structural continuity is deduced, and the generated virtual nodes inherit the same hierarchical attributes as the adjacent real nodes.

6. The RAG-based text block dynamic segmentation method according to claim 1, characterized in that: The process of performing adaptive segmentation on the detected topic boundaries includes the following steps: Based on the text content within the sliding window, the topic distribution vector of the current window is calculated. The topic distribution vector represents the probability weight of each topic within the window. If the text contains a directory structure, the window is initialized to align with the directory node boundary. If there is no directory structure, the window size is generated according to the initial setting. A symmetric probability distribution difference metric is used to compare the topic distribution vectors of adjacent windows; If the similarity is lower than the preset threshold, it is determined to be a sudden change in the topic; if the similarity is higher than the threshold but shows a continuous downward trend, it is determined to be a gradual change in the topic. The window size is dynamically adjusted based on the detection results; Insert split markers at mutation points and expandable split markers in gradient areas, and associate context across boundaries during retrieval. If a split point conflicts with a directory node, prioritize preserving directory integrity and insert a bidirectional index anchor at the conflicting location. Structured output of segmentation results: type tag, original directory location, related topic distribution vector and hierarchical integration.

7. The RAG-based text block dynamic segmentation method according to claim 6, characterized in that: The calculation process of topic similarity includes the following steps: Get the topic distribution vectors of adjacent windows as input, where each topic distribution vector contains the probability weight value of each topic in the window; if it is the first calculation, initialize the reference set containing all known topics; if it is not the first time, inherit the valid topic set confirmed by the previous calculation; Normalize the topic distribution vectors of adjacent windows to ensure that the sum of all components is 1; calculate the symmetric difference between the two distribution vectors, which satisfies the three basic properties of the distance function: non-negativity, symmetry, and triangle inequality; Map the distribution difference value to a similarity score using a monotonically decreasing function. Set an adaptively adjustable conversion threshold. The default value range is based on historical statistical results. Output a normalized similarity score with a fixed value range of [0,1]. Load a preset judgment threshold based on the characteristics of the current document type. Output the final score of the topic similarity of adjacent windows, compare the final score with the judgment threshold, and generate a sudden or gradual judgment result.

8. The RAG-based text block dynamic segmentation method according to claim 7, characterized in that: The process of outputting normalized similarity scores includes the following steps: Receive symmetric difference values ​​to represent the difference between the two topic distributions; verify that the input value meets three basic mathematical constraints; record the current difference value in the historical statistical sequence; establish a conversion relationship from difference value to similarity, dynamically determine the initial morphological parameters of the conversion curve based on the characteristics of historical data, and set the boundary conditions of the conversion curve: the minimum difference corresponds to the highest similarity, and the theoretical maximum difference corresponds to zero similarity; Substitute the current symmetric difference value into the mapping function to obtain the initial similarity, and calibrate the stability of the initial result based on the recent data fluctuation characteristics; Determine the reference baseline based on the distribution characteristics of historical data, adjust the calibrated topic similarity to the standard measurement interval, and use upper and lower limits to truncate the results to within the specified range; A dynamic baseline is established by comprehensively considering the historical similarity level and current document characteristics, the threshold is adjusted considering the topic diversity index and context consistency factors, and a boundary protection mechanism is applied to prevent the threshold from drifting beyond a reasonable range.

9. The RAG-based text block dynamic segmentation method according to claim 1, characterized in that: in, Hierarchical block reorganization includes: First-level block: inherits the directory structure segmentation node; Secondary block: semantic segmentation boundary applied within directory nodes; Three-level segmentation: dynamic granularity adjustment of sliding windows is implemented for complex paragraphs.

10. A text block dynamic segmentation system based on RAG, characterized in that: Include: The segmentation mode module is responsible for parsing the multi-level directory structure of the document and segmenting the text blocks based on the directory nodes; it detects paragraph separators and sentence end symbols as secondary segmentation boundaries; For documents with complete directories, a tree-like text block structure that strictly corresponds to the directory hierarchy is generated. For documents without a directory or with an incomplete directory, a hybrid segmentation mode of rules and semantics is used, retaining the existing directory nodes as the primary segmentation points and enabling semantic segmentation for supplementary segmentation in areas where the directory is missing. The tag confirmation module is responsible for performing latent Dirichlet allocation topic modeling on the continuous text stream and calculating the topic distribution vector of each text segment in real time. It calculates the topic similarity of adjacent windows through a sliding window. When the topic similarity exceeds a threshold, it is determined to be a topic boundary and performs adaptive segmentation on the detected topic boundary. It inserts hard segmentation markers at topic mutation points and uses soft segmentation for gradually changing topic areas. The block reorganization module is responsible for setting the initial window size of the sliding window according to the document type, monitoring the semantic density within the window in real time, and hierarchical block reorganization. When the rule segmentation and semantic segmentation results conflict: prioritize preserving the integrity of the directory structure, inserting bidirectional index anchors in the conflicting area, and maintaining the context association during retrieval.

Images