A text segmentation method and apparatus for retrieval-oriented augmented generation

Abstract

The application discloses a text segmentation method and device for retrieval augmented generation, and comprises the following steps: constructing a text tree for an original text, wherein the text tree takes the original text attribute as a root node, each level title as a sub-node, and a non-title paragraph as a leaf node; performing text block segmentation according to the text tree, so that each text block to be segmented starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained by each text block is a directory structure composed of all parent nodes corresponding to each level title and the original text attribute obtained step by step upwards, starting from the head leaf node contained by each text block; and the method and device can segment the text in a manner more suitable for a tree structure, reduce the loss of text information caused in the text segmentation process, and thus improve the recall effect of an RAG system.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a text segmentation method and apparatus for retrieval-enhanced generation. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is a model that combines retrieval and generation techniques. Current RAG systems face several significant challenges when processing large-scale text corpora, which greatly impact system performance and the quality of generated content. First, the limitations of vector representation are a key issue. Although modern embedding models can convert text into high-dimensional vectors, these vectors still have limitations in representing complex semantics and contextual information. For example, long texts or texts containing multiple topics may lose some key information after vectorization, leading to inaccurate or irrelevant retrieved information.

[0003] Secondly, the contextual constraints of generative models are also an important factor. Most generative models can only retain limited contextual information when processing long texts. This means that if the text segmentation is not fine enough, the model may ignore some important contextual details when generating the response, thus affecting the quality and coherence of the generated content.

[0004] Therefore, more accurate text segmentation becomes indispensable in RAG systems. Accurate text segmentation ensures that each text fragment contains sufficient information while avoiding redundancy or loss. By segmenting text into smaller, semantically independent fragments, RAG systems can more effectively retrieve relevant information and accurately pass this information to the generative model. This not only helps improve the relevance and accuracy of the generated content but also enhances its coherence and naturalness.

[0005] Furthermore, accurate text segmentation helps RAG systems better handle long texts and avoid retrieval errors caused by the limitations of vector representation. By segmenting text into smaller fragments, RAG systems can analyze the semantics and context of each fragment in greater detail, thereby improving the overall efficiency and effectiveness of the retrieval and generation process.

[0006] Since the structure of text is essentially a tree structure, parent-child relationships naturally exist. The directory structure of text will obviously contain information that is directly or indirectly related to it. However, there is currently no technology that can effectively utilize the directory structure that already exists or may exist in the text.

[0007] The current mainstream text segmentation method segments text based on punctuation marks and greedily combines the text into segments that approach a certain length. This fails to effectively utilize the tree structure of the text, resulting in significant information loss in the segmented text segments. This is detrimental to the text block retrieval of the RAG system and thus affects the subsequent model's response performance. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide a text segmentation method and apparatus for retrieval enhancement generation, which can segment text in a way that is more suitable for a tree structure, reduce the loss of text information during the text segmentation process, and thus improve the recall effect of the RAG system.

[0009] To achieve the above-mentioned objectives, an embodiment provides a text segmentation method for retrieval enhancement generation, comprising the following steps:

[0010] Construct a text tree for the original text, where the text tree uses the original text attributes as the root node, each level of headings as child nodes, and non-heading paragraphs as leaf nodes.

[0011] The text is segmented based on the text tree, so that each segmented text block starts with a directory structure and ends with a non-heading paragraph, and is constrained by a preset text block character limit. The directory structure contained in each text block is: starting from the head leaf node contained in each text block, the directory structure is composed of the headings and original text attributes of all parent nodes obtained level by level upward.

[0012] Preferably, constructing a text tree for the original text includes:

[0013] The original text is classified into paragraphs, where the level of the heading is its own level and the level of the body text is the maximum heading level plus one. After determining the level of each paragraph, the original text attribute is used as the root node, and the text tree is constructed in a way that the lower the level, the closer it is to the root node in the text tree.

[0014] Preferably, during the construction of the text tree, the length of the title is also determined. If the number of characters in the title exceeds 20% of the preset text block character limit, then the level where the title is located and all levels thereafter are considered to be marked as non-title paragraphs, but the title level itself remains unchanged. This marked non-title paragraph is only used to determine whether it is used as the end of the text block when merging text.

[0015] Preferably, the method further includes: during the process of dividing the text block of the current node, if the current segment containing the content of multiple nodes exceeds the preset text block character limit when it is added to the text block, the current segment will be subjected to an enhancement processing stage before being added to the text block, so as to constrain each text block to end with a non-heading paragraph.

[0016] Preferably, the enhancement processing stage before placing the currently extracted segment into a text block includes:

[0017] First, determine whether the previous segment in the current text block contains body text or a title marked as a non-title paragraph. If it does, backtrack to the content of the closest node to the current text block in the previous segment, either the body text or the title marked as a non-title paragraph, and use that as the end content of the current text block. The subsequent title node will then be used as the starting node for the next segment. If it does not contain body text or a title marked as a non-title paragraph, then the current segment will be added to the current text block. If the current segment contains body text or a title marked as a non-title paragraph, then the current segment will be used as the end content of the current text block. Otherwise, continue to the next node to divide the text block.

[0018] Preferably, the method further includes: before dividing the text blocks, merging the non-heading paragraphs in the text tree, specifically first merging the list blocks, reference blocks, table blocks, and code blocks that are syntactically considered as a whole; and then splitting all the merged list blocks, reference blocks, table blocks, code blocks, and ordinary text blocks to constrain each block to within a preset text block character limit.

[0019] Preferably, the method used for segmentation includes: first segmenting by line, then greedily merging; and when segmenting excessively long blocks, first segmenting by punctuation marks, and selecting to overlap one sentence of text on the left and right sides of the segmented block.

[0020] To achieve the above-mentioned objectives, the embodiments also provide a text segmentation device for retrieval enhancement generation, comprising:

[0021] The text tree building module is used to build a text tree for the original text, where the text tree uses the original text attributes as the root node, each heading level as a child node, and non-heading paragraphs as leaf nodes.

[0022] The text block segmentation module is used to segment text blocks based on the text tree, so that each segmented text block starts with a directory structure, ends with a non-heading paragraph, and is constrained by a preset text block character limit. The directory structure contained in each text block is: starting from the header leaf node contained in each text block, the directory structure is composed of the headings and original text attributes of all parent nodes obtained level by level upward.

[0023] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned text segmentation method for retrieval enhancement generation.

[0024] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described text segmentation method for retrieval enhancement generation.

[0025] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0026] This invention utilizes the directory structure of text and appends it to the beginning of the segmented text fragments, making the information presented in the text fragments more complete and easier for the recall process and model to understand and respond in the RAG system. At the same time, it segments according to the preset text block character limit and text structure, and ensures that the end of the segment is always a non-heading paragraph, reducing the impact of text segmentation on the loss of text information. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the text segmentation method for retrieval enhancement provided in the embodiment;

[0029] Figure 2 This is a flowchart of the text tree construction process provided in the embodiment;

[0030] Figure 3 This is a flowchart of text block division based on a text tree provided in the embodiment;

[0031] Figure 4 This is a schematic diagram of the text segmentation device for retrieval enhancement provided in the embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0033] like Figure 1 As shown in the embodiment, a text segmentation method for retrieval enhancement generation includes the following steps:

[0034] S1 constructs a text tree for the original text, where the original text attributes are the root node, each level of headings are child nodes, and non-heading paragraphs are leaf nodes.

[0035] In this embodiment, constructing the original text into a text tree facilitates subsequent text block segmentation. The specific process of constructing the text tree is as follows: The original text undergoes text format extraction, i.e., paragraph hierarchical classification. The heading level is its own level, and the body text level is the highest heading level plus one. After determining the level of each paragraph, the original text attribute is used as the root node, and the text tree is constructed according to the principle that lower-level text is closer to the root node in the text tree. It should be noted that the heading levels follow the conventional text heading levels, including level 1, level 2, level 3, etc., where level 2 headings are under level 1 headings, showing a clear hierarchical relationship. When only level 3 headings exist, the body text level is level 3 heading plus one, i.e., level 4. Based on this, in the constructed text tree, the root node is the original text attribute, the child nodes are the headings of each level, and the leaf nodes are non-heading paragraph text.

[0036] It should also be noted that during the construction of the text tree, the length of the title is also determined. When the number of characters in the title exceeds 20% of the preset text block character limit, the level where the title is located and all levels after it are considered to be marked as non-title paragraphs, but the title level itself remains unchanged. This marked non-title paragraph is only used to determine whether it is used as the end of the text block when merging text.

[0037] S2, based on the text tree, divides the text into blocks such that each divided text block starts with the directory structure, ends with a non-heading paragraph, and is constrained by a preset text block character limit.

[0038] In this embodiment, before segmenting the text blocks, non-heading paragraphs in the text tree are merged. Specifically, list blocks, quotation blocks, table blocks, and code blocks that are syntactically considered as a whole are first merged. To ensure that the word count limit is met, all merged list blocks, quotation blocks, table blocks, code blocks, and ordinary text blocks are then segmented to constrain each block to within the preset text block word count limit. The segmentation operation first segments by line and then greedily merges them, while also incorporating sliding window segmentation. When segmenting very long blocks, they are first segmented by punctuation marks, and then one sentence of text is overlapped on each side of the segment.

[0039] In this embodiment, when dividing text blocks, the text tree is traversed from top to bottom and from left to right. When the traversal reaches the head leaf node that should be considered the next text block, the headings of all parent nodes are retrieved level by level based on the position of the text block, until the original text attributes of the root node are added to form a directory structure. This directory structure is then added to the text block, and its word count is included in subsequent text block additions to prevent exceeding the word limit after merging. Adding the directory structure information of the text block to the text block can improve recall and enhance the model's understanding of the text block content.

[0040] like Figure 3 As shown in the embodiment, during the process of dividing the text block of the current node, if the current segment containing the content of multiple nodes exceeds the preset text block character limit when it is added to the text block, the current segment will undergo an enhancement process before being added to the text block, so as to constrain each text block to end with a non-heading paragraph.

[0041] The enhancement process before placing the currently extracted segment into a text block includes:

[0042] First, it checks if the previous segment in the current text block contains body text or a title marked as a non-title paragraph. If it does, it backtracks to the content of the closest node in the previous segment—either the body text or the title marked as a non-title paragraph—as the end of the current text block, and the subsequent title node becomes the start node for the next segment. If it doesn't contain body text or a title marked as a non-title paragraph, the current segment is added to the current text block. If the current segment contains body text or a title marked as a non-title paragraph, it becomes the end of the current text block; otherwise, it continues to the next node for text block segmentation. This ensures that the end of a text block is never a title, optimizing the structure and information contained in the segmented text blocks while preventing program loops.

[0043] like Figure 4 As shown in the embodiment, a text segmentation device 40 for retrieval enhancement generation includes a text tree construction module 41 and a text block segmentation module 42. The text tree construction module 41 is used to construct a text tree for the original text, wherein the text tree uses the original text attributes as the root node, each heading level as a child node, and non-heading paragraphs as leaf nodes. The text block segmentation module 42 is used to segment text blocks according to the text tree, such that each segmented text block starts with a directory structure, ends with a non-heading paragraph, and is constrained by a preset text block character limit. The directory structure contained in each text block is: starting from the header leaf node contained in each text block, a directory structure composed of the headings and original text attributes corresponding to all parent nodes obtained level by level upwards.

[0044] The core of this invention is to use a text directory structure to append to the segmented text fragments, making the information presented in the text fragments more complete. At the same time, it is constrained by a preset text block character limit, reducing the impact of text segmentation on the loss of text information, and making it easier for the recall process and model understanding and response in the RAG system.

[0045] It should be noted that the text segmentation device for retrieval-enhanced generation provided in the above embodiments should be illustrated using the above-described functional module divisions as examples when performing text segmentation. The functions can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the text segmentation device for retrieval-enhanced generation provided in the above embodiments and the text segmentation method embodiments for retrieval-enhanced generation belong to the same concept. The specific implementation process is detailed in the text segmentation method embodiments for retrieval-enhanced generation, and will not be repeated here.

[0046] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the aforementioned text segmentation method for retrieval enhancement generation, specifically including the following steps:

[0047] S1 constructs a text tree for the original text, where the text tree uses the original text attributes as the root node, each level of headings as child nodes, and non-heading paragraphs as leaf nodes.

[0048] S2, divide the text into blocks according to the text tree, so that each divided text block starts with a directory structure and ends with a non-heading paragraph, and is constrained by a preset text block character limit. The directory structure contained in each text block is: starting from the head leaf node contained in each text block, the directory structure is composed of the headings and original text attributes of all parent nodes obtained level by level upward.

[0049] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the text segmentation method for retrieval-enhanced generation described in S1-S2 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0050] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium having a program stored thereon. When executed by a processor, the program implements the above-described text segmentation method for retrieval enhancement generation, specifically including the following steps:

[0051] S1 constructs a text tree for the original text, where the text tree uses the original text attributes as the root node, each level of headings as child nodes, and non-heading paragraphs as leaf nodes.

[0052] S2, divide the text into blocks according to the text tree, so that each divided text block starts with a directory structure and ends with a non-heading paragraph, and is constrained by a preset text block character limit. The directory structure contained in each text block is: starting from the head leaf node contained in each text block, the directory structure is composed of the headings and original text attributes of all parent nodes obtained level by level upward.

[0053] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0054] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A search-oriented enhanced generation text segmentation method, characterized by, The method comprises the following steps: constructing a text tree for the original text, wherein the text tree takes the original text attribute as a root node, each level title as a child node, and a non-title paragraph as a leaf node; performing text block division according to the text tree, so that each text block to be divided starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block character limit, wherein the directory structure contained in each text block comprises all the titles and the original text attribute corresponding to the father nodes obtained level by level from the head leaf node contained in each text block as the starting point; when performing text block division, the text tree is traversed from top to bottom and from left to right, and if the current intercepted segment containing the contents of multiple nodes exceeds the preset text block character limit when being added to the text block, the current intercepted segment is subjected to an enhancement processing stage before being put into the text block, so as to constrain each text block to end with a non-title paragraph; the enhancement processing stage before the current intercepted segment is put into the text block comprises: firstly judging whether the last intercepted segment in the current text block contains a main text or a title marked as a non-title paragraph, if yes, backtracking to the main text or the title node content marked as a non-title paragraph of the node closest to the current text block to serve as the ending content of the current text block, and the title node thereafter serving as the starting node of the next intercepted segment, if not, dividing the current intercepted segment into the current text block, and if the current intercepted segment contains a main text or a title marked as a non-title paragraph, taking the current intercepted segment as the ending content of the current text block, otherwise, continuing text block division with the next node; in the construction process of the text tree, the length of the title is also determined, and if the number of characters of the title exceeds 20% of the preset text block character limit, it is considered that the level of the title and all the levels thereafter are marked as non-title paragraphs, but the title level itself is maintained, and the marked non-title paragraphs are only used to judge whether to be taken as the ending of the text block during text merging.

2. The search-oriented enhanced generation text segmentation method of claim 1, wherein, constructing a text tree for the original text, comprising: classifying the text paragraphs of the original text, wherein the level of the title is the title level itself, and the level of the main text is the maximum title level plus one, and after determining the level of each text, taking the original text attribute as a root node, constructing the text tree in a manner that the lower the level, the closer to the root node in the text tree.

3. The search-oriented enhanced generation text segmentation method of claim 1, wherein, further comprising: before text block division, performing merging processing on the non-title paragraphs in the text tree, specifically, firstly performing a merging operation on the list blocks, quote blocks, table blocks and code blocks that are considered as a whole in grammar; and then performing a division on all the merged list blocks, quote blocks, table blocks, code blocks and ordinary text blocks, so as to constrain each block obtained within the preset text block character limit.

4. The search-oriented enhanced generation text segmentation method of claim 3, wherein, The method for performing division comprises: firstly performing line division, and then performing greedy merging, and when cutting a long block, firstly cutting according to punctuation symbols, and selecting to overlap one sentence of text on the left and right of the cut block.

5. A search-oriented enhanced generation text segmentation apparatus, characterized by, ​ a text tree construction module, configured to construct a text tree for the original text, wherein the text tree takes the original text attribute as a root node, each title level as a sub-node, and a non-title paragraph as a leaf node; a text block segmentation module, configured to segment the text blocks according to the text tree, so that each segmented text block starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting from the head leaf node contained in each text block, all parent nodes obtained level by level correspond to the directory structure of each title and the original text attribute; when the text blocks are divided, the text tree is traversed from top to bottom and from left to right, and when the current text block division of the current node is performed, if the current intercepted segment containing multiple node contents exceeds the preset text block word limit when being added to the text block, the current intercepted segment is subjected to an enhancement processing stage before being put into the text block, so as to constrain each text block to end with a non-title paragraph; the enhancement processing stage before the current intercepted segment is put into the text block includes: firstly judging whether the last intercepted segment in the current text block contains a main text or a title marked as a non-title paragraph, if yes, backtracking to the main text or the title node content marked as a non-title paragraph of the node closest to the current text block to serve as the end content of the current text block, and the subsequent title node serves as the starting node of the next intercepted segment, if not, dividing the current intercepted segment into the current text block, and if the current intercepted segment contains a main text or a title marked as a non-title paragraph, taking the current intercepted segment as the end content of the current text block, otherwise, continuing the text block division of the next node; in the construction process of the text tree, the length of the title is also determined, and when the title word number exceeds 20% of the preset text block word limit, it is considered that the title level and all levels after the title level are marked as non-title paragraphs, but the title level itself is maintained unchanged, and the marked non-title paragraph is only used to judge whether to be taken as the end of the text block during text merging.

6. A computing device comprising a memory and one or more processors, the memory having stored therein executable code, wherein, The one or more processors execute the executable code to implement the text segmentation method for retrieval enhancement generation in any one of claims 1-4.

7. A computer readable storage medium characterized in that, A program is stored thereon, and the program is executed by the processor to implement the text segmentation method for retrieval enhancement generation in any one of claims 1-4.

Images