Text segmentation method and device for retrieval enhancement generation

Abstract

The invention discloses a text segmentation method and device for retrieval enhancement generation, and the method comprises the steps: constructing a text tree for an original text, and enabling the text tree to take an original text attribute as a root node, a title of each level as a child node, and a non-title paragraph as a leaf node; according to the method, text blocks are segmented according to a text tree, so that each segmented text block starts from a directory structure, ends with a non-title paragraph and is constrained by preset text block word number limitation, and the directory structure included in each text block is as follows: starting from a head leaf node included in each text block, starting from a non-title paragraph included in each text block, and ending with a non-title paragraph included in each text block; according to the method and the device, the text can be segmented in a mode more suitable for a tree structure, and text information loss caused in the text segmentation process is reduced, so that the recall effect of an RAG system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a text segmentation method and device for retrieval-enhanced generation. Background Art

[0002] ‌RAG system (Retrieval-Augmented Generation) is a model that combines retrieval and generation techniques. Current RAG systems face some significant challenges when processing large-scale text corpora, which greatly affect the performance of the system and the quality of generated content. First, the limitations of vector representation are a key issue. Although modern embedding models are able to convert text into high-dimensional vectors, these high-dimensional vectors still have certain limitations in representing complex semantics and contextual information. For example, long texts or texts containing multiple topics may lose some key information after vectorization, resulting in the retrieved information being inaccurate or relevant.

[0003] Secondly, the contextual limitation of the generative model is also an important factor. Most generative models can usually only retain limited contextual information when processing long texts. This means that if the text segmentation is not fine enough, the model may ignore some important contextual details when generating responses, thus affecting the quality and coherence of the generated content.

[0004] Therefore, more accurate text segmentation becomes indispensable in RAG systems. Accurate text segmentation ensures that each text segment contains enough information while avoiding information redundancy or loss. By segmenting text into smaller, semantically independent segments, RAG systems can more effectively retrieve relevant information and accurately pass this information to the generation model. This not only helps to improve the relevance and accuracy of the generated content, but also enhances the coherence and naturalness of the generated content.

[0005] In addition, accurate text segmentation can also help the RAG system better handle long texts and avoid retrieval errors caused by the limitations of vector representation. By segmenting the text into smaller segments, the RAG system can analyze the semantics and context of each segment more carefully, thereby improving the overall efficiency and effectiveness of the retrieval and generation process.

[0006] Since the structure of a text is essentially a tree structure, there is a natural parent-child relationship. The directory structure of the text will obviously contain information directly or indirectly related to it. However, there is currently no technology to effectively utilize the directory structure that originally exists or may exist in the text.

[0007] The current mainstream text segmentation method is to segment the text according to punctuation marks and greedily piece the text into segments of a certain length. It fails to effectively utilize the tree structure of the text, resulting in serious information loss in the segmented text segments, which is not conducive to the text block recall of the RAG system, thereby affecting the subsequent model answering effect. Summary of the invention

[0008] In view of the above, the purpose of the present invention is to provide a text segmentation method and device for retrieval enhanced generation, which can segment the text in a way that is more suitable for the tree structure, reduce the loss of text information caused during the text segmentation process, and thus improve the recall effect of the RAG system.

[0009] To achieve the above-mentioned object of the invention, an embodiment provides a text segmentation method for retrieval-enhanced generation, comprising the following steps: Build a text tree for the original text, where the text tree has the original text attributes as the root node, each level of title as the child node, and non-title paragraphs as the leaf node; The text blocks are segmented according to the text tree, so that each segmented text block starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and the directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained level by level.

[0010] Preferably, constructing a text tree for the original text includes: The original text is graded into paragraphs, where the level of the title is its own level, and the level of the body text is the maximum title level plus one. After determining the level of each paragraph of text, the original text attribute is used as the root node, and a text tree is constructed in such a way that the lower the level, the closer it is to the root node in the text tree.

[0011] Preferably, during the construction of the text tree, the length of the title is also determined. When the number of words in the title exceeds 20% of the preset text block word limit, the level where the title is located and all levels thereafter are considered to be marked as non-title paragraphs, but the title level itself is maintained unchanged. This marked non-title paragraph is only used to determine whether it is regarded as the end of a text block when text is merged.

[0012] Preferably, the method further includes: in the process of dividing the text block of the current node, if the current clip containing the contents of multiple nodes exceeds the preset text block word limit when added to the text block, the current clip will be subjected to an enhanced processing stage before being placed in the text block to constrain each text block to end with a non-title paragraph.

[0013] Preferably, the enhancement processing stage of the current intercepted segment before being placed into the text block includes: First, determine whether the previous clip in the current text block contains the main text or the title marked as a non-title paragraph. If it does, then go back to the node body or the title node content marked as a non-title paragraph in the previous clip that is closest to the current text block as the end content of the current text block, and the subsequent title node as the starting node of the next clip. If it does not contain it, the current clip is divided into the current text block. When it is determined that the current clip contains the main text or the title marked as a non-title paragraph, the current clip is used as the end content of the current text block, otherwise continue to the next node to divide the text block.

[0014] Preferably, the method further comprises: before dividing the text blocks, merging the non-heading paragraphs in the text tree, specifically first merging the list blocks, quotation blocks, table blocks, and code blocks that are grammatically regarded as a whole; and then segmenting all the merged list blocks, quotation blocks, table blocks, code blocks, and ordinary text blocks to constrain each obtained block to be within the preset text block word limit.

[0015] Preferably, the segmentation method includes: segmentation by line first, and then greedy merging. When cutting an extra-long block, it is first cut by punctuation marks, and a sentence of text is selected to overlap on the left and right of the cut block.

[0016] To achieve the above-mentioned purpose of the invention, the embodiment further provides a text segmentation device for retrieval enhancement generation, comprising: A text tree construction module is used to construct a text tree for the original text, wherein the text tree has the original text attributes as the root node, each heading level as a child node, and non-heading paragraphs as leaf nodes; A text block segmentation module is used to segment text blocks according to a text tree, so that each segmented text block starts with a directory structure, ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and a directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained step by step upward.

[0017] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned text segmentation method for retrieval-enhanced generation.

[0018] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned text segmentation method for retrieval-enhanced generation is implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention utilizes the directory structure of the text and appends it to the segmented text fragments, so that the information presented in the text fragments is more complete and easier for the recall process and model understanding and answering in the RAG system. At the same time, the text block is segmented according to the preset word count limit and text structure, and it is ensured that the end of the segmentation is definitely a non-title paragraph text, thereby reducing the impact of text segmentation on the loss of text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a text segmentation method for retrieval-enhanced generation provided by an embodiment; Figure 2 is a flowchart of constructing a text tree provided by an embodiment; Figure 3 is a flowchart of dividing text blocks based on a text tree provided in an embodiment; Figure 4 It is a structural schematic diagram of a text segmentation device for retrieval-enhanced generation provided in an embodiment. DETAILED DESCRIPTION

[0022] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0023] like Figure 1 As shown, the embodiment provides a text segmentation method for retrieval-enhanced generation, comprising the following steps: S1, constructs a text tree for the original text, where the text tree uses the original text attributes as the root node, each level of titles as child nodes, and non-title paragraphs as leaf nodes.

[0024] In the embodiment, constructing the original text as a text tree is more convenient for subsequent text block segmentation. The specific process of constructing the text tree is: extracting the text format of the original text, that is, grading the text paragraphs, wherein the level of the title is its own level, and the level of the text is the maximum title level plus one. After determining the level of each paragraph of text, the original text attribute is used as the root node, and the text tree is constructed in a manner that the lower the level, the closer to the root node in the text tree. It should be noted that the title level is in accordance with the conventional text title level, including a first-level title, a second-level title, a third-level title, etc., wherein the second-level title is under the first-level title, and has an obvious hierarchical relationship. When there is only a third-level title, the text level is the third-level title plus one, that is, the fourth level. Based on this, in the constructed text tree, the root node is the original text attribute, the child node is the title of each level, and the leaf node is the non-title paragraph text.

[0025] It should also be noted that during the construction of the text tree, the length of the title is also determined. When the number of words in the title exceeds 20% of the preset text block word limit, the level where the title is located and all levels after it are considered to be marked as non-title paragraphs, but the title level itself remains unchanged. This marked non-title paragraph is only used to determine whether it is regarded as the end of a text block when merging text.

[0026] S2, dividing the text blocks according to the text tree, so that each divided text block starts with a directory structure, ends with a non-title paragraph, and is constrained by a preset text block word limit.

[0027] In the embodiment, before the text block segmentation is performed, the non-title paragraphs in the text tree are merged, specifically, the list blocks, quotation blocks, table blocks, and code blocks that are grammatically regarded as a whole are first merged; in order to ensure that the word limit requirement is met, all the merged list blocks, quotation blocks, table blocks, code blocks, and ordinary text blocks are then segmented once to constrain each block to be within the preset text block word limit. Among them, the segmentation operation adopts first segmentation by line and then greedy merging, and at the same time draws on the sliding window segmentation. When cutting an overlong block, it will first be cut by punctuation marks, and a sentence of text will be overlapped on the left and right of the cut block.

[0028] In the embodiment, when dividing the text block, the text tree will be traversed from top to bottom and from left to right. When traversing to the head leaf node that should be regarded as the next text block, the position of the text block will be firstly targeted, and the titles of all levels corresponding to all parent nodes will be obtained step by step upward until the original text attribute corresponding to the root node is added to form a directory structure, and the directory structure is added to the text block. The number of words in the text block is counted in the subsequent addition of the text block to prevent the word limit from being exceeded after the merger. Adding the directory structure information of the text block to the text block can improve the recall effect and enhance the model's understanding of the content of the text block.

[0029] like Figure 3 As shown, in the embodiment, during the process of dividing the text block of the current node, if the current clip containing the contents of multiple nodes exceeds the preset text block word limit when added to the text block, the current clip will be subjected to an enhanced processing stage before being placed in the text block to constrain each text block to end with a non-title paragraph.

[0030] The enhancement processing stage of the current intercepted segment before being placed into the text block includes: First, determine whether the previous clip in the current text block contains the main text or the title marked as a non-title paragraph. If it does, then go back to the node closest to the current text block in the previous clip or the title node marked as a non-title paragraph as the end content of the current text block, and the subsequent title node as the starting node of the next clip. If it does not contain it, divide the current clip into the current text block, and when it is determined that the current clip contains the main text or the title marked as a non-title paragraph, then use the current clip as the end content of the current text block, otherwise continue to the next node to divide the text block. This is to ensure that the end of the text block is definitely not a title, optimize the structure of the segmented text block and the information contained, and avoid program dead loops.

[0031] like Figure 4 As shown, an embodiment provides a text segmentation device 40 for retrieval-enhanced generation, including a text tree construction module 41 and a text block segmentation module 42, wherein the text tree construction module 41 is used to construct a text tree for the original text, wherein the text tree uses the original text attributes as the root node, each title level as the child node, and the non-title paragraph as the leaf node; the text block segmentation module 42 is used to perform text block segmentation based on the text tree, so that each segmented text block starts with a directory structure, ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and a directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained step by step upward.

[0032] The main core of the method of the present invention is to use the directory structure of the text and append it to the segmented text fragments to make the information presented in the text fragments more complete. At the same time, constraints are imposed based on the preset word count limit of the text block to reduce the impact of text segmentation on the loss of text information, making it easier for the recall process and model understanding and answering in the RAG system.

[0033] It should be noted that the text segmentation device for retrieval-enhanced generation provided in the above embodiment should be illustrated by the division of the above functional modules when performing text segmentation. The above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the text segmentation device for retrieval-enhanced generation provided in the above embodiment belongs to the same concept as the text segmentation method embodiment for retrieval-enhanced generation. The specific implementation process is detailed in the text segmentation method embodiment for retrieval-enhanced generation, which will not be repeated here.

[0034] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the method for text segmentation for retrieval-enhanced generation is implemented, specifically including the following steps: S1, construct a text tree for the original text, where the text tree takes the original text attributes as the root node, the titles of each level as child nodes, and the non-title paragraphs as leaf nodes; S2, divide the text blocks according to the text tree, so that each divided text block starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and the directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained level by level.

[0035] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the text segmentation method for retrieval-enhanced generation described in S1-S2 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0036] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned text segmentation method for retrieval-enhanced generation is implemented, which specifically includes the following steps: S1, construct a text tree for the original text, where the text tree takes the original text attributes as the root node, the titles of each level as child nodes, and the non-title paragraphs as leaf nodes; S2, divide the text blocks according to the text tree, so that each divided text block starts with a directory structure and ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and the directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained level by level.

[0037] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0038] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A text segmentation method for retrieval-enhanced generation, characterized in that: The following steps are involved: Build a text tree for the original text, where the text tree has the original text attributes as the root node, each level of title as the child node, and non-title paragraphs as the leaf node; The text blocks are segmented according to the text tree, so that each segmented text block starts with a directory structure, ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and the directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained level by level.

2. The text segmentation method for retrieval-enhanced generation according to claim 1, characterized in that: Build a text tree for the original text, including: The original text is graded into paragraphs, where the level of the title is its own level, and the level of the body text is the maximum title level plus one. After determining the level of each paragraph of text, the original text attribute is used as the root node, and a text tree is constructed in such a way that the lower the level, the closer it is to the root node in the text tree.

3. The text segmentation method for retrieval-enhanced generation according to claim 1 or 2, characterized in that: During the construction of the text tree, the length of the title is also determined. When the number of words in the title exceeds 20% of the preset text block word limit, the level where the title is located and all levels after it are considered to be marked as non-title paragraphs, but the title level itself remains unchanged. This marked non-title paragraph is only used to determine whether it is regarded as the end of a text block when merging text.

4. The text segmentation method for retrieval-enhanced generation according to claim 3 is characterized in that: Also includes: During the process of dividing the text block of the current node, if the current clip containing the contents of multiple nodes exceeds the preset text block word limit when added to the text block, the current clip will be subjected to an enhanced processing stage before being placed in the text block to constrain each text block to end with a non-title paragraph.

5. The text segmentation method for retrieval-enhanced generation according to claim 4, characterized in that: The enhanced processing stage before the current intercepted fragment is put into the text block includes: First, determine whether the previous clip in the current text block contains the main text or the title marked as a non-title paragraph. If it does, then go back to the node body or the title node content marked as a non-title paragraph in the previous clip that is closest to the current text block as the end content of the current text block, and the subsequent title node as the starting node of the next clip. If it does not contain it, the current clip is divided into the current text block. When it is determined that the current clip contains the main text or the title marked as a non-title paragraph, the current clip is used as the end content of the current text block, otherwise continue to the next node to divide the text block.

6. The text segmentation method for retrieval-enhanced generation according to claim 1, characterized in that: Also includes: Before dividing the text blocks, the non-heading paragraphs in the text tree are merged. Specifically, the list blocks, quotation blocks, table blocks, and code blocks that are grammatically regarded as a whole are first merged; then all the merged list blocks, quotation blocks, table blocks, code blocks, and ordinary text blocks are segmented to constrain each block to be within the preset text block word limit.

7. The text segmentation method for retrieval-enhanced generation according to claim 6, characterized in that: The segmentation method includes: first segmentation by line, then greedy merging, and when cutting an extra-long block, it will first be cut by punctuation marks, and a sentence of text will be overlapped on the left and right sides of the cut block.

8. A text segmentation device for retrieval-enhanced generation, characterized in that: include: A text tree construction module is used to construct a text tree for the original text, wherein the text tree has the original text attributes as the root node, each heading level as a child node, and non-heading paragraphs as leaf nodes; A text block segmentation module is used to segment text blocks according to a text tree, so that each segmented text block starts with a directory structure, ends with a non-title paragraph, and is constrained by a preset text block word limit, wherein the directory structure contained in each text block is: starting with the head leaf node contained in each text block, and a directory structure consisting of all levels of titles and original text attributes corresponding to all parent nodes obtained step by step upward.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the text segmentation method for retrieval-enhanced generation described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the text segmentation method for retrieval-enhanced generation described in any one of claims 1 to 7 is implemented.

Images