Text partitioning method and system, retrieval enhancement generation device, electronic equipment, storage medium and computer program product

Abstract

The invention provides a text partitioning method and system, a retrieval enhancement generation device, electronic equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a to-be-processed text; based on a preset length threshold interval of the target text block, adding the to-be-processed text of which the length is smaller than an upper limit value of the length threshold interval into the text block set; segmenting the to-be-processed text of which the length is not less than the upper limit value of the length threshold interval to obtain segmented natural segments, and sequentially adding the segmented natural segments into a paragraph list; traversing the paragraph list, and adding segmented natural segments of which the lengths are within a length threshold interval into a text block set; carrying out character string splicing fusion on the segmented natural segments of which the lengths are smaller than the lower limit value of the length threshold interval; and performing semantic hierarchy boundary segmentation on the segmented natural segments of which the lengths are not less than the upper limit value of the length threshold interval. According to the method, the text block result set which is balanced in length and can better keep semantic integrity can be generated.

Description

Technical Field

[0001] The present invention relates to the field of retrieval enhancement technology, and in particular to a text segmentation method, system, retrieval enhancement generation device, electronic equipment, storage medium and computer program product. Background Art

[0002] Retrieval-enhanced generation is one of the most popular large-scale cutting-edge technologies. The retrieval-enhanced generation model combines language models and information retrieval technology. Specifically, when the model needs to generate text or answer questions, it will first retrieve relevant information from a large document collection, and then use the retrieved information to guide the generation of text, thereby improving the quality and accuracy of the prediction. The document collection must be reasonably organized and processed in order to perform effective and efficient information retrieval on it. In the process of building a retrieval-enhanced generation system, the basic link of text segmentation is often underestimated, but its importance cannot be ignored.

[0003] In the related art, the main methods of text segmentation are: fixed-size segmentation method, random segmentation method, sliding window method and context-aware segmentation method. However, the fixed-size segmentation method and the random segmentation method will truncate the natural semantic paragraphs of the text, and may lose key information at the segmentation boundary, affecting the integrity of the text information; the sliding window method may cause the text information inside the previous and next blocks to be roughly truncated; the context-aware segmentation method may cause the text blocks to be too small, too fragmented, and the segmentation is simple, without considering the semantic hierarchy structure of the text segmentation. Summary of the invention

[0004] The purpose of the present invention is to provide a text segmentation method, system, retrieval enhancement generation device, electronic device, storage medium and computer program product, which solve the problem of text block length balance while maintaining semantic integrity.

[0005] According to a first aspect of an embodiment of the present application, a text segmentation method is provided, the method comprising:

[0006] Get the text to be processed;

[0007] Based on a preset length threshold interval of a target text block, add the to-be-processed text whose length is less than the upper limit value of the length threshold interval into a text block set; segment the to-be-processed text whose length is not less than the upper limit value of the length threshold interval to obtain segmented natural segments, and add the segmented natural segments into a paragraph list in sequence; traverse the paragraph list, and add the segmented natural segments whose length is within the length threshold interval into the text block set; perform string concatenation and fusion on the segmented natural segments whose length is less than the lower limit value of the length threshold interval; and perform semantic level boundary segmentation on the segmented natural segments whose length is not less than the upper limit value of the length threshold interval.

[0008] In one implementation, the step of concatenating and fusing the segmented natural segments whose lengths are less than the lower limit of the length threshold interval comprises:

[0009] According to the paragraph list, select the smaller segmented natural paragraphs in the adjacent paragraphs as the splicing paragraphs;

[0010] Based on the length of the spliced ​​paragraph, the length of the segmented natural segment and the upper limit value of the length threshold interval, the spliced ​​paragraph and the natural segment are spliced ​​and merged;

[0011] The above operation is repeated until all the segmented natural segments whose length is less than the lower limit value of the length threshold interval are processed.

[0012] In one implementation, after concatenating and fusing the segmented natural segments whose lengths are less than the lower limit of the length threshold interval, the method includes:

[0013] The character string obtained by concatenating and merging the segmented natural paragraphs is added to the corresponding position of the segmented natural paragraph in the paragraph list, and the segmented natural paragraph is deleted from the paragraph list.

[0014] In one embodiment, before segmenting the segmented natural segments whose length is not less than the upper limit value of the length threshold interval into semantic level boundaries, the method includes:

[0015] Perform integer division and modulo operation on the length of the segmented natural segment and the threshold median respectively to obtain a first threshold quotient and a first threshold remainder, wherein the threshold median is an integer value obtained by averaging the upper limit and the lower limit of the length threshold interval;

[0016] Obtaining a target number of segments based on the first threshold quotient and the first threshold remainder;

[0017] A target segmentation length is obtained based on the target number of segments and the length of the natural segment of the segment.

[0018] In one embodiment, segmenting the segmented natural segments whose length is not less than the upper limit value of the length threshold interval by semantic level boundaries includes:

[0019] Segmenting the segmented paragraphs to obtain segmented sentences;

[0020] Segment the segmented sentences whose length is not less than the upper limit value of the length threshold interval to obtain segmented sub-sentences; cut the segmented sub-sentences whose length is not less than the upper limit value of the length threshold interval to obtain cut-off text blocks, and add the cut-off text blocks to the list to be spliced; add the segmented sub-sentences whose length is less than the upper limit value of the length threshold interval to the list to be spliced; add the segmented sentences whose length is less than the upper limit value of the length threshold interval to the list to be spliced, and the list to be spliced ​​is used for paragraph splicing reconstruction;

[0021] Initialize the string buffer, and obtain the text to be spliced ​​according to the list to be spliced;

[0022] Based on the target segmentation length, the length of the string buffer, and the length of the segmentation sentence, the text to be spliced ​​is added to the buffer or the string in the buffer is added to the paragraph list, the buffer is reset, and the text to be spliced ​​is added to the buffer;

[0023] Repeat the above steps until all the texts to be spliced ​​are processed.

[0024] In one embodiment, the step of performing length truncation on the segmented sentences having a length not less than the upper limit value of the length threshold interval to obtain a truncated text block, and adding the truncated text block to the to-be-joined list includes:

[0025] Dividing the length of the segmentation clause by a threshold median value to obtain a second threshold quotient and a second threshold remainder, wherein the threshold median value is an integer value obtained by averaging an upper limit value and a lower limit value of the length threshold interval;

[0026] Obtaining the number of intercepted segments based on the second threshold quotient and the second threshold remainder;

[0027] Obtaining a target truncation length based on the number of truncation segments and the length of the truncation sub-sentences;

[0028] Based on the target truncation length, the segmented sentences are truncation-lengthed to obtain truncation text blocks, and the truncation text blocks are added to the to-be-joined list.

[0029] According to a second aspect of the embodiment of the present application, a text segmentation system is provided, wherein the system is configured to execute the above method, and includes: an acquisition module and a segmentation module.

[0030] The acquisition module is configured to acquire the text to be processed;

[0031] The segmentation module is configured to add the to-be-processed text whose length is less than the upper limit of the length threshold interval into the text block set based on a preset length threshold interval of the target text block; segment the to-be-processed text whose length is not less than the upper limit of the length threshold interval to obtain segmented natural segments; directly add the segmented natural segments whose length is within the length threshold interval into the text block set; perform string concatenation and fusion on the segmented natural segments whose length is less than the lower limit of the length threshold interval to obtain the target text block; perform semantic level boundary segmentation on the segmented natural segments whose length is not less than the upper limit of the length threshold interval to obtain the target text block.

[0032] According to a third aspect of an embodiment of the present application, a retrieval enhancement generation device is provided, wherein the device is configured to: based on a text block set generated by the above method, perform text vectorization on text blocks in the text block set.

[0033] According to a fourth aspect of an embodiment of the present application, an electronic device includes a memory and a processor, wherein the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the above method.

[0034] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium stores a computer program thereon, characterized in that when the executable computer program in the storage medium is executed by a processor, the above method can be implemented.

[0035] According to a sixth aspect of an embodiment of the present application, a computer program product includes a computer program, characterized in that when the computer program is executed by a processor, the method described above is implemented.

[0036] Beneficial effect: The present invention can generate a text block result set with balanced length and better preserved semantic integrity, which can be used for subsequent processes of retrieval enhancement generation, and ultimately obtain better retrieval enhancement generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a text segmentation method according to an exemplary embodiment;

[0038] Figure 2 is a flow chart of a method for performing semantic level boundary segmentation on a text to be processed according to an exemplary embodiment;

[0039] Figure 3 is a flow chart of a method for segmenting a segmented natural segment into semantic level boundaries according to an exemplary embodiment;

[0040] Figure 4is a flow chart of a method for truncating the length of a segmented sentence according to an exemplary embodiment;

[0041] Figure 5 is a flow chart of a text segmentation method according to another exemplary embodiment;

[0042] Figure 6 is a block diagram of a text segmentation system according to an exemplary embodiment;

[0043] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0045] In the related art, the practical methods of text segmentation are as follows: (1) Fixed-size block segmentation method. This is the simplest and most common text segmentation method. Dividing the text into fixed-size blocks is suitable for datasets with similar content formats and sizes, such as news articles or blog posts. However, this method may ignore the contextual relationship of the text content, resulting in poor results in some application scenarios. (2) Random block segmentation method. If the dataset contains multiple document types, random-size blocks can be used for segmentation. This method may capture a wider range of semantic contexts and topics, but it may also cause the text to be interrupted and produce meaningless blocks. (3) Sliding window method. The sliding window method is a commonly used text segmentation technique that makes the new block overlap with the content of the previous block. This method can better capture the contextual information around each block and improve the semantic relevance of the entire system. However, it should be noted that the sliding window method may increase the storage space requirements and lead to redundant information. (4) Context-aware segmentation method. This method segments the text into small blocks based on semantic tags such as punctuation, Markdown layout tags, and HTML tags. This approach can maintain the context integrity of each block, but does not take into account the hierarchical structure of these semantic separation symbols, and there is room for improvement.

[0046] Due to the different sources, formats, and processing methods of the original document data, the natural segmentation of the original document is complex and can be divided into the following situations:

[0047] 1. The natural segmentation is reasonable, the paragraph length is moderate, and it falls within the predefined text block length threshold, which is suitable for subsequent algorithm processing.

[0048] 2. Some natural paragraphs are too fragmented, resulting in a large number of continuous paragraphs whose length is less than the lower limit of the length threshold. If this situation is not handled, the number of algorithm calls will increase; and because the natural paragraphs are too small, they are more likely to lack necessary context information. In this case, the fragmented paragraphs need to be merged.

[0049] 3. Some documents are long but have no paragraphs, or even only one paragraph. This is usually due to the author's writing style or the failure to properly retain the natural segmentation information during the acquisition and processing of the original information, resulting in only one natural paragraph in the entire article. If the length of such a paragraph is greater than or equal to the upper limit of the length threshold range, it needs to be segmented.

[0050] In addition, a general text has a semantic hierarchical structure such as paragraphs, sentences, and clauses. If you simply divide the text into blocks according to punctuation or other tags, you will not consider that these semantic separators have a hierarchical structure. When segmenting text, segmenting from the top to the bottom of the semantic level can preserve the semantic integrity of the text block as much as possible.

[0051] In order to solve the above technical problems, the present invention proposes a text segmentation method, system, retrieval enhancement generation device, electronic device, storage medium and computer program product, which can not only generate a set of text blocks with balanced length but also better retain semantic integrity.

[0052] An embodiment of the present invention provides a text segmentation method, such as Figure 1 As shown, the method includes:

[0053] Step 101: Obtain the text to be processed.

[0054] Step 102: Based on a preset length threshold interval of the target text block, add the to-be-processed text whose length is less than the upper limit value of the length threshold interval into the text block set; segment the to-be-processed text whose length is not less than the upper limit value of the length threshold interval to obtain segmented natural paragraphs, and add the segmented natural paragraphs into the paragraph list in sequence.

[0055] Step 103: traverse the paragraph list and add the segmented natural paragraphs whose lengths are within the length threshold range into the text block set.

[0056] Step 104: perform string concatenation and fusion on the segmented natural segments whose length is less than the lower limit of the length threshold interval; perform semantic level boundary segmentation on the segmented natural segments whose length is not less than the upper limit of the length threshold interval.

[0057] In this embodiment, the length of the text block is constrained based on a preset target text block length threshold interval, and the text block segmentation is recursively and iteratively performed from top to bottom according to the hierarchical semantic boundaries, thereby ensuring that the semantic integrity within the text block is maintained as much as possible while maintaining a balanced length.

[0058] The above briefly introduces the text segmentation method in the present invention. The following is a detailed introduction to the text segmentation method in the present invention.

[0059] Another exemplary embodiment of the present invention provides a text segmentation method, such as Figure 2 As shown, the method includes:

[0060] Step 201: Obtain the text to be processed.

[0061] Step 202: Determine whether the length of the text to be processed is less than the upper limit of the length threshold interval.

[0062] Step 203: If the length of the text to be processed is less than the upper limit of the length threshold interval, the text to be processed is added to the text block set.

[0063] Step 204: if the length of the text to be processed is not less than the upper limit of the length threshold interval, the text to be processed is segmented into segmented segments, and the segmented segments are sequentially added to the paragraph list.

[0064] Step 205: traverse the paragraph list to determine whether the length of the segmented natural paragraph in the paragraph list is within a preset target text block length threshold range.

[0065] Step 206: If the length of the segmented natural paragraph is within a preset target text block length threshold interval, the segmented natural paragraph is added to the text block set.

[0066] Step 207: If the length of the segmented natural paragraph is not within the preset target text block length threshold interval, determine whether the length of the segmented natural paragraph is less than the lower limit value of the length threshold interval.

[0067] In this embodiment, the segmented natural paragraphs that are not within the preset target text block length threshold interval can be set as paragraphs to be processed.

[0068] Step 208: If the length of the segmented natural segment is less than the lower limit of the length threshold interval, the segmented natural segment is concatenated and fused.

[0069] In this embodiment, the string concatenation and fusion of segmented natural segments can be performed by selecting the segmented natural segment with a smaller length from the adjacent segmented natural segments before and after, and selecting the segmented natural segment with a smaller length from the adjacent paragraphs as the concatenated paragraph; and concatenating and fusion the selected segmented natural segment with the current segmented natural segment. If the lengths of the segmented natural segments before and after are the same, the previous segmented natural segment is preferably selected. If there is only one segmented natural segment adjacent to the current segmented natural segment, this segmented natural segment is directly selected.

[0070] Step 209: Determine whether the length of the spliced ​​paragraph and the current segmented natural paragraph is less than the upper limit of the length threshold interval.

[0071] Step 210: If the length of the concatenated paragraph and the current segmented natural segment is less than the upper limit of the length threshold interval, the current natural segment is deleted from the paragraph list and added to the corresponding position of the segmented natural segment in the paragraph list. If the length of the concatenated paragraph and the current segmented natural segment is not less than the upper limit of the length threshold interval, the process returns to step 205.

[0072] In this embodiment, the selected paragraph can absorb the current paragraph and perform splicing and fusion while keeping the relative order of the segmented paragraphs unchanged, and then delete the current segmented paragraph element from the paragraph list.

[0073] Step 211: If the length of the segmented natural segment is not less than the lower limit of the length threshold interval, determine whether the length of the segmented natural segment is greater than or equal to the upper limit of the length threshold interval.

[0074] Step 212: If the length of the segmented natural paragraph is greater than or equal to the upper limit of the length threshold interval, the segmented natural paragraph is segmented at the semantic level boundary, and the segmented natural paragraph is deleted from the paragraph list, and the text blocks after the segmentation of the segmented natural paragraph are sequentially added to the corresponding positions of the segmented natural paragraph in the paragraph list. If the length of the segmented natural paragraph is less than the upper limit of the length threshold interval, return to step 205.

[0075] In this embodiment, for texts of appropriate length, direct output can avoid unnecessary processing steps and improve the efficiency of the overall processing flow. For texts exceeding the threshold, the text can be segmented without affecting the meaning of the text through semantic level boundary segmentation, which is conducive to maintaining the integrity and readability of the information. It can also flexibly adapt to texts of different lengths, whether short or long texts, can be properly processed.

[0076] An exemplary embodiment of the present invention provides a method for segmenting a natural segment into semantic level boundaries, such as Figure 3 As shown, the method for segmenting the natural segments into semantic level boundaries includes:

[0077] Step 301: Perform integer division and modulo operation on the length of the segmented natural segment and the threshold median respectively to obtain a first threshold quotient and a first threshold remainder, wherein the threshold median is an integer value obtained by averaging the upper limit value of the length threshold interval and the lower limit value of the length threshold interval;

[0078] Step 302: Determine whether the first threshold remainder is zero.

[0079] Step 303: If the first threshold remainder is zero, the first threshold quotient is used as the target segment number.

[0080] Step 304: If the first threshold remainder is not zero, the value obtained by adding 1 to the first threshold quotient is used as the target segment number.

[0081] Step 305: Based on the target number of segments, the target number of segments is divided by the length of the natural segment to obtain the target segmentation length.

[0082] Step 306: Initialize the list to be spliced, and divide the segmented paragraphs into sentences to obtain segmented sentences, and add the segmented sentences to the list of sentences to be processed in sequence. The list to be spliced ​​is used to store processed sentences for paragraph splicing and reconstruction.

[0083] Step 307: traverse the list of sentences to be processed, and determine whether the length of the segmented sentences in the list of sentences to be processed is within a preset target text block length threshold range.

[0084] Step 308: If the length of the segmented sentence is not within the preset target text block length threshold interval, determine whether the length of the segmented sentence in the list of sentences to be processed is greater than or equal to the upper limit value of the length threshold interval.

[0085] Step 309: If the length of the segmented sentence is greater than or equal to the upper limit of the length threshold interval, the segmented sentence is segmented into clauses to obtain segmented sub-sentences, and the segmented sub-sentences are sequentially added to the list to be spliced.

[0086] In this embodiment, the symbols used for sentence separation include a period, an exclamation mark, a question mark, and an ellipsis. The sentence separation symbol is reserved as the end of the previous sentence.

[0087] Step 310: traverse the list to be spliced ​​to determine whether the length of the segmented clauses in the list to be spliced ​​is within the preset target text block length threshold interval. If the length of the segmented clauses in the list to be spliced ​​is within the preset target text block length threshold interval, return to step 307.

[0088] In this embodiment, whether the length of the segmented clause in the list to be concatenated is within a preset target text block length threshold interval can be described as whether there is a clause to be processed.

[0089] Step 311: If the length of the segmented clause in the list to be concatenated is not within the preset target text block length threshold interval, determine whether the length of the segmented clause is greater than or equal to the upper limit value of the length threshold interval.

[0090] Step 312: If the length of the segmented sentence is greater than or equal to the upper limit of the length threshold interval, the segmented sentence is length-truncated to obtain a segmented text block, and the segmented text block is added to the list to be spliced.

[0091] Step 313: If the length of the segmented sentence is less than the upper limit of the length threshold interval, the segmented sentence is added to the list to be spliced.

[0092] Step 314: if the length of the segmented sentence is less than the upper limit of the length threshold interval, the segmented sentence is added to the list to be spliced, and the list to be spliced ​​is used for paragraph splicing and reconstruction, and the process returns to step 307.

[0093] Step 315: If the length of the segmented sentence is within a preset target text block length threshold range, the character string buffer is initialized.

[0094] Step 316: traverse the list to be spliced ​​to determine whether there is text to be spliced.

[0095] Step 317: If there is text to be spliced, determine whether the sum of the lengths of the text to be spliced ​​and the string buffer is greater than the target segmentation length. If the sum of the lengths of the text to be spliced ​​and the string buffer is not greater than the target segmentation length, execute step 320.

[0096] Step 318: If the sum of the length of the text to be concatenated and the length of the character string buffer is greater than the target segmentation length, the character string in the character string buffer is added to the paragraph list.

[0097] Step 319: Reset the buffer.

[0098] Step 320: Add the text to be concatenated into the string buffer, and continue to execute step 317.

[0099] Step 321: If there is no segmented sentence that needs to be concatenated and fused, the string in the string buffer is added to the paragraph list.

[0100] The length of the text block is constrained by using the upper and lower limits of the threshold, and the median of the threshold is calculated to guide and control the optimal segmentation length, so that the segmented text blocks can maintain semantic integrity as much as possible while ensuring the length is balanced. Even when falling back to truncation by length, attention is paid to maintaining the balance of the truncation results, and the most balanced results are obtained under the condition of limited computing overhead.

[0101] An exemplary embodiment of the present invention provides a method for truncating the length of a segmentation sentence, such as Figure 4 As shown, the method for cutting the length of the segmented sentence includes:

[0102] Step 401: Divide the length of the segmented clause by the median value of the threshold to obtain a second threshold quotient and a second threshold remainder, where the median value of the threshold is an integer value obtained by averaging the upper limit value of the length threshold interval and the lower limit value of the length threshold interval;

[0103] Step 402: Determine whether the second threshold remainder is zero.

[0104] Step 403: If the second threshold remainder is zero, the second threshold quotient is used as the number of segments to be cut.

[0105] Step 404: If the second threshold remainder is not zero, the value obtained by adding 1 to the second threshold quotient is used as the number of segments to be cut.

[0106] Step 405: Based on the number of truncation segments, the number of truncation segments is divided by the length of the segmented sub-sentences to obtain a target truncation length.

[0107] Step 406: Based on the target truncation length, the segmented sentences are truncated to obtain truncation text blocks, and the truncation text blocks are added to the list to be spliced.

[0108] In this embodiment, when the length of the segmented clause is truncated to the last section, all remaining parts are included in the last section.

[0109] In an exemplary embodiment, the ratio of the lower limit to the upper limit of a preferred length threshold interval is: threshold lower limit: threshold upper limit = 2:3, for example, the threshold lower limit is 800, the threshold upper limit is 1200, and the threshold median is 1000; or, the threshold lower limit is 400, the threshold upper limit is 600, and the threshold median is 500.

[0110] Another exemplary embodiment of the present invention provides a text segmentation method, such as Figure 5 The flowchart of the text segmentation method is shown in FIG. The method comprises:

[0111] Step 501: Obtain the text to be processed.

[0112] Step 502: Determine whether the length of the text to be processed is less than the upper limit value of the length threshold interval.

[0113] Step 503: If the length of the text to be processed is less than the upper limit of the length threshold interval, the text to be processed is added to the text block set.

[0114] Step 504: If the length of the text to be processed is not less than the upper limit of the length threshold interval, the text to be processed is segmented into segmented segments to obtain segmented segments and the segmented segments are sequentially added to the paragraph list.

[0115] Step 505: traverse the paragraph list to determine whether there is a paragraph to be processed in the paragraph list.

[0116] Step 506: If there are paragraphs to be processed, the paragraph list is used as an output text block set.

[0117] Step 507: If there is no paragraph to be processed in the paragraph list, determine whether the length of the segmented natural segment is less than the lower limit of the length threshold interval;

[0118] Step 508: If the length of the segmented natural segment is less than the lower limit of the length threshold interval, then the segmented natural segment with the smaller length among the preceding and succeeding adjacent segmented natural segments is selected.

[0119] Step 509: Determine whether the length of the selected segmented natural segment and the current segmented natural segment is less than the upper limit value of the length threshold interval.

[0120] Step 510: On the basis of maintaining the order, the selected segmented paragraph absorbs the current segmented paragraph, performs string concatenation and fusion, and deletes the current segmented paragraph from the paragraph list.

[0121] Step 511: If the length of the segmented natural segment is not less than the lower limit value of the length threshold interval, determine whether the length of the segmented natural segment is greater than or equal to the upper limit value of the length threshold interval.

[0122] Step 512: If the length of the segmented natural paragraph is greater than or equal to the upper limit value of the length threshold interval, the current segmented natural paragraph is deleted from the paragraph list, the segmented natural paragraph is segmented at the semantic level boundary, and the segmented and reorganized segmented natural paragraphs are added to the position of the current segmented natural paragraph in the paragraph list in sequence.

[0123] After the paragraph reorganization process in the above embodiment, the length of all text blocks is less than the upper threshold value; the length of the text blocks is not less than the lower threshold value when possible. The length of the result text block may be less than the lower limit of the length threshold interval because the original text length is very small or for the purpose of making the length of the text blocks in the result set more balanced.

[0124] The present invention performs paragraph reorganization with recursive iteration characteristics under the constraint of a preset target text block length threshold interval, and performs text segmentation from top to bottom along the semantic level boundary, so as to retain the semantic integrity in the text block as much as possible. Because natural paragraphs are the most natural semantic fragments, the present invention segments the text in natural paragraphs as much as possible. When segmentation according to natural paragraph boundaries is not possible, sentence boundaries and short sentence boundaries are recursively used for segmentation. This is to ensure that the semantic integrity in the text block is maintained as much as possible while ensuring that the length is balanced.

[0125] An exemplary embodiment provides a text segmentation system, such as Figure 6 As shown, the text segmentation system is configured to execute the text segmentation method of the above embodiment, including: an acquisition module 601 and a segmentation module 602,

[0126] An acquisition module 601 is configured to acquire a text to be processed;

[0127] The segmentation module 602 is configured to add the to-be-processed text whose length is less than the upper limit of the length threshold interval into the text block set based on the preset length threshold interval of the target text block; segment the to-be-processed text whose length is not less than the upper limit of the length threshold interval to obtain segmented natural segments, and add the segmented natural segments into the paragraph list in sequence; traverse the paragraph list, and add the segmented natural segments whose length is within the length threshold interval into the text block set; perform string concatenation and fusion on the segmented natural segments whose length is less than the lower limit of the length threshold interval; and perform semantic level boundary segmentation on the segmented natural segments whose length is not less than the upper limit of the length threshold interval.

[0128] An exemplary embodiment also provides a retrieval enhancement generation device, which is configured to vectorize the text in the text block set based on the text block segmentation method of the above embodiment. The device is applied to the intelligent retrieval question-answering process to produce a better retrieval enhancement generation effect.

[0129] The above-mentioned goal of balancing paragraph length is to provide the retrieval enhancement generation device with a subsequent text vectorization process. While retaining semantic integrity as much as possible, text length balancing can ensure the effect of text vectorization. It solves the problem that the information diversity of the vectorized input text is too different due to uneven length, which leads to recall problems when the vectors generated by such text blocks are used for subsequent retrieval enhancement generation. Compared with the average text block length, when the text block is too short, the information diversity after vectorization is insufficient and the information is single, which will produce similarity overfitting with specific retrieval input text, and may cause false recall problems. Compared with the average text block length, when the text block is too long, the information diversity after vectorization is excessive, the information is noisy, and it is similar to a large number of inputs, but the similarity is insufficient, which may cause insufficient recall problems.

[0130] Based on the text segmentation results of the above embodiment, the retrieval enhancement generation process can be served by text vectorization. The retrieval enhancement generation process can also be served by constructing an inverted index based on entities and keywords for accurate and rapid recall of information.

[0131] An embodiment of the present invention further proposes an electronic device, comprising a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the text segmentation method of any of the above embodiments.

[0132] Figure 7 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Figure 7 The device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions that can be executed by the processing component 922, such as an application. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above-mentioned test verification environment construction method and testing method.

[0133] The device 900 may also include a power component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0134] An embodiment of the present invention further provides a computer-readable storage medium. When an executable computer program in the storage medium is executed by a processor, the text segmentation method of any of the above embodiments can be implemented.

[0135] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, which can be executed by the processing component 922 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0136] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the text segmentation method of any of the above embodiments when executed by a processor.

[0137] Regarding the device in the above embodiment, the specific manner in which the processor performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0138] In addition to the beneficial effects described in the above embodiments, the advantages of text segmentation in the present invention are: 1. Optimizing resource utilization. Large language models often encounter the problem of excessive resource consumption when processing long texts. Through text segmentation, we can cut long documents into smaller, independent parts, thereby reducing the demand for computing resources. This divide-and-conquer strategy enables the retrieval enhancement generation system to process large-scale text data more efficiently. 2. Improve retrieval efficiency. In the retrieval enhancement generation system, retrieval efficiency directly affects the response speed and accuracy of the model. Smaller text blocks are obtained through text segmentation. Smaller text blocks can speed up the search and matching process because each text block contains more concentrated information, reducing the interference of irrelevant information. This enables the model to find the most relevant content to the user query more quickly. 3. Improve generation quality. Text segmentation also helps to improve the quality of model-generated content. By segmenting long documents into multiple small blocks, the model can focus more on the content of each small block, thereby generating more accurate and relevant responses. This segmented processing method enables the model to better understand the intention of the user's query and give more accurate answers.

[0139] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A text segmentation method, characterized in that: The method comprises: Get the text to be processed; Based on a preset length threshold interval of the target text block, add the to-be-processed text whose length is less than the upper limit of the length threshold interval into the text block set; segment the to-be-processed text whose length is not less than the upper limit of the length threshold interval to obtain segmented natural paragraphs, and add the segmented natural paragraphs into the paragraph list in sequence; Traverse the paragraph list, add the segmented natural paragraphs whose length is within the length threshold interval into the text block set; perform string concatenation and fusion on the segmented natural paragraphs whose length is less than the lower limit value of the length threshold interval; and perform semantic level boundary segmentation on the segmented natural paragraphs whose length is not less than the upper limit value of the length threshold interval.

2. The method according to claim 1, characterized in that The step of concatenating and fusing the segmented natural segments whose length is less than the lower limit of the length threshold interval comprises: According to the paragraph list, select the smaller segmented natural paragraphs in the adjacent paragraphs as the splicing paragraphs; Based on the length of the spliced ​​paragraph, the length of the segmented natural segment and the upper limit value of the length threshold interval, the spliced ​​paragraph and the natural segment are spliced ​​and merged; The above operation is repeated until all the segmented natural segments whose length is less than the lower limit value of the length threshold interval are processed.

3. The method according to claim 2, characterized in that After the segmented natural segments whose lengths are less than the lower limit of the length threshold interval are concatenated and merged, the method includes: The character string obtained by concatenating and fusing the segmented paragraphs is added to the corresponding position of the segmented paragraph in the paragraph list, and the segmented paragraph is deleted from the paragraph list.

4. The method according to claim 1, characterized in that Before segmenting the segmented natural segments whose length is not less than the upper limit value of the length threshold interval into semantic level boundaries, the method includes: Perform integer division and modulo operation on the length of the segmented natural segment and the threshold median respectively to obtain a first threshold quotient and a first threshold remainder, wherein the threshold median is an integer value obtained by averaging the upper limit and the lower limit of the length threshold interval; Obtaining a target number of segments based on the first threshold quotient and the first threshold remainder; A target segmentation length is obtained based on the target number of segments and the length of the natural segment of the segment.

5. The method according to claim 1, characterized in that The segmenting of the natural segments whose length is not less than the upper limit value of the length threshold interval into semantic level boundaries includes: Segmenting the segmented paragraphs to obtain segmented sentences; Segment the segmented sentences whose length is not less than the upper limit value of the length threshold interval to obtain segmented sub-sentences; cut the segmented sub-sentences whose length is not less than the upper limit value of the length threshold interval to obtain cut-off text blocks, and add the cut-off text blocks to the list to be spliced; add the segmented sub-sentences whose length is less than the upper limit value of the length threshold interval to the list to be spliced; Adding the segmented sentences whose length is less than the upper limit of the length threshold interval to a list to be spliced, wherein the list to be spliced ​​is used for paragraph splicing and reconstruction; Initialize the string buffer, and obtain the text to be spliced ​​according to the list to be spliced; Based on the target segmentation length, the length of the string buffer, and the length of the segmentation sentence, the text to be spliced ​​is added to the buffer or the string in the buffer is added to the paragraph list, the buffer is reset, and the text to be spliced ​​is added to the buffer; Repeat the above steps until all the texts to be spliced ​​are processed.

6. The method according to claim 5, characterized in that The step of performing length cut on the segmented clauses whose length is not less than the upper limit value of the length threshold interval to obtain a cut text block, and adding the cut text block to the to-be-joined list includes: Dividing the length of the segmentation clause by a threshold median value to obtain a second threshold quotient and a second threshold remainder, wherein the threshold median value is an integer value obtained by averaging an upper limit value and a lower limit value of the length threshold interval; Obtaining the number of intercepted segments based on the second threshold quotient and the second threshold remainder; Obtaining a target truncation length based on the number of truncation segments and the length of the truncation sub-sentences; Based on the target truncation length, the segmented sentences are truncation-lengthed to obtain truncation text blocks, and the truncation text blocks are added to the to-be-joined list.

7. A text segmentation system, characterized in that: The system is configured to execute the method according to any one of claims 1 to 6, including: an acquisition module and a segmentation module, The acquisition module is configured to acquire the text to be processed; The segmentation module is configured to add the to-be-processed text whose length is less than the upper limit of the length threshold interval into the text block set based on the preset length threshold interval of the target text block; segment the to-be-processed text whose length is not less than the upper limit of the length threshold interval to obtain segmented natural paragraphs, and add the segmented natural paragraphs into the paragraph list in sequence; Traverse the paragraph list, add the segmented natural paragraphs whose length is within the length threshold interval into the text block set; perform string concatenation and fusion on the segmented natural paragraphs whose length is less than the lower limit value of the length threshold interval; and perform semantic level boundary segmentation on the segmented natural paragraphs whose length is not less than the upper limit value of the length threshold interval.

8. A search enhancement generation device, characterized in that: The device is configured to perform text vectorization on the text in the text block set generated based on the text block set according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images