Self-adaptive text partitioning method, retrieval enhancement method and system and storage medium

Abstract

The invention belongs to the technical field of retrieval enhancement generation, and particularly relates to a self-adaptive text partitioning method, a retrieval enhancement method, a system and a storage medium. The partitioning method comprises the steps of calculating a segmentation tendency score between a current sentence and a next adjacent sentence based on a semantic boundary score between adjacent sentences and a semantic density of a corresponding candidate window; when the segmentation tendency score exceeds a segmentation threshold value, segmentation is carried out after a segmentation point is set between the current sentence and the next adjacent sentence; after segmentation or when the segmentation tendency score is below the segmentation threshold value, judging whether the next adjacent sentence is the last sentence of the current text information or not; if the next adjacent sentence is the last sentence of the current text information, ending segmentation of the current text information to obtain a plurality of text blocks; otherwise, taking the next adjacent sentence as the current sentence to calculate the segmentation tendency score. According to the method, the text blocks with complete semantics and relatively single semantic hierarchy can be accurately segmented.

Description

Technical Field

[0001] This invention belongs to the field of retrieval enhancement generation technology, and particularly relates to adaptive text segmentation methods, retrieval enhancement methods, systems and storage media. Background Technology

[0002] Retrieval-Augmented Generation (RAG) models have made significant progress in tasks such as open-domain question answering, dialogue systems, and content generation. RAG models retrieve relevant search information based on the current need and then generate response text based on this information.

[0003] To improve the accuracy and efficiency of RAG models in responding to responses, the retrieval information processed by RAG models is typically segmented from text information. Segmenting a text into several semantically concentrated text blocks can improve the processing speed of a single text. Furthermore, considering that a text may contain invalid information, segmenting it and directly filtering out invalid text blocks, while only processing valid text blocks, can effectively improve the accuracy of the response text generated by the RAG model.

[0004] Existing segmentation techniques either employ a fixed-length, one-size-fits-all approach or rely solely on paragraph structure. Segmentation methods relying solely on paragraph structure may result in text blocks containing too many complete information entries and too many semantic layers, leading to the inclusion of much invalid information and reducing the accuracy of subsequent response text generated by the RAG model. The one-size-fits-all approach suffers from the same drawbacks as the paragraph-structure-only method when the segment length is set too long; when the segment length is set too short, the text block may contain no complete information. Furthermore, regardless of the segment length, the beginning and end information of a text block may be incomplete (e.g., a single word is segmented).

[0005] To overcome the shortcomings of the one-size-fits-all approach, although overlapping text between the text block and the previous text block has been added to the beginning of the text block, this still cannot guarantee the semantic integrity of the overlapping text, let alone the semantic integrity of the ending text of each text block. The text blocks obtained by the one-size-fits-all approach prevent the downstream RAG model from processing a single valid text block individually, even if the valid text block is surrounded by invalid text blocks. In order to ensure the semantic coherence and information integrity of the current valid text block, the downstream RAG model must temporarily store the text blocks before and after the valid text block, which significantly increases the overall storage space (compared to a single text message, the segmented text blocks have significantly more overlapping text, and invalid text blocks need to be temporarily stored), reducing the efficiency of the RAG model in generating response text.

[0006] Therefore, how to accurately segment text blocks that are semantically complete and have a relatively simple semantic level has become an urgent problem to be solved in the field of retrieval enhancement generation technology. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide an adaptive text segmentation method that can accurately segment text blocks with complete semantics and relatively simple semantic levels.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The adaptive text segmentation method includes the following steps:

[0010] Based on the semantic boundary scores between adjacent sentences and the semantic density of the corresponding candidate windows, the segmentation tendency score between the current sentence and the next adjacent sentence is calculated. When the segmentation tendency score exceeds the segmentation threshold, a segmentation point is set between the current sentence and the next adjacent sentence before segmentation. After segmentation or when the segmentation tendency score is below the segmentation threshold, it is determined whether the next adjacent sentence is the last sentence of the current text information. If the next adjacent sentence is the last sentence of the current text information, the segmentation of the current text information ends, resulting in several text blocks. If the next adjacent sentence is not the last sentence of the current text information, the next adjacent sentence is used as the current sentence to calculate the segmentation tendency score.

[0011] Preferably, obtaining semantic boundary scores further includes the following sub-steps:

[0012] Step 1: Obtain the semantic vectors of each sentence in the current text information D;

[0013] Step 2, construct adjacent sentences S i and S i+1 The eigenvectors X(S) between i ,S i+1 ):

[0014] X(S i ,S i+1 )=[V i V i+1 E P (S i ),F(S i+1 ),L(S i ),L(S i+1 ),P(S i ,D)];

[0015] Among them, V i and V i+1 They respectively represent the sentence S i and S i+1 The corresponding sentence vector; E P (S i ) represents sentence S i One-hot encoded vector of the type of ending punctuation mark; F(S i+1 ) represents sentence S i+1 The parameter for the first sentence of a paragraph is used to represent sentence S. i+1 Is it the first sentence of a paragraph? L(S) i ) and L(S i+1 ) represent sentences S respectively i and S i+1 The corresponding length feature vector; P(S) i D) represents sentence S i Positional features within the current text information D;

[0016] Step 3: Using a pre-trained binary classification model, based on the feature vector X(S) i ,S i+1 ), calculate adjacent sentences S i and S i+1 The semantic boundary score between them. i ,S i+1 ):

[0017] z=ω T ×X(S i ,S i+1 )+b;

[0018] Where e represents the natural constant; z represents the intermediate quantity; ω represents the weight vector of the pre-trained binary classification model; T represents the transpose; and b represents the bias term of the pre-trained binary classification model.

[0019] Preferably, obtaining the semantic density of the corresponding candidate window also includes the following:

[0020] Step 1': Starting from the previous split point, sequentially traverse to sentence S.i+1 All sentences between them constitute the current sentence S. i and S i+1 The corresponding candidate window W(S) i S i+1 ); Let S be the first sentence to the right of the previous split point. k Then W(S) i S i+1 )={S k S k+1 S i S i+1};

[0021] Step 2', calculate the current candidate window W(S) i S i+1 The semantic density D[W(S)] i S i+1 )]:

[0022]

[0023] Among them, V i+1 Indicates the relationship with sentence S i+1 The corresponding sentence vector; V t Indicates the relationship with sentence S t The corresponding sentence vector; sim(V t V i+1 ) represents the sentence vector V t With sentence vector V i+1 Semantic similarity between them.

[0024] Preferably, calculating the segmentation tendency score between adjacent sentences also includes the following:

[0025] Step 1″: Calculate the semantic similarity between adjacent sentence vectors based on cosine similarity;

[0026] Step 2″ uses a decision model to calculate the segmentation tendency score between adjacent sentences based on the semantic similarity and corresponding semantic density between adjacent sentence vectors:

[0027] Split(S i S i+1 ) = M split ([sim(V i V i+1 ), D[W(S i S i+1 Score(S)], Score(S) i S i+1 )];θ split );

[0028] Among them, sim(Vi , V i+1 ) represents the semantic similarity between adjacent sentences S i and S i+1 ; Split(S i , S i+1 ) represents the segmentation tendency score between adjacent sentences S i and S i+1 ; M split represents the decision model function; θ split represents the learnable parameter of the decision model function; M split ([sim(V i , V i+1 ), D[W(S i , S i+1 ), Score(S i , S i+1 )]; θ split ) represents using the decision model function M split with learnable parameter θ split to calculate the segmentation tendency score between adjacent sentences S i , V i+1 ), D[W(S i , S i+1 ), Score(S i , S i+1 )] based on the input [sim(V i and S i+1 ).

[0029] The present invention also provides a retrieval enhancement method, including the following: after the RAG system recalls m1 pieces of text information according to the requirement, use the adaptive text chunking method as described above to chunk the m1 pieces of text information into m2 text chunks; then, after the RAG system recalls m3 effective text chunks from the m2 text chunks according to the requirement, output a corresponding response based on the effective text chunks; where m3 < m2, and both m1 and m2 are positive integers.

[0030] The present invention also provides a retrieval enhancement method, including the following: use the adaptive text chunking method as described above to chunk all text information into text chunks; then, after the RAG system recalls several text chunks according to the requirement, output a corresponding response based on the text chunks. <00This invention also provides a text segmentation system, comprising: a semantic boundary scoring module, a semantic density module, a segmentation tendency scoring module, and a segmentation module; the current text information is sent to the semantic boundary scoring module, which, upon receiving the latest segmentation point information from the segmentation module, begins to determine the position of the current sentence in the current text information; if the current sentence is not the last sentence of the current text information, it calculates the semantic boundary score between the current sentence and the next adjacent sentence, and then sends the semantic boundary score and the latest segmentation point information to the semantic density module; if the current sentence is the last sentence of the current text information, the action ends; the semantic density module calculates the semantic density of the current candidate window and sends it, along with the semantic boundary score between the current sentence and the next adjacent sentence, to the segmentation tendency scoring module; the segmentation tendency scoring module calculates the segmentation tendency score and sends it to the segmentation module; the segmentation module determines whether to segment the current sentence and the next adjacent sentence based on a segmentation threshold, and sends the segmentation point information to the semantic boundary scoring module; each module is programmed or configured to execute the steps of the adaptive text segmentation method described above.

[0032] The present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured to perform the adaptive text segmentation method described above.

[0033] The present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured to perform a retrieval enhancement method as described above.

[0034] The beneficial effects of this invention are as follows:

[0035] (1) The adaptive text segmentation method of the present invention can segment any text information into text blocks that are semantically complete, have a relatively simple semantic level, and conform to the characteristics of the text information, which greatly improves the output efficiency and accuracy of the downstream RAG model.

[0036] (2) The adaptive text segmentation method of the present invention does not employ a fixed-length, one-size-fits-all approach, nor does it rely solely on paragraph structure for segmentation, and it does not require adding overlapping text between the segmented block and the previous text block at the beginning of the segmentation block. Instead, it uses sentences with relatively complete semantics and a relatively simple semantic hierarchy as the smallest segmentation unit. Based on the semantic boundary score between adjacent sentences and the semantic density of the corresponding candidate window, it calculates the segmentation tendency score between adjacent sentences, and then determines whether segmentation should be performed between adjacent sentences based on the segmentation tendency score. Therefore, the text blocks obtained by the segmentation method of the present invention do not have overlapping text, do not increase the storage space of the text block relative to the text information, reduce the storage resource occupation of the text block, and do not increase the computational overhead in the downstream RAG model processing process due to overlapping text.

[0037] (3) This invention ensures that the header and tail information of the segmented text blocks are complete; it also ensures that sentences at the same semantic level are located in the same text block as much as possible. This allows the downstream RAG model to process a text block independently without temporarily storing the adjacent text blocks before and after it, further reducing the storage resource occupation of the text blocks.

[0038] (4) In the adaptive text segmentation method of the present invention, in step 2':

[0039] ①Because in calculating the candidate window W(S) i-1 ,S i When calculating the semantic density of a sentence, the sentence vector V has already been calculated. i The average semantic similarity between the vectors and other sentence vectors in the current candidate window, and finally, the vectors that are not in the adjacent sentence S. i-1 and S i Setting a dividing point between adjacent sentences S indicates that they are adjacent sentences. i With candidate window W(S) i-1 ,S i There is a high degree of semantic consistency among the other sentences in the text;

[0040] ② Calculate the current candidate window W(S) i ,S i+1 The semantic density of sentence S is used to determine the semantic density of sentence S. i+1 Is it because of the adjacent sentence S? i Significant semantic differences exist between adjacent sentences, necessitating the establishment of segmentation points. However, considering only the semantic differences between adjacent sentences may lead to misjudgment (for example, sometimes all sentences in a text closely revolve around a single point, but one sentence uses irony or metaphor, resulting in low semantic similarity between that sentence and its neighbors; therefore, segmenting text based on this criterion would lead to serious problems such as semantic incoherence and information loss due to a lack of key context). Therefore, we construct candidate windows to introduce sentences with semantic similarity to adjacent sentences S. i Several other sentences that have a high degree of semantic consistency are further calculated to find the relationship between these other sentences and sentence S. i+1 The semantic consistency between sentences is then averaged to determine the meaning of "sentence S". i+1 Is it related to the adjacent sentence S? i The semantic density of “excessive semantic differences” will have higher accuracy and robustness;

[0041] ③ Calculate the candidate window W(S) i ,S i+1 When calculating the semantic density of a sentence vector V, we do not need to calculate the semantic density of the sentence vector V. i+1The semantic similarity between any two sentence vectors other than those can also guarantee high accuracy and robustness, significantly reducing the amount of computation.

[0042] (5) The segmentation method of this invention determines whether there are segmentation points between adjacent sentences based on the information of the entire text and the concentrated information of local parts, rather than relying solely on the remaining text information after segmentation. Therefore, the segmentation method of this invention has extremely high versatility for text information with any writing structure, and the determination of segmentation points is more flexible and scientific. This invention can dynamically adjust the granularity of segmentation, making the text block size more reasonable, effectively avoiding the forced separation of closely related semantic content (such as a complete argument or step description), and the semantic content of the segmented text blocks is naturally separated, with stronger cohesion and more complete information.

[0043] (6) The retrieval enhancement method of the present invention greatly improves the efficiency and accuracy of the RAG system in generating response text. Attached Figure Description

[0044] Figure 1 This is a flowchart of the adaptive text segmentation method of the present invention. Detailed Implementation

[0045] To make the technical solution of the present invention clearer and more explicit, the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the present invention without creative effort all fall within the protection scope of the present invention.

[0046] Example 1

[0047] like Figure 1 The flowchart shown is a process flow of the adaptive text segmentation method of the present invention, which includes the following:

[0048] Based on the semantic boundary scores between adjacent sentences and the semantic density of the corresponding candidate windows, the segmentation tendency score between the current sentence and the next adjacent sentence is calculated. When the segmentation tendency score exceeds the segmentation threshold, a segmentation point is set between the current sentence and the next adjacent sentence before segmentation. After segmentation or when the segmentation tendency score is below the segmentation threshold, it is determined whether the next adjacent sentence is the last sentence of the current text information. If the next adjacent sentence is the last sentence of the current text information, the segmentation of the current text information ends, resulting in several text blocks. If the next adjacent sentence is not the last sentence of the current text information, the next adjacent sentence is used as the current sentence to calculate the segmentation tendency score.

[0049] Obtaining semantic boundary scores also includes the following sub-steps:

[0050] Step 1: Obtain the semantic vectors of each sentence in the current text information D.

[0051] A text message D containing n sentences is represented as: D={S1,...,S...} i ,S i+1 ,...,S n}; where S i Let S represent the i-th sentence in text information D, where 1 ≤ i ≤ n. Use a vector model to represent sentence S. i Encode into the corresponding sentence vector V i In this embodiment, the vector model used is the BGE-Large-zh-v1.5 model.

[0052] Optionally, before determining whether there are segmentation points between adjacent sentences in a text message, the non-textual information is first uniformly converted into text form.

[0053] Step 2, construct adjacent sentences S i and S i+1 The eigenvectors X(S) between i ,S i+1 ):

[0054] X(S i ,S i+1 )=[V i V i+1 E P (S i ),F(S i+1 ),L(S i ),L(S i+1 ),P(S i ,D)];

[0055] Among them, V i and V i+1 They respectively represent the sentence S i and S i+1 The corresponding sentence vector, in this embodiment, has a dimension of d; the current sentence is sentence S. i E P (S i ) represents sentence S i The one-hot encoded vector for the ending punctuation mark type, in this embodiment the one-hot encoded vector has a dimension of n, where n represents the number of punctuation marks; F(S i+1 ) represents sentence S i+1 The parameter of the first sentence of the paragraph is a one-dimensional scalar used to represent sentence S. i+1Whether a sentence is the first sentence of a paragraph is determined in this embodiment using binary encoding to obtain a one-dimensional scalar. A scalar value of 1 indicates the first sentence of the paragraph, while a scalar value of 0 indicates it is not the first sentence. This can be determined by sentence S. i+1 Determine if a newline character or indentation exists before the sentence S. i+1 Is it the first sentence of a paragraph? L(S) i ) and L(S i+1 ) represent sentences S respectively i and S i+1 The corresponding length feature vector is a 3-dimensional vector. Based on the character length of the sentence, it can be divided into short sentences, medium sentences, and long sentences, all represented using one-hot encoding. Sentences with fewer than 30 characters are short sentences, those with more than 30 characters but less than 100 characters are medium sentences, and those with more than 100 characters are long sentences; P(S i D) represents sentence S i The positional feature in the current text information D is a one-dimensional scalar normalized between 0 and 1.

[0056] The above vectors and scalars are concatenated to form an eigenvector X(S) with a total dimension of (2d+n+3). i ,S i+1 ).

[0057] In this embodiment, the sentence vector dimension d = 1024 (using the BGE-Large-zh-v1.5 model); the number of punctuation types n = 6 (including period, question mark, exclamation mark, semicolon, colon, and ellipsis).

[0058] Step 3: Using a pre-trained binary classification model, based on the feature vector X(S) i ,S i+1 ), calculate adjacent sentences S i and S i+1 The semantic boundary score between them. i ,S i+1 ):

[0059] z=ω T ×X(S i ,S i+1 )+b;

[0060] Where e represents the natural constant; z represents the intermediate quantity; ω represents the weight vector of the pre-trained binary classification model; T represents the transpose; and b represents the bias term of the pre-trained binary classification model.

[0061] Obtaining the semantic density of the corresponding candidate window also includes the following:

[0062] Step 1': Starting from the previous split point, sequentially traverse to sentence S. i+1All sentences between them constitute the current sentence S. i and S i+1 The corresponding candidate window W(S) i ,S i+1 ); Let S be the first sentence to the right of the previous split point. k Then W(S) i ,S i+1 )={S k ,S k+1 ,...,S i ,S i+1}

[0063] If the current text information D has not been segmented so far, then W(S) i ,S i+1 )={S1,S2,...,S i ,S i+1}

[0064] Step 2', calculate the current candidate window W(S) i ,S i+1 The semantic density D[W(S)] i ,S i+1 )]:

[0065]

[0066] Among them, V i+1 Indicates the relationship with sentence S i+1 The corresponding sentence vector; V t Indicates the relationship with sentence S t The corresponding sentence vector; sim(V t V i+1 ) represents the sentence vector V t With sentence vector V i+1 Semantic similarity between them.

[0067] Semantic density D[W(S)] i ,S i+1 By calculating the sentence vector V i+1 The semantic similarity between the sentence vectors and other sentence vectors in the current candidate window is used to measure the semantic consistency of all sentences in the current window.

[0068] D[W(S i ,S i+1 [)] is the adjacent sentence S i and S i+1 The corresponding semantic density.

[0069] The calculation of segmentation tendency scores between adjacent sentences also includes the following:

[0070] Step 1″: Calculate the semantic similarity between adjacent sentence vectors based on cosine similarity;

[0071] Step 2″ uses a decision model to calculate the segmentation tendency score between adjacent sentences based on the semantic similarity and corresponding semantic density between adjacent sentence vectors:

[0072] Split(S i ,S i+1 ) = M split ([sim(V i V i+1 ),D[W(S i ,S i+1 )],Score(S i ,S i+1 )];θ split );

[0073] Among them, Split(S) i ,S i+1 ) indicates the sentence S adjacent to it. i and S i+1 The tendency to split between them; M split Represents the decision model function; θ split The learnable parameters representing the decision model function are trained on a specific dataset prepared in advance by technical personnel; M split ([sim(V i V i+1 ),D[W(S i ,S i+1 )],Score(S i ,S i+1 )];θ split ) indicates the use of a learnable parameter θ split Decision model function M split Based on the input [sim(V i V i+1 ),D[W(S i ,S i+1 )],Score(S i ,S i+1 )] Calculate adjacent sentences S i and S i+1 The tendency to divide between them is scored.

[0074] In step 2″, the decision model is a small feedforward neural network (MLP). The MLP automatically learns how to extract latent information from the input features and performs non-linear combinations to predict whether adjacent sentences should be segmented, thereby adapting to and feeding back the current text information characteristics and segmentation preferences.

[0075] The adaptive text segmentation method of this invention can segment any text information into text blocks that are semantically complete, have a relatively simple semantic hierarchy, and conform to the characteristics of the text information, thereby greatly improving the output efficiency and accuracy of downstream RAG models.

[0076] The adaptive text segmentation method of this invention does not employ a fixed-length, one-size-fits-all approach, nor does it rely solely on paragraph structure for segmentation, and it avoids adding overlapping text between the segmented block and the previous block at the beginning. Instead, it uses sentences with relatively complete semantics and a simple semantic hierarchy as the smallest segmentation unit. Based on the semantic boundary scores between adjacent sentences and the semantic density of the corresponding candidate windows, it calculates the segmentation tendency scores between adjacent sentences, and then determines whether to segment between adjacent sentences based on the segmentation tendency scores. Therefore, the segmentation method of this invention produces text blocks without overlapping text, does not increase the storage space of text blocks relative to text information, reduces the storage resource occupation of text blocks, and does not increase the computational overhead in the downstream RAG model processing due to overlapping text.

[0077] This invention ensures that the header and footer information of the segmented text blocks are complete; it also strives to ensure that sentences at the same semantic level (i.e., several sentences describing the same thing, with the same semantic level) are located in the same text block. This allows the downstream RAG model to process a text block independently without temporarily storing adjacent text blocks, further reducing the storage resource consumption of text blocks.

[0078] In step 2', because the candidate window W(S) is calculated... i-1 ,S i When calculating the semantic density of a sentence, the sentence vector V has already been calculated. i The average semantic similarity between the vectors and other sentence vectors in the current candidate window, and finally, the vectors that are not in the adjacent sentence S. i-1 and S i Setting a dividing point between adjacent sentences S indicates that they are adjacent sentences. i With candidate window W(S) i-1 ,S i There is high semantic consistency among other sentences in the candidate window W(S) i-2 ,S i-1 The analysis for the following is similar and will not be repeated here. Furthermore, we calculate the current candidate window W(S). i ,S i+1 The semantic density of sentence S is used to determine the semantic density of sentence S. i+1 Is it because of the adjacent sentence S? iSignificant semantic differences exist between adjacent sentences, necessitating the establishment of segmentation points. However, considering only the semantic differences between adjacent sentences may lead to misjudgment (for example, sometimes all sentences in a text closely revolve around a single point, but one sentence uses irony or metaphor, resulting in low semantic similarity between that sentence and its neighbors; therefore, segmenting text based on this criterion would lead to serious problems such as semantic incoherence and information loss due to a lack of key context). Therefore, we introduce candidate windows to introduce sentences with similar semantic similarity to adjacent sentences S. i Several other sentences that have a high degree of semantic consistency are further calculated to find the relationship between these other sentences and sentence S. i+1 The semantic consistency between sentences is then averaged to determine the meaning of "sentence S". i+1 Is it related to the adjacent sentence S? i The semantic density of "excessively large semantic differences" will have higher accuracy and robustness. Furthermore, based on the above analysis, therefore, in calculating the candidate window W(S)... i ,S i+1 When calculating the semantic density of a sentence vector V, we do not need to calculate the semantic density of the sentence vector V. i+1 The semantic similarity between any two sentence vectors other than those can also guarantee high accuracy and robustness, which also greatly reduces the amount of computation.

[0079] The segmentation method of this invention is based on the information of the entire text (for example, calculating the feature vector between adjacent sentences requires sentence S). i The position vector P(S) in the entire text information D i D) Sentence S i+1 The parameter vector F(S) of the first sentence of the paragraph i+1 Based on the local concentrated information (in step 2'), the method determines whether there are segmentation points between adjacent sentences, rather than relying solely on the remaining text information after segmentation. Therefore, the segmentation method of this invention has high versatility for text information with any writing structure, and the determination of segmentation points is more flexible and scientific. This invention can dynamically adjust the granularity of segmentation, making the text block size more reasonable, effectively avoiding the forced separation of closely related semantic content (such as a complete argument or step description), and the segmented text blocks have naturally distinct semantic content, stronger cohesion, and more complete information.

[0080] The present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform the adaptive text segmentation method as described above.

[0081] Example 2

[0082] This invention also provides a retrieval enhancement method, comprising the following steps: After the RAG system retrieves m1 text messages according to demand, it uses the adaptive text segmentation method described in Example 1 to divide the m1 text messages into m2 text blocks; the RAG system then retrieves m3 valid text blocks from the m2 text blocks according to demand, and outputs the corresponding response based on the valid text blocks. Wherein, m3 < m2, and m1 and m2 are both positive integers.

[0083] The specific implementation methods of "RAG system recalling m1 text information according to demand" and "RAG system then recalling m3 valid text blocks from m2 text blocks according to demand, and RAG system outputting the corresponding response based on the valid text blocks" can be implemented by those skilled in the art, and will not be elaborated here.

[0084] The present invention provides a search enhancement method that significantly improves the efficiency and accuracy of the RAG system in generating response text.

[0085] Verified by technical personnel, the retrieval enhancement method of this embodiment was evaluated on the SuperCLUE-RAG Chinese native retrieval enhancement generation benchmark dataset. Compared with the prior art, which directly uses the RAG model to obtain text slices based on fixed length for responses, the retrieval recall (Recall@5) increased from 0.65 to 0.71, an improvement of 9.2%; and the answer precision (EM Score) increased from 0.58 to 0.63, an improvement of 8.6%.

[0086] The present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform a retrieval enhancement method as described above.

[0087] Example 3

[0088] The present invention also provides a retrieval enhancement method, comprising the following: using an adaptive text segmentation method as described in Example 1 to segment all text information into text blocks; after the RAG system recalls several text blocks as needed, it outputs the corresponding response based on the text blocks.

[0089] The present invention provides a retrieval enhancement method that significantly improves the accuracy of the response text generated by the RAG system.

[0090] The present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform a retrieval enhancement method as described above.

[0091] Example 4

[0092] The present invention also provides a text segmentation system, comprising: a semantic boundary scoring module, a semantic density module, a segmentation tendency scoring module, and a segmentation module;

[0093] The current text information is sent to the semantic boundary scoring module. After receiving the latest segmentation point information from the segmentation module, the semantic boundary scoring module begins to determine the position of the current sentence in the current text information. If the current sentence is not the last sentence of the current text information, the semantic boundary score between the current sentence and the next adjacent sentence is calculated, and the semantic boundary score and the latest segmentation point information are sent to the semantic density module. If the current sentence is the last sentence of the current text information, the action ends.

[0094] The semantic density module calculates the semantic density of the current candidate window and sends it, along with the semantic boundary score between the current sentence and the next adjacent sentence, into the segmentation tendency score module.

[0095] The segmentation tendency score module calculates the segmentation tendency score and then sends it to the segmentation module;

[0096] The segmentation module determines whether to segment the current sentence and the next adjacent sentence based on the segmentation threshold, and sends the segmentation point information into the semantic boundary scoring module;

[0097] Each module is programmed or configured to perform the steps of the adaptive text segmentation method as described in Example 1.

[0098] The technologies, shapes, and structures not described in detail in this invention are all well-known technologies. It should also be noted that the above are merely preferred embodiments of this invention and are not intended to limit the scope of the invention. The components or steps in the embodiments of this invention can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered equivalent solutions to this application and should all fall within the protection scope of this invention.

Claims

1. An adaptive text segmentation method, characterized in that, Includes the following steps: The segmentation tendency score between the current sentence and the next adjacent sentence is calculated based on the semantic boundary score between adjacent sentences and the semantic density of the corresponding candidate window. When the segmentation tendency score exceeds the segmentation threshold, a segmentation point is set between the current sentence and the next adjacent sentence before segmentation is performed; after segmentation or when the segmentation tendency score is below the segmentation threshold, it is determined whether the next adjacent sentence is the last sentence of the current text information. If the next adjacent sentence is the last sentence of the current text information, then the segmentation of the current text information ends, resulting in several text blocks; if the next adjacent sentence is not the last sentence of the current text information, then the next adjacent sentence is used as the current sentence to calculate the segmentation tendency score.

2. The adaptive text segmentation method according to claim 1, characterized in that, Obtaining semantic boundary scores also includes the following sub-steps: Step 1: Obtain the semantic vectors of each sentence in the current text information D; Step 2, construct adjacent sentences S i and S i+1 The eigenvectors X(S) between i ,S i+1 ): X(S i ,S i+1 )=[V i ,V i+1 ,E P (S i ),F(S i+1 ),L(S i ),L(S i+1 ),P(S i ,D)]; Among them, V i and V i+1 They respectively represent the sentence S i and S i+1 The corresponding sentence vector; E P (S i ) represents sentence S i One-hot encoded vector of the type of ending punctuation mark; F(S i+1 ) represents sentence S i+1 The parameter for the first sentence of a paragraph is used to represent sentence S. i+1 Is it the first sentence of a paragraph? L(S) i ) and L(S i+1 ) represent sentences S respectively i and S i+1 The corresponding length feature vector; P(S) i D) represents sentence S i Positional features within the current text information D; Step 3: Using a pre-trained binary classification model, based on the feature vector X(S) i ,S i+1 ), calculate adjacent sentences S i and S i+1 The semantic boundary score between them. i ,S i+1 ): z=ω T ×X(S i ,S i+1 )+b; Where e represents the natural constant; z represents the intermediate quantity; ω represents the weight vector of the pre-trained binary classification model; T represents the transpose; and b represents the bias term of the pre-trained binary classification model.

3. The adaptive text segmentation method according to claim 2, characterized in that, Obtaining the semantic density of the corresponding candidate window also includes the following: Step 1': Starting from the previous split point, sequentially traverse to sentence S. i+1 All sentences between them constitute the current sentence S. i and S i+1 The corresponding candidate window W(S) i ,S i+1 ); Let S be the first sentence to the right of the previous split point. k Then W(S) i ,S i+1 )={S k ,S k+1 ,...,S i ,S i+1 }; Step 2', calculate the current candidate window W(S) i ,S i+1 The semantic density D[W(S)] i ,S i+1 )]: Among them, V i+1 Indicates the relationship with sentence S i+1 The corresponding sentence vector; V t Indicates the relationship with sentence S t The corresponding sentence vector; sim(V t V i+1 ) represents the sentence vector V t With sentence vector V i+1 Semantic similarity between them.

4. The adaptive text segmentation method according to claim 3, characterized in that, The calculation of segmentation tendency scores between adjacent sentences also includes the following: Step 1″: Calculate the semantic similarity between adjacent sentence vectors based on cosine similarity; Step 2″ uses a decision model to calculate the segmentation tendency score between adjacent sentences based on the semantic similarity and corresponding semantic density between adjacent sentence vectors: Split(S i ,S i+1 )=M split ([sim(V i ,V i+1 ),D[W(S i ,S i+1 )],Score(S i ,S i+1 )];θ split ); Where, sim(V i V i+1 ) indicates adjacent sentences S i and S i+1 Semantic similarity between them; Split(S) i ,S i+1 ) indicates the sentence S adjacent to it. i and S i+1 The tendency to split between them; M split Represents the decision model function; θ split M represents the learnable parameters of the decision model function. split ([sim(V i V i+1 ),D[W(S i ,S i+1 )],Score(S i ,S i+1 )];θ split ) indicates the use of a learnable parameter θ split Decision model function M split Based on the input [sim(V i V i+1 ),D[W(S i ,S i+1 )],Score(S i ,S i+1 )] Calculate adjacent sentences S i and S i+1 The tendency to divide between them is scored.

5. A retrieval enhancement method, characterized in that, The process includes the following: After the RAG system retrieves m1 text messages according to the demand, it uses the adaptive text segmentation method described in any of claims 1-4 to divide the m1 text messages into m2 text blocks; the RAG system then retrieves m3 valid text blocks from the m2 text blocks according to the demand, and outputs the corresponding response based on the valid text blocks; wherein, m3 < m2, and m1 and m2 are both positive integers.

6. A retrieval enhancement method, characterized in that, Includes the following: using the adaptive text segmentation method as described in any of claims 1-4 to segment all text information into text blocks; the RAG system then recalls several text blocks as needed and outputs the corresponding response based on the text blocks.

7. A text segmentation system, characterized in that, include: Semantic boundary scoring module, semantic density module, segmentation tendency scoring module, and segmentation module; The current text information is sent to the semantic boundary scoring module. After receiving the latest segmentation point information from the segmentation module, the semantic boundary scoring module begins to determine the position of the current sentence in the current text information. If the current sentence is not the last sentence of the current text information, the semantic boundary score between the current sentence and the next adjacent sentence is calculated, and the semantic boundary score and the latest segmentation point information are sent to the semantic density module. If the current sentence is the last sentence of the current text information, the action ends. After calculating the semantic density of the current candidate window, the semantic density module sends it, along with the semantic boundary score between the current sentence and the next adjacent sentence, to the segmentation tendency scoring module; the segmentation tendency scoring module calculates the segmentation tendency score and sends it to the segmentation module; the segmentation module determines whether to segment the current sentence and the next adjacent sentence based on the segmentation threshold, and sends the segmentation point information to the semantic boundary scoring module; each module is programmed or configured to perform the steps of the adaptive text segmentation method as described in any of 1-4.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to perform the adaptive text segmentation method as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to perform a retrieval enhancement method as described in claim 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to perform a retrieval enhancement method as described in claim 6.

Images