Text partitioning method for retrieval enhancement generation system

Abstract

The invention relates to the technical field of natural language processing, in particular to a text partitioning method for a retrieval enhancement generation system, and the method comprises the following steps: obtaining all technical texts of a technical text database in an RAG system, and performing feature analysis on semantic units in each technical text to obtain technical text data; and respectively obtaining a topic bidirectional coherence coefficient of each semantic unit, and completing the combination of adjacent semantic units in each technical text by using the topic bidirectional coherence coefficient. The invention aims to solve the problems of context information loss and insufficient applicability in the single technical field in the partitioning process so as to improve the accuracy and practicability of text partitioning processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a text chunking method for retrieval augmented generation system. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) system is an engineering framework that combines large language models with external knowledge retrieval to enhance question and answer capabilities. In the construction of RAG system, text chunking is a crucial technology. By breaking down text data into smaller text blocks, efficient content embedding can be achieved, which can significantly improve the relevance and accuracy of text recall content.

[0003] Current text chunking processing methods mainly include fixed length-based or sliding window-based methods. These methods can efficiently process large-scale text chunking in a short time, with high efficiency. However, they also have some limitations: on the one hand, they are prone to loss of context information, affecting the coherence of text semantics; on the other hand, these methods are generally suitable for general domain text processing, and are difficult to meet the precision requirements of specific domain text processing. SUMMARY

[0004] To solve the above technical problems, the present application provides a text chunking method for retrieval augmented generation system to solve the problems of loss of context information and insufficient applicability in single technical field during chunking.

[0005] An embodiment of the present application provides a text chunking method for retrieval augmented generation system, which comprises the following steps: obtaining technical text of a technical text database in the RAG system; all words appearing in the technical text in the subject library of its technical field are recorded as subject words respectively; using an association rule mining algorithm to obtain the association rules and their confidence levels of each subject word in the technical text, connecting the two subject words with association rules by edges to form a directed graph T1, and taking the reciprocal of the confidence level of the association rule as the weight of the directed edge between the two nodes; segmenting the technical text into multiple semantic units; using the average level of the reciprocal of all shortest path lengths between corresponding nodes in the directed graph T1 between all subject words of adjacent two semantic units as the theme leading coefficient of the previous semantic unit; taking the average of the theme leading coefficients of any two adjacent semantic units as the theme bidirectional coherence coefficient of the next semantic unit; The segmentation threshold of the topic bidirectional coherence coefficient of all semantic units in the technical text is obtained to determine whether to merge adjacent semantic units in the technical text.

[0006] Preferably, the topic library is a word library composed of technical terms used in all technical topics in the technical field of the technical text.

[0007] Preferably, in the association rule mining algorithm, all topic words in each sentence in the technical text are taken as a transaction.

[0008] Preferably, the direction of the directed edge is from the node corresponding to the antecedent topic word in the association rule to the node corresponding to the consequent topic word.

[0009] Preferably, the segmentation method of the semantic units is any one of the text segmentation methods based on sentence-level or chapter-level semantic units.

[0010] Preferably, the starting point and the ending point of the shortest path are respectively determined by the node in the directed graph T1 corresponding to the topic word of the previous semantic unit and the topic word of the next semantic unit in the adjacent two semantic units.

[0011] Preferably, if a certain topic word appears in both adjacent semantic units, the length of the shortest path between the nodes corresponding to the topic word in the directed graph T1 is assigned a preset value.

[0012] Preferably, the segmentation threshold is further obtained by the maximum inter-class variance algorithm.

[0013] Preferably, the method for determining whether to merge adjacent semantic units in the technical text is: The left and right adjacent semantic units of the semantic unit A(i) in the technical text are sequentially traversed; If the topic bidirectional coherence coefficient of the traversed semantic unit is greater than or equal to the segmentation threshold, and the total character length of the traversed semantic unit and the semantic unit A(i) is less than the maximum input limit length of the large model used in the RAG system, then the traversed semantic unit and the semantic unit A(i) are merged to obtain a new semantic unit.

[0014] Preferably, when there is no adjacent semantic unit in the technical text that meets the merging condition, all the semantic units obtained by merging in the technical text are taken as the text segmentation result of the technical text.

[0015] The present application has at least the following beneficial effects: 1. The present application uses the dependency relationship between all theme words in technical text to construct a directed graph T1, compared with the existing method of directly using the co-occurrence matrix of the vocabulary to reflect the relationship between the words, it can measure the technical association, technical comparison, technical complementarity and other direct or indirect dependency relationships between the technical terms appearing in the technical text more quickly and accurately, thereby improving the accuracy of the subsequent evaluation of the possibility of the technical content involved in the first semantic unit in the adjacent two semantic units leading to the technical content involved in the second semantic unit; 2. The present application calculates the theme bidirectional coherence coefficient of the adjacent next semantic unit by analyzing the shortest path of the adjacent semantic unit in the directed graph T1, considering the repetitiveness of the technical terms mentioned in multiple continuous semantic units with the same technical concept, technical problem or technical function in the technical text, and the dependency of the technical content represented by the technical terms mentioned in the next semantic unit on the technical content represented by the technical terms mentioned in the previous semantic unit in function, flow, logic or implementation, which can effectively evaluate whether multiple continuous semantic units in the technical text express the same technical concept, technical problem or technical function; 3. The present application merges each semantic unit block in the technical text based on the theme bidirectional coherence coefficient, which ensures that the subsequent blocking obtained text block can contain a more complete technical concept, technical problem or technical function, while not exceeding the maximum input limit of the large model used in the subsequent RAG system; 4. The present application uses all semantic units obtained by merging based on the theme bidirectional coherence coefficient as the result of text blocking of technical text, compared with the existing fixed length or sliding window based text blocking method, on the one hand, the present application can more effectively divide the text content expressing the same technical concept, technical problem or technical function in single technical field text into the same text block, compared with the existing method, it can better capture the context information in the technical text, ensure the coherence of the text semantics, and better preserve the original semantic structure of the text; on the other hand, the present application is optimized for the characteristics of single technical field text, by using the theme library corresponding to the technical field to mine the dependency features between theme words in technical text, it can better process the text of specific field, compared with the existing method, this method can significantly improve the accuracy and applicability of text blocking, better meet the special needs of single technical field, and thus improve the overall effect of text processing. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0017] Figure 1 A flowchart of a text chunking method for retrieving an enhanced generation system provided by the present application is provided. Figure 2 A specific method flowchart of merging semantic units of each technical text in a technical text database in an RAG system provided by an embodiment of the present application is provided. Figure 3 A method flowchart of mining dependency relationships between all words in a technical text provided by an embodiment of the present application is provided. Figure 4 A method flowchart of analyzing the theme coherence between adjacent semantic units provided by an embodiment of the present application is provided. Figure 5 A schematic diagram of a directed graph T1 without weight provided by an embodiment of the present application is provided. In Figure 5 In the figure, each solid circle represents a node in the directed graph T1, and the theme word corresponding to each node is marked on the node. The arrowed connection between nodes is a directed edge. DETAILED DESCRIPTION

[0018] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purposes, the specific embodiments, structures, features and effects of a text chunking method for retrieving an enhanced generation system according to the present application are described in detail below in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0020] The specific scheme of a text chunking method for retrieving an enhanced generation system provided by the present application is described in detail below in combination with the accompanying drawings.

[0021] A text chunking method for retrieving an enhanced generation system provided by an embodiment of the present application is provided.

[0022] Specifically, a text chunking method for retrieving an enhanced generation system is provided, please refer to Figure 1 The method comprises the following steps: obtaining technical text of a technical text database in the RAG system; all words of the technical text appearing in the subject library of the technical field to which the technical text belongs are recorded as subject words respectively; an association rule mining algorithm is used to obtain the association rule and the confidence of each subject word in the technical text, two subject words with the association rule are connected by edges to form a directed graph T1, and the reciprocal of the confidence of the association rule is taken as the weight of the directed edge between the two nodes; the technical text is divided into a plurality of semantic units; the average level of the reciprocal of the length of all the shortest paths between the corresponding nodes in the directed graph T1 between all the subject words of the adjacent two semantic units is taken as the subject leading coefficient of the previous semantic unit adjacent thereto; and the average of the subject leading coefficients of any two adjacent semantic units is taken as the subject bidirectional coherence coefficient of the subsequent semantic unit adjacent thereto; obtaining a segmentation threshold of the subject bidirectional coherence coefficient of all semantic units in the technical text to determine whether to merge the adjacent semantic units in the technical text.

[0023] In this embodiment, the specific method flow chart for merging the semantic units of each technical text of the technical text database in the RAG system is shown in the attached Figure 2 The steps are as follows: Step 1: Obtain all technical texts of the technical text database in the RAG system.

[0024] This application takes the RAG system for enhancing the retrieval of knowledge in any technical field as an example, and the technical field can be natural language processing, speech processing or machine learning field, etc. This embodiment selects the natural language processing field for analysis, and obtains the subject library K of the technical field to which the RAG system belongs, i.e. the word library composed of technical terms used in all technical subjects of the technical field, which can be constructed by technical personnel in the related technical field.

[0025] And take any one technical text A in the technical text database of the RAG system in this technical field as an example for subsequent analysis, the technical text is the text content extracted from academic papers, journals, blogs or technical manuals, etc.

[0026] Step 2: By analyzing the characteristics of each semantic unit in each technical text, the subject bidirectional coherence coefficient of each semantic unit is obtained respectively.

[0027] In order to ensure that the text blocks obtained by text blocking of technical texts contain a relatively complete technical concept, technical problem or technical function and do not exceed the maximum input limit length of the large model used in the subsequent RAG system, the technical text is divided into a plurality of semantic units, and then the features between each semantic unit obtained by division and the semantic units adjacent thereto are analyzed to determine whether to merge, so that the merged semantic units can meet the two requirements at the same time, thereby realizing text blocking of technical texts in a single technical field.

[0028] S1: Specifically, in the present application, the technical text A is first divided into a plurality of semantic units, and the character length of each semantic unit obtained by division is less than the preset character length. The division method of the semantic units is to use any one of the text division methods based on sentence-level or chapter-level semantic units. After the technical text A is divided, a continuous text block can be obtained.

[0029] In the text division method based on sentence-level semantic units, a sentence is the basic unit of expressing complete semantics and usually contains a subject-predicate-object structure. For example, "he likes reading" is a complete sentence. In the text division method based on chapter-level semantic units, there are two division methods of paragraphs and chapters. A paragraph is a semantic unit expressing a complete theme in the text and usually consists of multiple sentences. A chapter is a larger semantic unit, such as an entire article or a dialogue, used to express complex logical relationships and themes.

[0030] In this embodiment, the paragraph division method in the text division method based on chapter-level semantic units is selected. The paragraph division method in the text division method based on chapter-level semantic units is a known technology, and the specific process is not described again.

[0031] The preset character length is less than the maximum input limit length L of the large model used in the RAG system. In this embodiment, the Embedding model in the large model used in the RAG system is the BGE-M3 model, which can currently process input texts with a maximum length of 8192. The maximum input limit length L in this embodiment is determined to be 8192 characters, and the preset character length is set to 1 / 4 of the maximum input limit length L. In other embodiments, it can also be set to 1 / 8, 1 / 16, etc., which can be set by the implementer according to the actual situation.

[0032] S2: In a technical text, multiple technical terms with technical dependencies (such as functional dependencies, process dependencies), technical comparisons (such as performance comparisons, application scenario comparisons), technical complementarities (such as functional complementarities, application scenario complementarities) and the like often appear in multiple sentences of the technical text at the same time, because there is some logical and thematic connection between these technical terms, which need to be discussed together in the text to fully express a certain technical concept, solve a certain technical problem or implement a certain technical function.

[0033] Therefore, in order to subsequently combine multiple continuous semantic units expressing the same technical concept, technical problem or technical function in the technical text into a complete semantic unit block, and avoid the fragmentation of the same technical concept, technical problem or technical function in the text block of the finally obtained technical text, the present application first mines the dependencies between all words in the technical text, so as to utilize the dependencies for subsequently combining multiple continuous semantic units with the same technical concept, technical problem or technical function into a complete semantic unit block.

[0034] Among them, the method flowchart provided by the present application for mining the dependencies between all words in the technical text is as shown in the accompanying Figure 3 The specific steps are as follows: S2.1: The present application records all the words appearing in the technical text A in the subject library K of the technical field to which the technical text A belongs as subject words respectively, and forms a subject word set B, which is used to represent the set composed of the words corresponding to the technical terms used in all the technical subjects appearing in the technical text A.

[0035] S2.2: Further, the present application utilizes the association rule mining algorithm to obtain the association rules and their confidence degrees of each subject word in the technical text A, connects the two subject words with an edge if there is an association rule between them, forms a directed graph T1, and takes the reciprocal of the confidence degree of the association rule as the weight of the directed edge between the two nodes.

[0036] As a preferred embodiment, in the association rule mining algorithm, all the subject words in each sentence in the technical text A are taken as a transaction.

[0037] In other embodiments, all the subject words between every two sentence ending punctuation marks in the technical text A can also be taken as a transaction. The sentence ending punctuation marks are full stop, question mark, exclamation mark and the like, which are used to indicate the end of a sentence and the tone. In addition, when the technical text A has only a single sentence ending punctuation mark at the beginning or end, all the subject words between the beginning or end and the first or last character are taken as a transaction.

[0038] In the present application, the method for obtaining all association rules of each subject term in the subject term set B and the confidence thereof according to all transactions in the technical text A is: taking the subject term set B and all transactions in the technical text A as the input of the association rule mining algorithm, and taking each subject term in the subject term set B as the item of the item set in the association rule mining algorithm, and outputting all association rules of each subject term in the subject term set B and the confidence corresponding to each association rule.

[0039] It should be understood that the confidence of each association rule is used to evaluate the probability that the subject term corresponding to the former item in the association rule directly leads to the subject term corresponding to the latter item in the technical text A. The greater the confidence is, the greater the probability that the subject term corresponding to the former item in the association rule directly leads to the subject term corresponding to the latter item in the technical text A is.

[0040] Wherein, the association rule mining algorithm can be Apriori algorithm, FP-Tree algorithm, XFP-Tree algorithm, etc. The present embodiment selects the FP-Tree algorithm, wherein the FP-Tree algorithm is a known technology, and the specific process is not described again.

[0041] In the present application, the method for constructing the directed graph T1 is: taking each subject term in the subject term set B as a node in the directed graph T1 respectively; connecting the two nodes corresponding to the two subject terms in the subject term set B which have an association rule between them in the directed graph T1 to construct a directed edge, the direction of the directed edge is from the node corresponding to the former item subject term to the node corresponding to the latter item subject term in the association rule, and the reciprocal of the confidence of the association rule is taken as the weight of the directed edge between the two nodes. Thus, the directed graph T1 formed by the subject terms in the subject term set B is obtained, wherein the construction of the directed graph is a known technology, and the specific process is not described again.

[0042] Wherein, the length of any path in the directed graph is the sum of the weights of all directed edges passed by the path. In the present embodiment, a schematic diagram of an unweighted directed graph T1 is provided, as shown in FIG. 1. Figure 5

[0043] It should be understood that in the directed graph T1, if there is a path between two nodes, there will be a direct or indirect dependency relationship between the subject terms corresponding to the two nodes, i.e. in the technical text A, the technical content represented by the subject term corresponding to the starting point of the path can directly or indirectly lead to the technical content represented by the subject term corresponding to the terminal point of the path through the technical content represented by multiple subject terms.

[0044] ​The longer the path is, the more directed edges the path passes through or the greater the weight of the directed edges the path passes through. (1) When the path passes through more directed edges, i.e., the path connects the start point to the end point through multiple intermediate nodes, i.e., the technical content represented by the start point corresponding topic term leads to the technical content represented by the end point corresponding topic term, the leading cannot be completed by fewer intermediate node corresponding topic terms, but needs to indirectly rely on more other technical content to complete the leading action, indicating that the leading degree between the start point and the end point corresponding topic terms of the path is weak; otherwise, it is strong. (2) When the path passes through directed edges with greater weight, i.e., the confidence of the directed edges connected by the path is smaller, i.e., the probability that the topic term corresponding to the former node of the directed edge directly leads to the topic term corresponding to the latter node of the directed edge in the technical text A is smaller, i.e., the leading degree of the former node to the latter node of the directed edge passed by the path is weak, indicating that the leading degree between the start point and the end point corresponding topic terms of the path is weak; otherwise, it is strong.

[0045] S3: For two adjacent semantic units in the technical text A, if the multiple technical terms mentioned in the former semantic unit repeatedly appear in the latter semantic unit, or the technical content represented by the technical terms mentioned in the latter semantic unit depends on the technical content represented by the technical terms mentioned in the former semantic unit in function, process, logic or implementation, it can be generally considered that the technical content involved in the former semantic unit can lead to the technical content involved in the latter semantic unit.

[0046] Therefore, in order to analyze whether there is repetition, dependence or other characteristics between the technical contents of adjacent semantic units, the present application analyzes the theme coherence between adjacent semantic units, so as to more accurately judge whether the semantic units can be merged subsequently.

[0047] In the present application, the analysis method flow chart of theme coherence between adjacent semantic units is as shown in the accompanying Figure 4 The specific steps are as follows: S3.1: The present application uses the average level of the reciprocals of the lengths of all shortest paths between the corresponding nodes of all theme terms between the two adjacent semantic units in the directed graph T1 as the theme leading coefficient of the adjacent former semantic unit.

[0048] It should be understood that the theme leading coefficient is used to evaluate the possibility of the technical content involved in the adjacent former semantic unit leading to the technical content involved in the adjacent latter semantic unit. The greater the theme leading coefficient is, the greater the possibility of leading to the technical content involved in the adjacent latter semantic unit is, which can better indicate that the two adjacent semantic units are coherent in technical concept, technical problem or technical function.

[0049] The start point and the end point of the shortest path are respectively the node corresponding to the theme word of the previous semantic unit and the node corresponding to the theme word of the next semantic unit in the directed graph T1, and are used to represent the degree of leading of the theme word of the previous semantic unit to the theme word of the next semantic unit in the two adjacent semantic units, that is, the degree of dependence of the technical content represented by the theme word of the next semantic unit on the technical content represented by the theme word of the previous semantic unit in the function, process, logic or implementation. The shorter the length of the shortest path is, the stronger the degree of leading of the theme word from the previous to the next is; otherwise, the weaker the degree of leading of the theme word from the previous to the next is.

[0050] In addition, it should be noted that the length of the shortest path is the sum of the weights of all directed edges passing through the shortest path.

[0051] The shortest path algorithm can adopt Dijkstra algorithm, SPFA (queue-optimized Bellman-Ford) algorithm, Floyd algorithm, etc., and the Dijkstra algorithm is selected in the embodiment, wherein the Dijkstra algorithm is a known technology, and the specific process is not described herein.

[0052] It is worth noting that if a theme word appears in both of the two adjacent semantic units, the length of the shortest path between the nodes corresponding to the theme word in the directed graph T1 is assigned a preset value.

[0053] The preset value is 1 in the embodiment, which is used to represent that the theme word is connected with itself by an edge in the directed graph T1, and the weight of the edge is assigned as 1, which is less than or equal to the length of the shortest path between any two theme words in the directed graph T1.

[0054] It is worth noting that if the semantic unit is the last semantic unit of the technical text A, the value of the theme leading coefficient of the second last semantic unit in the technical text A is taken as the theme leading coefficient of the last semantic unit.

[0055] S3.2: Further, the application takes the mean of the theme leading coefficients of any two adjacent semantic units as the theme bidirectional coherence coefficient of the next semantic unit adjacent thereto.

[0056] Specifically, in the present embodiment, taking the semantic unit A(i-1) and the semantic unit A(i) of the technical text A as an example, the average of the theme leading coefficients between the semantic unit A(i-1) and the semantic unit A(i) of the technical text A is denoted as the theme bidirectional coherence coefficient of the semantic unit A(i), which is used to evaluate the coherence degree of the technical content involved in the semantic unit A(i) and its left and right adjacent two semantic units in the technical concept, technical problem or technical function in the technical text A. Wherein the larger the theme bidirectional coherence coefficient of the semantic unit A(i) is, the more likely the technical content involved in the semantic unit A(i) and its left and right adjacent two semantic units expresses the same technical concept, technical problem or technical function.

[0057] It is worth noting that if the semantic unit is the first semantic unit of the technical text A, the value of the theme leading coefficient of the semantic unit is taken as the theme bidirectional coherence coefficient of the semantic unit.

[0058] Step three: using the theme bidirectional coherence coefficient, the merging of adjacent semantic units in each technical text is completed.

[0059] If a certain technical concept, technical problem or technical function in the technical text needs multiple continuous semantic units to be completely expressed, the starting semantic unit and the ending semantic unit in these semantic units and their adjacent two semantic units will usually have a smaller value in the coherence degree of the technical content involved in the technical concept, technical problem or technical function.

[0060] Therefore, the present application obtains the segmentation threshold of the theme bidirectional coherence coefficient of all semantic units in the technical text to determine whether to merge the adjacent semantic units in the technical text.

[0061] Specifically, in the present embodiment, the theme bidirectional coherence coefficients of all semantic units in the technical text A are taken as the input of the maximum inter-class variance algorithm, and the output segmentation threshold R2 is used to evaluate whether a certain semantic unit in the technical text A can segment the technical content involved in multiple semantic units at its adjacent positions, wherein the maximum inter-class variance algorithm is a known technology and the specific process is not described again.

[0062] In other embodiments, the experience method, 3-sigma principle, clustering algorithm and other methods can be used to perform binary classification on the theme bidirectional coherence coefficients of all semantic units in the technical text A to determine the threshold, which is used to determine whether the adjacent semantic units need to be merged.

[0063] Since the text in the RAG system is divided into blocks, the length of the characters in the text block obtained usually cannot exceed the maximum input limit of the large model used in the subsequent RAG system.

[0064] Therefore, as a preferred embodiment, the embodiment takes the i-th semantic unit A(i) of the technical text A as an example, and the method for determining whether to merge adjacent semantic units in the technical text is as follows: in the technical text A, the semantic units on the left and right sides of the semantic unit A(i) are sequentially traversed, if the subject bidirectional coherence coefficient of the traversed semantic unit is greater than or equal to the segmentation threshold R2, and the total character length of the traversed semantic unit and the semantic unit A(i) is less than the maximum input limit length L, then the traversed semantic unit and the semantic unit A(i) are merged to obtain a new semantic unit, which is used to represent a text block formed by a plurality of continuous semantic units having the same technical concept, technical problem or technical function in the technical text A.

[0065] As another embodiment, the method for determining whether to merge adjacent semantic units in the technical text can also be as follows: if there is a semantic unit in the technical text A whose subject bidirectional coherence coefficient is greater than or equal to the segmentation threshold R2, and the total character length of the semantic unit and its left adjacent semantic unit is less than the maximum input limit length, then the semantic unit and its left adjacent semantic unit are merged, and the subject bidirectional coherence coefficient of the merged semantic unit is recalculated until there is no semantic unit in the technical text that meets the merging condition, and the merging operation is stopped.

[0066] Finally, after there is no adjacent semantic unit in the technical text A that meets the merging condition, all the semantic units obtained by merging in the technical text A are taken as the text block result of the technical text.

[0067] The above technical features constitute the best embodiment of the present application, which has strong adaptability and optimal implementation effect. Non-essential technical features can be added or reduced according to actual needs to meet the needs of different situations.

Claims

1. A text segmentation method for a retrieval enhancement generation system, characterized in that, The method includes the following steps: Retrieve technical text from the technical text database in the RAG system; All words appearing in the technical text within the topic library of its respective technical field are denoted as topic words. The association rule mining algorithm is used to obtain the association rule and its confidence level for each topic word in the technical text. Two topic words with association rules are connected by an edge to form a directed graph T1. The reciprocal of the confidence level of the association rule is used as the weight of the directed edge between the two nodes. The technical text is segmented into multiple semantic units; the average of the reciprocals of the shortest path lengths between corresponding nodes in the directed graph T1 between all topic words of two adjacent semantic units is used as the topic elicitation coefficient of the preceding semantic unit; the average of the topic elicitation coefficients of any two adjacent semantic units is used as the topic bidirectional coherence coefficient of the following semantic unit. The segmentation threshold of the topic bidirectional coherence coefficient of all semantic units in the technical text is obtained to determine whether to merge adjacent semantic units in the technical text.

2. The text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, The subject library is a terminology database consisting of technical terms used in all technical topics within the technical field of the technical text.

3. The text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, In association rule mining algorithms, all topic words in each sentence of a technical text are treated as a transaction.

4. The text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, The direction of the directed edge is from the node corresponding to the preceding topic term in the association rule to the node corresponding to the following topic term.

5. The text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, The semantic unit segmentation method can be any one of the text segmentation methods based on sentence-level or document-level semantic units.

6. The text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, The starting point and ending point of the shortest path are the nodes in the directed graph T1 corresponding to the topic words of the preceding and following semantic units in two adjacent semantic units, respectively.

7. The text segmentation method for a retrieval enhancement generation system as described in claim 6, characterized in that, If a certain topic word appears in both adjacent semantic units, then the length of the shortest path between the corresponding nodes of that topic word in the directed graph T1 is assigned a preset value.

8. The text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, The segmentation threshold is further obtained using the maximum inter-class variance algorithm.

9. A text segmentation method for a retrieval enhancement generation system as described in claim 5, characterized in that, The method for determining whether to merge adjacent semantic units in the technical text is as follows: The semantic units adjacent to the left and right sides of semantic unit A(i) in the technical text are traversed sequentially. If the topic bidirectional coherence coefficient of the traversed semantic unit is greater than or equal to the segmentation threshold, and the total character length of the traversed semantic unit and semantic unit A(i) is less than the maximum input limit length of the large model used in the RAG system, then the traversed semantic unit and semantic unit A(i) are merged to obtain a new semantic unit.

10. A text segmentation method for a retrieval enhancement generation system as described in claim 1, characterized in that, When there are no adjacent semantic units in the technical text that meet the merging conditions, all semantic units obtained after merging in the technical text are taken as the text block result of the technical text.

Images