RAG enhancement method for coarse and fine granularity fusion segmentation and semantic connection

Abstract

The invention belongs to the technical field of natural language processing, and provides a coarse and fine granularity fusion segmentation and semantic connection RAG enhancement method. Fine-grained segmentation and coarse-grained retrieval are combined to guide RAG to generate a plurality of independent results, then the results are subjected to RRF reordering, single measurement ordering recall is improved into multi-factor fusion recall, and the ranking of the obtained final retrieval results is not a result singly paying attention to retrieval scores any more, so that the accuracy of the retrieval results is improved. Instead, a total ranking result of fine-grained document correlation, user intention, multi-path recall document segment importance and the like is considered, so that accurate and complete information required for answering the questions of the user can be found more accurately. Therefore, the accuracy of RAG retrieval is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and particularly to a RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection. Background Art

[0002] RAG (Retrieval-Augmented Generation) is a natural language processing technology that combines retrieval and generation, aiming to improve the information accuracy and richness in text generation tasks. RAG works through the following two main steps: 1. Retrieval: Before generating text, the model retrieves relevant information or documents from an external knowledge base according to the user input. The knowledge base can be a fixed document set, an encyclopedia, or other large text databases. Common retrieval methods include TF-IDF-based, BM25-based, or semantic retrieval models based on deep neural networks.

[0003] 2. Generation: After retrieving relevant documents, the language model uses this information as assistance to generate answers or continue writing text.

[0004] By combining retrieval and generation, RAG can generate more accurate and information-rich answers when facing complex problems or requiring citation of factual content. This method effectively makes up for the problem that the knowledge of generative language models is restricted by training data. Generative language models tend to "fabricate" information in the absence of explicit external knowledge, also known as the "hallucination" of language models. RAG improves the reliability and accuracy of text generation by introducing retrieved factual information from the knowledge base.

[0005] However, in practical applications, the performance bottleneck of RAG mainly focuses on whether the context information required to answer questions can be accurately and completely recalled during the retrieval process. The current mainstream RAG methods still face the following challenges: Balance between segmentation granularity and retrieval accuracy: When the recall granularity of segmentation is too fine, the recall result cannot obtain all the context information; when the recall granularity of segmentation is too coarse, the embedding model will lose information when compressing a large amount of text information into a vector of a fixed length, ultimately leading to a rapid decline in retrieval accuracy.

[0006] Semantic incoherence of the text after segmentation: Since the segmented text is stored independently and retrieved independently, non-semantic segmentation will result in the lack of context semantics of the paragraph, affecting the performance of RAG.

[0007] Ambiguity of the original question: The original question from the user may be very ambiguous. The key information in the question may be present in the previous rounds of conversations. The original question from the user cannot well express the user's intention, resulting in the retrieval system being unable to correctly retrieve the correct information from the database.

[0008] Limitations of single relevance metric: Affected by the performance of the embedding model, using a single metric (such as Euclidean distance, cosine similarity, etc.) to recall the content in the database is often too one-sided. In actual engineering, both the recall rate and accuracy of this method are relatively low. Summary of the Invention

[0009] To solve the technical problem that the existing RAG technology cannot accurately and completely recall the context information required to answer questions during the retrieval process, the present invention proposes a RAG enhancement method that combines coarse-grained and fine-grained segmentation and semantic connection, which improves the retrieval performance of the RAG technology and enhances the semantic integrity, recall rate, and accuracy of the RAG.

[0010] The specific solution is as follows: A RAG enhancement method that combines coarse-grained and fine-grained segmentation and semantic connection S1. Train the distil BERT semantic relevance classifier: Perform downstream task training on the distilBERT model based on a mixed training set with positive and negative examples to obtain the distil BERT semantic relevance classifier. S2. Segmentation and storage of documents: Coarsely segment the documents to obtain coarsely segmented texts. Use the distil BERT semantic relevance classifier for semantic splicing, merge the coarsely segmented texts with relevant semantics, and store them in the coarse-grained text segment library. Then, perform fine-grained segmentation on the coarsely segmented texts with relevant semantics to obtain fine-grained document segments. Establish a mapping relationship, i.e., a mapping index, between the coarsely segmented texts and the fine-grained document segments, and jointly store the fine-grained document segments and the mapping index in the fine-grained text segment library. S3. Fine-grained retrieval and multi-way recall: Use a large language model to combine with the user's question for intention recognition to obtain an intention recognition result. Use the intention recognition result and the user's question in the current round as retrieval sources to perform multi-way recall retrieval in the fine-grained text segment library to obtain fine-grained results, i.e., the first-stage retrieval results. S4: Coarse-grained recall: Based on the first-stage retrieval results and the mapping index in S2, recall the corresponding coarsely grained results with context information in the first-stage retrieval results as the second-stage retrieval results. S5: Rearrange the second-stage retrieval results described in S4 by RRF: For each document segment in the second-stage retrieval results, use the RRF rearrangement algorithm to de-duplicate and rank them. The RRF rearrangement algorithm calculates the importance of each document segment based on the retrieval frequency of the document segment and calculates the ranking score. The higher the retrieval frequency, the higher the score, and duplicate document segments are removed; The final retrieval result is the second-stage retrieval result containing the ranking score and the de-duplicated results.

[0011] Preferably, the mixed training set is constructed with positive and negative samples in a ratio of 2:1, and the positive and negative samples are obtained based on an open-source dataset.

[0012] Preferably, in step S2, the documents are roughly segmented by setting the text length.

[0013] Preferably, the method for generating the coarse-grained text segment library in step S2 is as follows: S21: Coarse-grained segmentation: Segment the text content according to the set text length to obtain the coarse-grained segmented text; S22: Grouping and merging: Set the sliding window to 2 for semantic splicing, and store the spliced coarse-grained segmented text in the coarse-grained text segment library.

[0014] Preferably, the method for semantic splicing in step S22 is to group the first coarse-grained segmented text and the next coarse-grained segmented text based on context sorting and determine whether there is a semantic relationship in the context. If there is a semantic relationship, continue to judge whether there is a semantic relationship with the next coarse-grained segmented text after splicing until there is no semantic relationship, and store it in the coarse-grained text segment library.

[0015] Preferably, in step S3, the user queries include historical user queries and current user queries. Intent recognition is performed based on the user queries to obtain n intents of the user, and multi-way recall retrieval is performed on the fine-grained text segment library based on the n intents and the current user query.

[0016] Preferably, in step S3, the fine-grained retrieval result is a retrieval result with incomplete context information.

[0017] Preferably, in step S5, the method for calculating the ranking score for each document is as follows: (1) Where d represents the document segment in the second-stage retrieval result, R represents the ranking list of different document segments, r(d) is the position of the document segment d in the ranking list, and k is a set constant used to smooth the document segment score and prevent the weight of high-ranked document segments from being too large.

[0018] Preferably, in step S5, the retrieval frequency is the number of identical document segments in the retrieval results of the second stage, that is, the number of identical coarse-grained results obtained by mapping indexes based on different fine-grained results.

[0019] The beneficial effects of the present invention are as follows: The present invention proposes a RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection. First, the present invention separates the segmentation granularity and the recall granularity, that is, after the present invention uses coarse-grained segmentation, it then performs fine-grained segmentation to obtain a group of fine document segments, and establishes a mapping index, which are respectively stored in the coarse-grained text segment library and the fine-grained text segment library. Second, in order to maintain the segmentation speed, the present invention adopts a non-semantic segmentation method (segmentation according to text length), and through a semantic relevance classifier, a sliding window with a window size of 2 (that is, the relevance of two document segments is judged at a time) is used to perform semantic splicing on the coarse-grained text segments. Since semantic correlation analysis is performed on the document segments, two contextually relevant document segments can be effectively identified, and the integrity of information such as numbered information is effectively maintained. Third, the present invention uses n user intentions and the user's question itself as retrieval sources for retrieval to achieve multi-way recall. In particular, using user intentions as common retrieval sources greatly increases the retrieval accuracy of RAG in the case of vague user questions or consecutive questions. At the same time, the multi-way recall method adopted by the present invention obtains a large number of duplicate texts, and the ranking information contained in these duplicate texts themselves can be effectively utilized by the RRF algorithm, increasing the RAG recall accuracy. Fourth, through the coarse-grained recall results, the present invention enables the RAG process to maintain the context information of the retrieval results without reducing the retrieval accuracy, that is, the coarse-grained retrieval ensures the context integrity of the recalled information and improves the retrieval performance of RAG.

[0020] In summary, the present invention combines fine-grained segmentation with coarse-grained retrieval to guide RAG to generate multiple individual results, and then re-ranks these results using RRF, improving the single metric ranking recall to a multi-factor fusion recall. The ranking of the final retrieval results will no longer be a result that only focuses on the retrieval score, but a combined ranking result that takes into account fine-grained document relevance, user intentions, the importance of multi-way recall document segments, etc., and can more accurately find the precise and complete information required to answer the user's question. Therefore, the present invention improves the accuracy of RAG retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of a RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection.

[0022] Figure 2 It is the overall flowchart of the RAG enhancement method in the embodiment.

[0023] Figure 3It is a training flow chart of the semantic relevance classifier in the embodiment.

[0024] Figure 4 It is a flow chart of the segmentation and warehousing of documents in the embodiment.

[0025] Figure 5 It is a flow chart of the fine-grained and coarse-grained document mapping relationship and retrieval in the embodiment.

[0026] Figure 6 It is a flow chart of intent recognition and multi-way recall retrieval in the embodiment.

[0027] Figure 7 It is a diagram showing the retrieval results of milvus collection documents based on this method in the embodiment. Specific implementation manners

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0029] As Figure 1-2 shown, a RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection S1. Train the distil BERT semantic relevance classifier: perform downstream task training on the distilBERT model based on a mixed training set with positive and negative examples to obtain the distil BERT semantic relevance classifier; S2. Segmentation and warehousing of documents: perform coarse-grained segmentation on the documents to obtain coarse-grained segmented texts, perform semantic splicing using the distil BERT semantic relevance classifier, merge the semantically relevant coarse-grained segmented texts, and store them in the coarse-grained text segment library; perform fine-grained segmentation on the semantically relevant coarse-grained segmented texts to obtain fine-grained document segments, establish the mapping relationship, i.e., the mapping index, between the coarse-grained segmented texts and the fine-grained document segments, and store the fine-grained document segments and the mapping index together in the fine-grained text segment library; S3. Fine-grained retrieval and multi-way recall: use a large language model to perform intent recognition in combination with the user's question to obtain an intent recognition result; use the intent recognition result and the user's question in the current round as retrieval sources to perform multi-way recall retrieval in the fine-grained text segment library to obtain the fine-grained results, i.e., the first-stage retrieval results; S4. Coarse-grained recall: based on the first-stage retrieval results and the mapping index in S2, recall the corresponding coarse-grained results with context information in the first-stage retrieval results as the second-stage retrieval results; S5. Rearrange the second-stage retrieval results described in S4 by RRF: For each document segment in the second-stage retrieval results, use the RRF rearrangement algorithm to perform deduplication and ranking. The RRF rearrangement algorithm calculates the importance of each document segment based on the retrieval frequency of the document segment and calculates the ranking score. The higher the retrieval frequency, the higher the score, and duplicate document segments are removed; the second-stage retrieval results containing the ranking score and after deduplication are used as the final retrieval results.

[0030] Preferably, in order to maintain the speed of document segmentation, the present invention adopts a traditional non-semantic segmentation method. Although the segmentation is performed with a coarse granularity, there is still a situation where the context information is lost in the segmented document segments. Therefore, the present invention trains a semantic relevance classifier based on the distil BERT pre-trained language model to determine whether there is semantic relevance between two sentences. As Figure 3 , shows the training process of the semantic relevance classifier. The mixed training set is composed of positive and negative samples constructed in a ratio of 2:1. The positive and negative samples are obtained based on an open-source dataset.

[0031] Preferably, in step S2, the document is segmented with a coarse granularity by setting the text length.

[0032] Preferably, the method for generating the coarse-grained text segment library in S2 is as follows: S21: Coarse-grained segmentation: The text content is segmented according to the set text length to obtain the coarse-grained segmented text; S22: Grouping and merging: Set the sliding window to 2 for semantic splicing, and store the spliced coarse-grained segmented text into the coarse-grained text segment library.

[0033] The sliding window of 2 is a constant of the present invention and cannot be set manually. This parameter is related to the distil BERT semantic relevance classifier. Each time, only the context relevance of 2 text segments is explored. If they are relevant, they are merged and then judged for relevance with the next document segment; if they are not relevant, the first document segment in the window is stored in the library, and the second document segment is judged for relevance with the next document segment (keeping the window at 2).

[0034] Preferably, the method of semantic splicing in step S22 is as follows: Based on the context sorting, the first coarse-grained segmented text and the next coarse-grained segmented text are set as a group, and it is judged whether there is a semantic relationship in the context. If there is a semantic relationship, after splicing, it is continued to judge whether there is a semantic relationship in the context with the next coarse-grained segmented text until there is no semantic relationship, and then it is stored in the coarse-grained text segment library.

[0035] As Figure 4As shown, the process of splitting and storing documents in the database is presented. RAG uses two different granularities for document splitting and recall. During the splitting process, fine-grained splitting is adopted. When short texts are vectorized into fixed-length vectors, information loss is minimized to improve the accuracy of RAG. During the retrieval process, coarse-grained retrieval is used to recall coarse-grained texts, and the recall results preferably contain complete contexts to improve the performance of RAG. To achieve coarse-grained recall, the present invention stores both the document segments obtained by coarse-grained splitting and those obtained by fine-grained splitting in the database simultaneously, constructs a database, uses the document segments obtained by fine-grained splitting as indexes to retrieve the document segments obtained by coarse-grained splitting, and under the condition of maintaining retrieval accuracy, maximally preserves the context information of the retrieved information. The mapping relationship between the coarse-grained and fine-grained documents in the database and the retrieval process are as Figure 5 shown.

[0036] Using the user's question directly as the database retrieval source often does not perform well. Sometimes, the intention of the user's question is implied in the previous conversation, and it is obviously wrong to use only the question in the current conversation as the database retrieval source. Therefore, in step S3, the user's question includes the historical user's question and the current user's question. Intent recognition is performed based on the user's question to obtain n intentions of the user, and multi-way recall retrieval is performed on the fine-grained text segment library based on the n intentions and the current user's question, as Figure 6 shown.

[0037] Preferably, in step S3, the fine-grained retrieval result is a retrieval result with incomplete context information.

[0038] Preferably, in step S5, the method for calculating the ranking score for each document is: (1) where d represents the document segment in the retrieval result of the second stage, R represents the ranking list of different document segments, r(d) is the position of the document segment d in the ranking list, and k is a constant used to smooth the document segment score to prevent the weights of high-ranked document segments from being too large.

[0039] By performing RRF rearrangement on the multi-way retrieval results, the final result ranking obtained by the present invention will no longer be a result that only focuses on the retrieval score, but a combined ranking result that takes into account the relevance of fine-grained documents, user intentions, the importance of multi-way recalled document segments, etc., and can more accurately find the precise and complete information required to answer the user's question.

[0040] The RRF rearrangement actually relies on the frequency of occurrence of the retrieval results for sorting. In the S3 and S4 stages, namely the fine-grained retrieval and coarse-grained recall stages, there may be multiple fine-grained retrieval results corresponding to the same coarse-grained document segment. For example, the fine-grained document segments a1, a2, and a3 all come from the coarse-grained segmented text A. Then, when calculating the RRF for the document segment A, the frequency is 3. The multi-factor fusion recall is reflected in: 1. Multi-way recall of the user's question itself and the user's intention (subjective cognitive factor). 2. RRF rearrangement based on the retrieval frequency, where the frequency represents the accuracy of the current coarse-grained segmented text (document importance factor).

[0041] Preferably, in step S5, the retrieval frequency is the number of the same document segments in the retrieval results of the second stage, that is, the number of the same coarse-grained results obtained through the mapping index based on different fine-grained results.

[0042] As Figure 7 shown, this example uses the actual milvus collection existing in the existing production environment for detailed description: The fine-grained text segment library and the coarse-grained text segment library of the present invention are two separate libraries. For the convenience of display here, the two libraries are combined and displayed in one view.

[0043] For the milvus collection in the example, the main_idx column represents the unique index of the document segment, and the father_idx represents the fine-grained and coarse-grained mapping. First, the present invention performs coarse-grained segmentation on the document, then uses the distill BERT semantic relevance classifier to merge the context with semantic connections, and then puts the processed coarse-grained segmented text into the coarse-grained text segment library, and sets the father_idx of the coarse-grained text segment to 0. Then, the coarse-grained text segment is further segmented into fine-grained segments and put into the fine-grained text segment library. The main_idx of the coarse-grained text segment corresponding to the fine-grained text segment is used as its father_idx and stored in the library together as the fine-grained and coarse-grained mapping. For example: the father_idx of the fine-grained text segment with main_idx of 30001 is 20117, that is, the coarse-grained text segment corresponding to the 30001 text segment is the text segment with main_idx of 20117. (Note: The fine-grained text segment library and the coarse-grained text segment library are shown as one library, but in fact, the present invention stores the fine-grained and coarse-grained text segments in two independent libraries respectively).

[0044] In the retrieval stage, first, multiple user intents are identified by using the user's historical query information and the user's current query. Then, the user's query and each user intent are respectively used to retrieve in the fine-grained text segment library to find the father_idx corresponding to the first-stage retrieval result. The coarse-grained text segment with the main_idx being the father_idx of the first-stage retrieval result is found in the coarse-grained text segment library as the second-stage retrieval result. After that, the second-stage retrieval results are scored and re-ranked using RRF to obtain the final retrieval result.

[0045] It should be noted that the above-described specific embodiments can enable those skilled in the art to understand the present invention and creation more comprehensively, but do not limit the present invention and creation in any way. Therefore, although this specification has described the present invention and creation in detail with reference to the drawings and embodiments, those skilled in the art should understand that the present invention and creation can still be modified or equivalently replaced. In short, all technical solutions and their improvements that do not depart from the spirit and scope of the present invention and creation should be covered by the protection scope of the patent of the present invention and creation.

Claims

1. A method for enhancing RAG by fusing coarse-grained and fine-grained segmentation with semantic connection, characterized in that S1. Train a distil BERT semantic relevance classifier: perform downstream task training on the distilBERT model based on a mixed training set with positive and negative examples to obtain a distil BERT semantic relevance classifier; S2. Segmentation and storage of documents: Coarsely segment the document to obtain coarsely segmented text, use the distil BERT semantic relevance classifier for semantic splicing, merge the coarsely segmented text with semantic relevance, and store it in the coarsely grained text segment library; further perform fine-grained segmentation on the coarsely segmented text with semantic relevance to obtain fine-grained document segments, establish a mapping relationship, i.e., a mapping index, between the coarsely segmented text and the fine-grained document segments, and store the fine-grained document segments and the mapping index together in the fine-grained text segment library; S3. Fine-grained retrieval and multi-way recall: Use a large language model to combine with the user's question for intent recognition to obtain an intent recognition result; use the intent recognition result and the user's question in the current round as retrieval sources to perform multi-way recall retrieval in the fine-grained text segment library to obtain fine-grained results, i.e., the first-stage retrieval results; S4. Coarse-grained recall: Based on the first-stage retrieval results and the mapping index in S2, recall the corresponding coarsely grained results with context information in the first-stage retrieval results as the second-stage retrieval results; S5. Perform RRF re-ranking on the second-stage retrieval results in S4: Use the RRF re-ranking algorithm to de-duplicate and rank each document segment in the second-stage retrieval results. The RRF re-ranking algorithm calculates the importance of each document segment based on the retrieval frequency of the document segment and calculates the ranking score. The higher the retrieval frequency, the higher the score, and duplicate document segments are removed; use the second-stage retrieval results including the ranking score and after de-duplication as the final retrieval results.

2. An RAG enhancement method based on the fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, The mixed training set is composed of positive and negative samples constructed in a ratio of 2:1, and the positive and negative samples are obtained based on an open-source dataset.

3. A RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, In step S2, the document is coarsely segmented by setting the text length.

4. A RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, The generation method of the coarsely grained text segment library in S2 is as follows: S21: Coarse-grained segmentation: Segment the text content according to the set text length to obtain coarsely segmented text; S22: Grouping and merging: Set the sliding window to 2 to perform semantic splicing on the coarsely segmented text, and store the spliced coarsely segmented text in the coarsely grained text segment library.

5. A RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, The method of semantic splicing in step S22 is to set the first coarsely segmented text and the next coarsely segmented text as a group based on context sorting, and judge whether there is a semantic relationship in the context. If there is a semantic relationship, continue to judge whether there is a semantic relationship with the next coarsely segmented text after splicing until there is no semantic relationship, and store all the processed coarsely segmented text in the coarsely grained text segment library.

6. A RAG enhancement method based on the fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, In step S3, the user questions include historical user questions and current user questions. Based on the user questions, intent recognition is performed to obtain n intents of the user. Based on the n intents and the current user question, multi-channel recall retrieval is performed on the fine-grained text segment library.

7. A RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that In step S3, the fine-grained retrieval result is a retrieval result with incomplete context information.

8. A RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, In step S5, the method for calculating the ranking score for each document is as follows: （1） Where d represents the document segment in the second-stage retrieval result, R represents the ranking list of different document segments, r(d) is the position of the document segment d in the ranking list, and k is a set constant.

9. A RAG enhancement method for fine-grained and coarse-grained fusion segmentation and semantic connection according to claim 1, characterized in that, In step S5, the retrieval frequency is the number of the same document segments in the second-stage retrieval result, that is, the number of the same coarse-grained results obtained by mapping indexes based on different fine-grained results.

Images