Text block semantic coherence detection method oriented to RAG question and answer system

Abstract

The invention belongs to the field of artificial intelligence technology machine learning and semantic understanding, and relates to semantic analysis and semantic recognition, in particular to a text block semantic coherence detection method oriented to an RAG question and answer system, which specifically quantifies semantic loss of different blocks, assists in rejecting sampling to construct a supervised fine tuning (SFT) and reinforcement learning (RL) training set, and improves the semantic coherence of the different blocks. Therefore, the text partitioning capability of the large language model is improved, and the retrieval effect and the answer generation quality of the RAG system are optimized. According to the method, the semantic consistency in the blocks and the semantic jump degree at the boundaries of the blocks are quantified, an objective basis is provided for evaluating the quality of the text blocks, the effectiveness of the algorithm is verified through design experiments, and the method has important significance in improving the text blocking capacity of a large language model, the retrieval effect of an RAG question and answer system and the answer generation quality.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, machine learning and semantic understanding, and relates to semantic analysis and semantic recognition, and specifically to a text segmentation semantic coherence detection method for a RAG question-answering system. Background Art

[0002] Retrieval-augmented generation (RAG) is a natural language processing technique that combines information retrieval with text generation, using an external knowledge base to supplement the context of a large language model and generate responses. This technique combines the parameterized internal knowledge of a large language model with non-parametric external knowledge, leveraging the generative capabilities of information retrieval and large language models to improve the performance of various NLP tasks. Its main goal is to improve the accuracy and contextual relevance of text generation while reducing the hallucination effect.

[0003] An obvious challenge currently facing RAG is that the context window size of large language models is limited, which restricts their ability to fully understand long documents. A common solution is to break down large amounts of text into smaller, more focused segments so that LLM can process each part more accurately, thereby ensuring full understanding of each part. This chunking approach enables LLM to conduct detailed analysis of unstructured data, helping them to build a more comprehensive and coherent understanding of the entire document. However, how to effectively chunk text directly affects the effectiveness of retrieval and the accuracy of answer generation. Reasonable text chunking can keep the semantics within each chunk coherent, while maintaining clear semantic boundaries between different chunks, thereby helping to improve the accuracy of retrieval and the quality of generated content.

[0004] Traditional chunking methods, such as fixed-size chunking, segment text into uniform chunks, often ignoring the underlying text structure. Recursive chunking methods use delimiters such as punctuation to iteratively segment text, allowing it to more naturally adapt to content changes. Chunking methods based on document structural elements can automatically segment text based on elements such as titles and tables, thereby more effectively capturing the semantic and structural information of the document. In semantic-based segmentation, Lukasik et al. proposed three Transformer-based segmentation model architectures. Cross-segment BERT utilizes only the local context surrounding each candidate segment, BERT+Bi-LSTM captures representations of variable-length sentence sequences, and Hierarchical BERT encodes the entire document. Through model distillation, this reduces the number of parameters while maintaining performance comparable to larger models. Zhang et al. proposed SeqModel, which leverages BERT to simultaneously encode multiple sentences, modeling dependencies between longer contexts before calculating sentence embeddings. Finally, it predicts whether to segment the text after each sentence. This model also uses an adaptive sliding window approach to further accelerate inference without compromising accuracy.

[0005] Although current text segmentation methods have improved the semantic coherence of the segments to a certain extent, some problems still exist in practical applications. A significant limitation is the lack of an effective evaluation metric to measure the semantic coherence of text segments and its impact on the retrieval performance and answer generation quality of RAG question-answering systems. To address this issue, this paper proposes a method for detecting the semantic coherence of text segments in RAG question-answering systems.

[0006] In the prior art, Chinese patent application No. 202411646238.1 discloses a RAG performance improvement method based on process optimization. This method carefully disassembles and optimizes the entire process of the RAG system from data preparation to generation, especially optimizing the common problems of inaccurate retrieval and answer hallucination in practical applications. By using a two-level segmentation method for parent documents and child documents, taking into account both context retention and recall granularity, the relevance of the retrieval is improved. By introducing a context-driven large language model rewriting mechanism, semantic reconstruction is performed on ambiguous or omitted reference questions, thereby enhancing the clarity of user queries and the quality of retrieval. Although this method has achieved certain results, the query rewriting and multi-round questioning mechanism are heavily dependent on the performance of the large language model and prompt word engineering techniques. The differences in effects between different models may be significant, and the adaptability to resource-constrained scenarios is poor. At the same time, the segmentation strategy used belongs to shallow structural block segmentation, which does not reflect a deep understanding of semantic coherence. The present invention proposes an effective evaluation index to measure the semantic coherence of text blocks at the level of deep semantic analysis of the content of text blocks, thereby providing a reference for optimizing and improving the retrieval effect and answer generation quality of the RAG question-answering system, which is essentially different from the existing technology. Summary of the Invention

[0007] The purpose of this invention is to provide a method for detecting semantic coherence in text chunks for RAG question-answering systems, thereby alleviating the limitations of traditional text chunking methods in RAG question-answering systems. By calculating the internal consistency loss and boundary coherence loss of text chunks, this method can deeply evaluate the semantic coherence of text chunks. This method also uses rejection sampling as a reward mechanism to fine-tune the large language model, effectively improving the large language model's text chunking capabilities and thereby optimizing the retrieval performance and answer generation quality of the RAG question-answering system.

[0008] The technical solution of the present invention is:

[0009] A method for detecting semantic coherence of text blocks for a RAG question answering system includes the following steps:

[0010] Step 1: Calculate the semantic similarity of adjacent sentence pairs within a text block and combine it with the sentence importance weight to obtain the internal consistency loss to evaluate the semantic coherence within the block;

[0011] Step 2: Calculate the semantic similarity at the boundaries of adjacent blocks and introduce a cross-block semantic comparison mechanism to obtain boundary coherence loss to evaluate the degree of semantic jump between blocks;

[0012] Step 3: Seven large language models were selected to segment the text. The RAGAS evaluation framework was used to evaluate the retrieval performance and answer generation quality of each model's segmentation results. Based on the evaluation results, the effectiveness of the two semantic coherence detection methods was verified.

[0013] Step 4: Design a reward mechanism based on the internal consistency loss and boundary coherence loss, perform rejection sampling on the text segmentation results, assist in constructing the supervised fine-tuning (SFT) and reinforcement learning (RL) training datasets of the large language model, and fine-tune the large language model to improve the text segmentation effect.

[0014] More specific steps are:

[0015] Step 1: Set the sentence set S in the text block = {s1,s2,...,s n}, where n is the total number of sentences and si is the i-th sentence; calculate the semantic similarity C(s i ,s i+1 ); use TF-IDF method to calculate the importance weight of the sentence in the text W = {w1,w2,...,w n-1 The difference between the semantic similarity of adjacent sentence pairs and 1 is taken as the semantic loss, and the weighted sum is performed to obtain the total internal consistency loss of the text block internal_loss, which is calculated as follows:

[0016]

[0017] When the semantics of sentences in the same text block are highly consistent, the cosine similarity is close to 1 and the loss tends to 0; conversely, if there is a semantic jump in the text block, the semantic loss will increase significantly, indicating that there may be unreasonable segmentation problems within the block.

[0018] Step 2: Set the text block set B = {B1, B2, ..., B n}, each text block Contains several sentences, mi represents the number of sentences in the block Bi; for each block boundary, calculate the semantic similarity between the last sentence of the block and the first sentence of the next block Introducing a cross-block semantic comparison mechanism, the semantic similarity between the last sentence of the current block and the first sentence of the next k-1 blocks is calculated, and the average value is taken. The difference from 1 is taken as the semantic loss, and the sum is calculated to obtain the total loss boundary_loss of the text block boundary coherence. The calculation formula is:

[0019]

[0020] Step 3: First, use seven large language models, glm3, glm4, gpt4, kimi, qwen2.5-14b, qwen2.5-32b, and qwen2.5-72b, to divide each document into four blocks. Use the block results of each time as the knowledge base to evaluate the retrieval effect and answer generation quality of the RAG system. The evaluation framework uses the RAGAS framework, and the evaluation results of each model are the average of the four evaluation results; then, the average internal consistency loss and boundary coherence loss of each evaluation result are calculated to verify the effectiveness of the two loss calculation methods.

[0021] Step 4: First, set appropriate internal consistency loss and boundary coherence loss thresholds, and perform rejection sampling on the four segmentation results of each model. The data that meets the threshold range is used as positive samples in the SFT training dataset and the reinforcement learning training dataset, otherwise it is used as negative samples in the reinforcement learning training dataset; then, use the constructed training dataset to fine-tune qwen2.5-14b on 8 Nvidia A100 GPU servers, setting the training parameters lr = 1e-4, epochs = 3, warmup_ratio = 0.1, lr_scheduler_type = cosine, per_device_train_size = 2, gradient_accumulation_steps = 2; through model fine-tuning training, its text segmentation ability is improved, thereby optimizing the retrieval effect and answer generation quality of the RAG question answering system.

[0022] Furthermore, after step 4, the fine-tuned qwen2.5-14b-sft block results are tested.

[0023] The present invention evaluates the semantic coherence of text blocks by calculating the internal consistency loss and boundary coherence loss of text blocks, and then assists in reject sampling to construct a training set to fine-tune the large language model, providing strong technical support for improving the text segmentation ability of the large language model and optimizing the retrieval effect and answer generation quality of the RAG question-answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings in the embodiments. The following drawings are only drawings in the embodiments of the present invention.

[0025] Figure 1 is a flow chart of an internal consistency algorithm according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a boundary coherence loss algorithm according to an embodiment of the present invention;

[0027] Figure 3 This is a flow chart of text segmentation using a large language model in an embodiment of the present invention;

[0028] Figure 4 1. This is a graph showing the semantic loss and evaluation effect changes of text segmentation results of different large language models in an embodiment of the present invention;

[0029] Figure 5 This is a flow chart of constructing a training data set by rejection sampling in an embodiment of the present invention;

[0030] Figure 6 This is a graph showing the loss curve for fine-tuning the model in the embodiment of the present invention.

[0031] Figure 7 3 is a comparison chart of the model evaluation results after fine-tuning in the embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below with reference to the accompanying drawings and examples.

[0033] See also Figure 1-2 , a text segmentation semantic coherence detection method for RAG question answering system, comprising the following steps:

[0034] Step 1: Calculate the semantic similarity of adjacent sentence pairs within a text block and combine it with the sentence importance weight to obtain the internal consistency loss to evaluate the semantic coherence within the block;

[0035] Step 2: Calculate the semantic similarity at the boundaries of adjacent blocks and introduce a cross-block semantic comparison mechanism to obtain boundary coherence loss to evaluate the degree of semantic jump between blocks;

[0036] Step 3: Select seven large language models to segment the text. Use the RAGAS evaluation framework to evaluate the retrieval effect and answer generation quality corresponding to the segmentation results of each model. Based on the evaluation results, verify the effectiveness of the two semantic coherence detection methods.

[0037] Step 4: Design a reward mechanism based on the internal consistency loss and boundary coherence loss, perform rejection sampling on the text segmentation results, assist in constructing the supervised fine-tuning (SFT) and reinforcement learning (RL) training datasets of the large language model, and fine-tune the large language model to improve the text segmentation effect.

[0038] Furthermore, in step 5, the fine-tuned qwen2.5-14b-sft block results are tested.

[0039] More specific steps are:

[0040] In an embodiment, step 1 includes: (1): for each sentence set S in a text block, n}, use the Embedding model stella-v2-zh to get each sentence s i The semantic embedding vector of the vector set E={e1,e2,...,e n}; (2): Calculate each pair of adjacent sentences s in the sentence set i and s i+1 The semantic similarity C(s i ,s i+1 )=e i @e i+1 ; (3): Use TF-IDF method to calculate the importance weight of each sentence in the text W = {w1,w2,...,w n-1}; (4): Take the difference between the semantic similarity of adjacent sentence pairs and 1, multiply it by their importance weights, and perform weighted summation on the internal consistency loss of each sentence pair. The specific formula is:

[0041]

[0042] The algorithm implementation is shown in the table:

[0043]

[0044] In the embodiment, step 2 includes: (1): taking a set of multiple text blocks after the text is segmented, B = {B1, B2, ..., B n}, each text block (2): Use the Embedding model stella-v2-zh to get the first sentence of each text block i,1 and the final sentence Semantic Embedding Vector (3): Set the block range k and calculate the semantic similarity between the last sentence of the block and the first sentence of the subsequent k-1 adjacent blocks Take the average The difference from 1 is the boundary coherence loss of the current sentence, and the sum is calculated. The specific formula is:

[0045]

[0046] The algorithm implementation is shown in the table:

[0047]

[0048]

[0049] In the embodiment, step 3 includes: (1) using seven large language models, glm3, glm4, gpt4, kimi, qwen2.5-14b, qwen2.5-32b, and qwen2.5-72b, to divide each document into four blocks. The block division process is as follows: Figure 3 As shown in Figure 2; (2) Use the results of each block as the knowledge base and use the RAGAS framework to evaluate the retrieval effect and answer generation quality of the RAG system; (3) Based on the evaluation results in (2), visualize the changes in the average answer generation effect and average retrieval effect of each text block method and the corresponding semantic loss, as shown in Figure 2. Figure 4 shown.

[0050] In the embodiment, step 4 includes: (1) setting an internal consistency loss threshold and a boundary coherence loss threshold, performing rejection sampling on the text segmentation result of step (1) in step (3), and constructing a supervised fine-tuning (SFT) and reinforcement learning (RL) training dataset. The process is as follows: Figure 5 (2) Set the training parameters lr = 1e-4, epochs = 3, warmup_ratio = 0.1, lr_scheduler_type = cosine, per_device_train_size = 2, gradient_accumulation_steps = 2, and perform supervised fine-tuning (SFT) training on qwen2.5-14b on 8 Nvidia A100 GPU servers to obtain the fine-tuned model qwen2.5-14b_sft-all. The training loss change curve is shown as follows: Figure 6 shown.

[0051] In the embodiment, step 5 includes (1) using qwen2.5-14b_sft-all to perform text segmentation, and using the results as the knowledge base, and using the RAGAS framework to evaluate the retrieval effect and answer generation quality of the RAG system; (2) calculating the semantic loss of the qwen2.5-14b_sft-all text segmentation results and comparing them, such as Figure 7 shown.

[0052] By calculating the internal consistency loss and boundary coherence loss of text blocks, the present invention can deeply evaluate the semantic coherence of text blocks, and use rejection sampling as a reward mechanism to fine-tune the large language model, effectively improving the text block segmentation ability of the large language model, thereby optimizing the retrieval effect and answer generation quality of the RAG question-answering system.

[0053] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A text segmentation semantic coherence detection method for RAG question answering system, characterized by: The following steps are involved: Step 1: Calculate the semantic similarity of adjacent sentence pairs within a text block and combine it with the sentence importance weight to obtain the internal consistency loss to evaluate the semantic coherence within the block; Step 2: Calculate the semantic similarity at the boundaries of adjacent blocks and introduce a cross-block semantic comparison mechanism to obtain boundary coherence loss to evaluate the degree of semantic jump between blocks; Step 3: Seven large language models were selected to segment the text. The RAGAS evaluation framework was used to evaluate the retrieval performance and answer generation quality of each model's segmentation results. Based on the evaluation results, the effectiveness of the two semantic coherence detection methods was verified. Step 4: Design a reward mechanism based on the internal consistency loss and boundary coherence loss, perform rejection sampling on the text segmentation results, assist in constructing the supervised fine-tuning SFT and reinforcement learning RL training datasets of the large language model, and fine-tune the large language model to improve the text segmentation effect.

2. The method for detecting semantic coherence of text blocks in a RAG question-answering system according to claim 1, wherein: The step 1 is specifically as follows: setting a sentence set in the text block , where n is the total number of sentences, s i For the i-th sentence; calculate the adjacent sentences s i and s i+1 The semantic similarity between ; Use TF-IDF method to calculate the importance weight of sentences in text ; Take the difference between the semantic similarity of adjacent sentence pairs and 1 as the semantic loss, and perform weighted summation to obtain the total internal consistency loss of the text block internal_loss, which is calculated as follows: When the semantics of sentences in the same text block are highly consistent, the cosine similarity is close to 1 and the loss tends to 0; conversely, if there is a semantic jump in the text block, the semantic loss will increase significantly, indicating that there may be unreasonable segmentation problems within the block.

3. The method for detecting semantic coherence of text blocks in a RAG question-answering system according to claim 1, wherein: The step 2 is specifically as follows: setting a set of text blocks , each text block Contains several sentences, m i Represents block B i The number of sentences in For each block boundary, calculate the semantic similarity between the last sentence of the block and the first sentence of the next block ; Introduce a cross-block semantic comparison mechanism, calculate the semantic similarity between the last sentence of the current block and the first sentence of the next k-1 blocks, and take the average value The difference from 1 is taken as the semantic loss, and the sum is calculated to obtain the total loss boundary_loss of the text block boundary coherence. The calculation formula is: 。 4. The method for detecting semantic coherence of text blocks in a RAG question-answering system according to claim 1, wherein: The specific steps of step 3 are as follows: first, each document is divided into four blocks using seven large language models, namely glm3, glm4, gpt4, kimi, qwen2.5-14b, qwen2.5-32b, and qwen2.5-72b. The results of each block are used as the knowledge base to evaluate the retrieval effect and answer generation quality of the RAG system. The evaluation framework uses the RAGAS framework, and the evaluation results of each model are the average of the four evaluation results; then, the average internal consistency loss and boundary coherence loss of each evaluation result are calculated to verify the effectiveness of the two loss calculation methods.

5. The method for detecting semantic coherence of text blocks in a RAG question-answering system according to claim 1, wherein: The step 4 is specifically as follows: first, setting appropriate internal consistency loss and boundary coherence loss thresholds, performing rejection sampling on the four block results of each model, and using data that meets the threshold range as positive samples in the SFT training dataset and the reinforcement learning training dataset, otherwise used as negative samples in the reinforcement learning training dataset; Then, using the constructed training dataset, qwen2.5-14b was fine-tuned on 8 Nvidia A100 GPU servers, with training parameters set to lr=1e-4, epochs=3, warmup_ratio=0.1, lr_scheduler_type=cosine, per_device_train_size=2, and gradient_accumulation_steps=2. Through model fine-tuning training, its text segmentation ability was improved, thereby optimizing the retrieval effect and answer generation quality of the RAG question-answering system.

6. The method for detecting semantic coherence of text blocks in a RAG question-answering system according to claim 1, wherein: After step 4, the fine-tuned qwen2.5-14b-sft block results are tested.

Images