A text chunk semantic coherence detection method for RAG question answering system

Abstract

The present application belongs to the field of artificial intelligence technology machine learning and semantic understanding, relates to semantic analysis and semantic recognition, and is a text blocking semantic coherence detection method for RAG question and answer system, specifically by quantifying the semantic loss of different blocks, assisting in constructing supervised fine tuning (SFT) and reinforcement learning (RL) training sets through rejection sampling, and then improving the text blocking capability of large language models, and optimizing the retrieval effect and answer generation quality of the RAG system. The present application quantifies the semantic consistency within the blocks and the semantic jump degree at the block boundaries, providing an objective basis for evaluating the quality of text blocking, and the effectiveness of the algorithm is verified through experiments, which is of great significance to improving the text blocking capability of large language models, the retrieval effect and answer generation quality of the RAG question and answer system.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence technology machine learning and semantic understanding, and relates to semantic analysis and semantic recognition, in particular to a text blocking semantic coherence detection method for RAG question and answer system. BACKGROUND

[0002] Retrieval Augmented Generation (RAG) is a natural language processing technology that combines information retrieval with text generation, using external knowledge bases to supplement the context of large language models and generate responses. This technology combines the parameterized internal knowledge of large language models and non-parameterized external knowledge, utilizing the information retrieval and generation capabilities of large language models to improve the performance of various NLP tasks. The main goal is to improve the accuracy and context relevance of text generation while reducing the hallucination effect.

[0003] A significant challenge currently faced by RAG is that the context window size of large language models is limited, restricting the ability to fully understand long documents. The commonly used solution is to break down large amounts of text into smaller, more focused segments, so that LLMs can process each part more accurately, ensuring a thorough understanding of each segment. This blocking method allows LLMs to analyze unstructured data in detail, helping them build a more comprehensive and coherent understanding of the entire document. However, how to effectively block the text directly affects the effectiveness of retrieval and the accuracy of answer generation. Reasonable text blocking can maintain semantic coherence within each block, while clearly distinguishing the semantics between different blocks, thereby helping to improve the accuracy of retrieval and the quality of generated content.

[0004] In the traditional chunking method, the fixed-size chunking method divides the text into uniform chunks, but often ignores the underlying text structure. The recursive chunking method iteratively divides the text using delimiters such as punctuation marks, making it more naturally adapt to content changes. The chunking method based on document structure elements can automatically divide according to elements such as titles and tables, thus more effectively capturing the semantic and structural information of the document. In semantic-based segmentation, Lukasik et al. proposed three Transformer-based segmentation model architectures, of which Cross-segment BERT only uses the local context around each candidate breakpoint, BERT+Bi-LSTM is used to capture the sequence representation of sentences of indefinite length, and Hierarchical BERT encodes the entire document, through model distillation, the parameter quantity becomes less, but the performance is comparable to large models. Zhang et al. proposed SeqModel, which uses BERT to encode multiple sentences simultaneously, models the dependency between longer contexts, then calculates sentence vectors, and finally predicts whether to perform text segmentation after each sentence. The model also uses an adaptive sliding window method to further speed up inference without reducing accuracy.

[0005] Although the current text chunking method improves the semantic coherence of chunking to some extent, there are still some problems in practical application. One of the significant shortcomings is the lack of an effective evaluation index to measure the semantic continuity after text chunking, and its influence on the retrieval effect and answer generation quality of the RAG question answering system. In order to solve this problem, the invention proposes a text chunking semantic coherence detection method for RAG question answering system.

[0006] In the prior art, Chinese patent application No. 202411646238.1 discloses a RAG performance improvement method based on process optimization. The method carefully disassembles and optimizes the whole process of the RAG system from data preparation to generation, especially optimizes the problems such as inaccurate retrieval and answer illusion commonly encountered in practical applications. By using the two-level splitting method of parent document and sub-document, the context retention and recall granularity are considered, and the relevance of retrieval is improved. By introducing the context-driven large language model rewriting mechanism, the semantic reconstruction of the ambiguous or omitted reference problem is carried out, and the definiteness of user query and the retrieval quality are enhanced. Although the method has achieved certain effect, the query rewriting and multi-round questioning mechanism seriously depend on the performance of large language model and the engineering skill of prompt word, and the effect difference between different models may be significant, and the adaptability to resource limited scene is poor. At the same time, the splitting strategy used belongs to shallow structure block, and the deep understanding of semantic coherence is not embodied. The present application proposes an effective evaluation index to measure the semantic coherence of text block in the aspect of deep semantic analysis of text block content, and further provides a reference for optimizing and improving the retrieval effect and answer generation quality of RAG question and answer system, which is essentially different from the prior art. SUMMARY

[0007] The purpose of the present application is to provide a text block semantic coherence detection method for RAG question and answer system to alleviate the limitations of traditional text block method in RAG question and answer system. The method can deeply evaluate the semantic coherence of text block by calculating the internal consistency loss and boundary coherence loss of text block, and use it as a reward mechanism to fine-tune large language model through rejection sampling, effectively improve the text block ability of large language model, and further optimize the retrieval effect and answer generation quality of RAG question and answer system.

[0008] The technical scheme of the present application is:

[0009] A text block semantic coherence detection method for RAG question and answer system, comprising the following steps:

[0010] Step 1: calculate the semantic similarity of adjacent sentence pairs in the text block, and get the internal consistency loss by combining the sentence importance weight, to evaluate the semantic coherence within the block;

[0011] Step 2: calculate the semantic similarity at the boundary of adjacent blocks, introduce the cross block semantic comparison mechanism to get the boundary coherence loss, to evaluate the semantic jump degree between blocks;

[0012] Step 3: select seven large language models to block the text, use RAGAS evaluation framework to evaluate the retrieval effect and answer generation quality corresponding to the block result of each model, and based on the evaluation result, verify the effectiveness of the two semantic coherence detection methods;

[0013] Step 4: Based on the internal consistency loss and boundary coherence loss, design a reward mechanism to reject sampling of the text segmentation results, assist in constructing supervised fine-tuning (SFT) and reinforcement learning (RL) training datasets for the large language model, and fine-tune the large language model to improve the text segmentation effect.

[0014] The more specific steps are as follows:

[0015] Step 1: Define the set of sentences in the text block as S = {s1, s2, ..., s...} n}, where n is the total number of sentences, and si is the i-th sentence; calculate the semantic similarity C(s) between adjacent sentences si and si+1. i ,s i+1 The importance weights of sentences in the text, W = {w1, w2, ..., w ... n-1 The semantic similarity of adjacent sentence pairs is summed to obtain the semantic loss, and the difference between the sum and 1 is used as the semantic loss. A weighted sum is then applied to obtain the total internal consistency loss of the text block, internal_loss, calculated as follows:

[0016]

[0017] When sentences within the same text block have highly consistent semantics, the cosine similarity approaches 1, and the loss tends to be 0. Conversely, if there are semantic jumps within the text block, the semantic loss will increase significantly, indicating that there may be unreasonable segmentation within the block.

[0018] Step 2: Define the set of text blocks B = {B1, B2, ..., B} n}, each text block It contains several sentences, where mi represents the number of sentences in block Bi; for each block boundary, the semantic similarity between the last sentence of the block and the first sentence of the next block is calculated. A cross-block semantic comparison mechanism is introduced, which calculates the semantic similarity between the last sentence of the current block and the first sentence of the next k-1 blocks, and takes the average of the results. The difference between the two values ​​is used as the semantic loss. These are summed to obtain the total text block boundary coherence loss, `boundary_loss`, calculated using the following formula:

[0019]

[0020] Step 3: First, use the seven large language models glm3, glm4, gpt4, kimi, qwen2.5-14b, qwen2.5-32b, qwen2.5-72b to perform 4 times of partitioning on each document, and use the partitioning result of each time as the knowledge base to evaluate the retrieval effect and answer generation quality of the RAG system. The evaluation framework uses the RAGAS framework, and the evaluation result of each model is the average of the four evaluation results. Then, the average value of the internal consistency loss and the boundary coherence loss of each evaluation result is calculated to verify the effectiveness of the two loss calculation methods.

[0021] Step 4: First, set appropriate internal consistency loss and boundary coherence loss thresholds, and perform rejection sampling on the four partitioning results of each model. The data that meets the threshold range is used as the positive sample in the SFT training data set and the reinforcement learning training data set, otherwise it is used as the negative sample in the reinforcement learning training data set. Then, using the constructed training data set, fine-tune qwen2.5-14b on 8 Nvidia A100 GPU servers, set the training parameters lr=1e-4, epochs=3, warmup_ratio=0.1, lr_scheduler_type=cosine, per_device_train_size=2, gradient_accumulation_steps=2; Through model fine-tuning, the text partitioning capability is improved, and the retrieval effect and answer generation quality of the RAG question and answer system are optimized.

[0022] Further, after step 4, the fine-tuned qwen2.5-14b-sft partitioning result is tested.

[0023] The application calculates the internal consistency loss and boundary coherence loss of the text block to evaluate the semantic coherence of the text block, and then assists in rejection sampling to construct the training set to fine-tune the large language model, which provides strong technical support for improving the text partitioning capability of the large language model and optimizing the retrieval effect and answer generation quality of the RAG question and answer system. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly and specifically illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings in the embodiments. The following drawings are only the drawings in the embodiments of the application.

[0025] Figure 1 is the internal consistency algorithm flowchart in the embodiments of the application;

[0026] Figure 2 is the boundary coherence loss algorithm flowchart in the embodiments of the application;

[0027] Figure 3 is a text chunking flowchart using large language models in embodiments of the present application;

[0028] Figure 4 is a text chunking result semantic loss and evaluation effect change chart of different large language models in embodiments of the present application;

[0029] Figure 5 is a flowchart of constructing a training data set using rejection sampling in embodiments of the present application;

[0030] Figure 6 is a model fine-tuning training loss curve chart in embodiments of the present application;

[0031] Figure 7 is a comparison chart of evaluation results using the fine-tuned model in embodiments of the present application. DETAILED DESCRIPTION

[0032] The present application will be further described below by means of the accompanying drawings and embodiments.

[0033] Reference Figures 1-2 A text chunking semantic coherence detection method for RAG question answering systems, comprising the following steps:

[0034] Step 1: Calculate the semantic similarity of adjacent sentence pairs within a text block, and obtain the internal consistency loss by combining the sentence importance weight, to evaluate the semantic coherence within the block;

[0035] Step 2: Calculate the semantic similarity at the boundary of adjacent blocks, introduce a cross-block semantic comparison mechanism to obtain the boundary coherence loss, to evaluate the degree of semantic jump between blocks;

[0036] Step 3: Select seven large language models to chunk the text, use the RAGAS evaluation framework to evaluate the retrieval effect and answer generation quality of the chunking results of each model, and based on the evaluation results, verify the effectiveness of the two semantic coherence detection methods.

[0037] Step 4: Based on the internal consistency loss and boundary coherence loss, design a reward mechanism to reject sampling the text chunking results, assist in constructing the supervised fine-tuning (SFT) and reinforcement learning (RL) training data set of the large language model, and fine-tune the large language model to improve the text chunking effect.

[0038] Further, step 5, test the fine-tuned qwen2.5-14b-sft chunking results.

[0039] More specifically, the steps are:

[0040] In the embodiment, step 1 includes: (1): for each text block, the sentence set S = {s1, s2, ..., s...} n}, using the Embedding model stella-v2-zh to obtain the s of each sentence i The semantic embedding vectors are used to obtain the vector set E = {e1, e2, ..., e}. n};(2): Calculate each pair of adjacent sentences s in the sentence set i and s i+1 The semantic similarity C(s) between i ,s i+1 ) = e i @e i+1 (3): The importance weight W = {w1, w2, ..., w1} in the text is calculated using the TF-IDF method. n-1};(4): Take the difference between the semantic similarity of adjacent sentence pairs and 1, and multiply it by their importance weights. Then, perform a weighted summation of the internal consistency loss for each sentence pair. The specific formula is as follows:

[0041]

[0042] The algorithm implementation is shown in the table below:

[0043]

[0044] In this embodiment, step 2 includes: (1): taking a set B = {B1, B2, ..., B} of multiple text blocks after the text has been segmented. n}, each text block (2): Use the Embedding model stella-v2-zh to get the first sentence s of each text block. i,1 and the last sentence Semantic embedding vector (3): Set the block segmentation range k, and calculate the semantic similarity between the last sentence of the segment and the first sentences of the next k-1 adjacent segments. Take the average value The difference between the value of 1 and 1 is the boundary coherence loss of the current sentence, and these values ​​are summed. The specific formula is as follows:

[0045]

[0046] The algorithm implementation is shown in the table below:

[0047]

[0048]

[0049] In the embodiment, step 3 includes: (1) using seven large language models, glm3, glm4, gpt4, kimi, qwen2.5-14b, qwen2.5-32b, and qwen2.5-72b, to divide each document into blocks four times, as follows: Figure 3 As shown; (2) Using the results of each segmentation as the knowledge base, the retrieval performance and answer generation quality of the RAG system are evaluated using the RAGAS framework; (3) Based on the evaluation results in (2), the changes in the average answer generation performance and average retrieval performance of each text segmentation method with the corresponding semantic loss are visualized and analyzed, such as Figure 4 As shown.

[0050] In this embodiment, step 4 includes: (1) setting an internal consistency loss threshold and a boundary coherence loss threshold, rejecting sampling of the text segmentation results in step (3) (1), and constructing a supervised fine-tuning (SFT) and reinforcement learning (RL) training dataset, as follows. Figure 5 As shown; (2) Set the training parameters lr=1e-4, epochs=3, warmup_ratio=0.1, lr_scheduler_type=cosine, per_device_train_size=2, gradient_accumulation_steps=2, and perform supervised fine-tuning (SFT) training on qwen2.5-14b on an 8-GPU Nvidia A100 server to obtain the fine-tuned model qwen2.5-14b_sft-all. The training loss change curve is shown in the figure. Figure 6 As shown.

[0051] In this embodiment, step 5 includes (1) using qwen2.5-14b_sft-all to perform text segmentation, using the results as a knowledge base, and using the RAGAS framework to evaluate the retrieval performance and answer generation quality of the RAG system; (2) calculating and comparing the semantic loss of the qwen2.5-14b_sft-all text segmentation results, such as... Figure 7 As shown.

[0052] This invention, by calculating the internal consistency loss and boundary coherence loss of text blocks, can deeply evaluate the semantic coherence of text blocks and use them as a reward mechanism for rejection sampling to fine-tune the large language model, effectively improving the text block segmentation capability of the large language model, and thus optimizing the retrieval effect and answer generation quality of the RAG question answering system.

[0053] The above is only the preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification of the above embodiment without departing from the technical solution content of the present application and according to the technical essence of the present application still belongs to the scope of the technical solution of the present application.

Claims

1. A method for detecting semantic coherence of text segments in a RAG question-answering system, characterized in that, Includes the following steps: Step 1: Calculate the semantic similarity of adjacent sentence pairs within a text block, and combine the sentence importance weights to obtain the internal consistency loss, so as to evaluate the semantic coherence within the block; Step 2: Calculate the semantic similarity at the boundaries of adjacent blocks, introduce a cross-block semantic comparison mechanism to obtain the boundary coherence loss, and evaluate the degree of semantic jump between blocks; Step 3: Select seven large language models to segment the text, use the RAGAS evaluation framework to evaluate the retrieval performance and answer generation quality of each model's segmentation results, and verify the effectiveness of the two semantic coherence detection methods based on the evaluation results. Step 4: Based on the internal consistency loss and boundary coherence loss, design a reward mechanism to reject sampling of the text segmentation results, assist in constructing the supervised fine-tuning SFT and reinforcement learning RL training dataset of the large language model, and fine-tune the large language model to improve the text segmentation effect. Step 2 specifically involves: defining a set of text blocks B = {B1, B2, ..., B...} n }, each text block Contains several sentences, m i Indicates block B i The number of sentences in the text; For each block boundary, calculate the semantic similarity between the last sentence of the block and the first sentence of the next block. A cross-block semantic comparison mechanism is introduced, which calculates the semantic similarity between the last sentence of the current block and the first sentence of the next k-1 blocks, and takes the average of the results. The difference between the two values ​​is used as the semantic loss. These are summed to obtain the total text block boundary coherence loss, `boundary_loss`, calculated using the following formula:

2. The text segmentation semantic coherence detection method for RAG question answering systems as described in claim 1, characterized in that: Step 1 specifically involves: defining a set of sentences S = {s1, s2, ..., s...} in the text block. n }, where n is the total number of sentences, s i This is the i-th sentence; Calculate adjacent sentences s i and s i+1 The semantic similarity C(s) between i ,s i+1 The importance weights of sentences in the text, W = {w1, w2, ..., w ... n-1 The semantic similarity of adjacent sentence pairs is summed to obtain the semantic loss, and the difference between the sum and 1 is used as the semantic loss. A weighted sum is then applied to obtain the total internal consistency loss of the text block, internal_loss, calculated as follows: .

3. The text segmentation semantic coherence detection method for RAG question answering systems as described in claim 1, characterized in that: Step 3 specifically involves the following steps: First, each document is divided into four blocks using seven large language models: glm3, glm4, gpt4, kimi, qwen2.5-14b, qwen2.5-32b, and qwen2.5-72b. The results of each block division are used as the knowledge base to evaluate the retrieval performance and answer generation quality of the RAG system. The evaluation framework is the RAGAS framework, and the evaluation result for each model is the average of the four evaluation results. Then, the average of the internal consistency loss and boundary coherence loss for each evaluation result is calculated to verify the effectiveness of the two loss calculation methods.

4. The text segmentation semantic coherence detection method for RAG question answering systems as described in claim 1, characterized in that: Step 4 specifically involves: First, setting appropriate thresholds for internal consistency loss and boundary coherence loss, rejecting sampling of the four block results for each model, using data that meets the threshold range as positive samples in the SFT training dataset and reinforcement learning training dataset, and otherwise using them as negative samples in the reinforcement learning training dataset. Then, using the constructed training dataset, qwen2.5-14b was fine-tuned on an 8-GPU server with the training parameters set as lr=1e-4, epochs=3, warmup_ratio=0.1, lr_scheduler_type=cosine, per_device_train_size=2, gradient_accumulation_steps=2.

5. The text segmentation semantic coherence detection method for RAG question answering systems as described in claim 1, characterized in that: After step 4, the fine-tuned qwen2.5-14b-sft block results are tested.

Images