Automatic evaluation method and system for improving RAG system stability
By designing automated evaluation methods and systems, and employing multi-dimensional evaluation indicators and a visual interface, the challenges of RAG system evaluation have been solved, the system's stability and development efficiency have been improved, and directions for optimization have been provided.
Patent Information
- Application Number
- CN202511453110.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-11
- Publication Date
- 2026-02-24
AI Technical Summary
Existing technologies are insufficient for comprehensively and accurately evaluating RAG systems. Traditional methods are time-consuming and labor-intensive, lack effective improvement paths, and lack systematic evaluation tools, resulting in unsatisfactory performance of RAG systems in practical applications.
Design an automated evaluation method, including multi-dimensional evaluation metrics, systematic annotation rules, and A/B experiments. By calculating recall, ranking, and generating quality metrics, and combining a visual interface to provide efficient feedback, comprehensively diagnose the RAG system.
It has enabled an efficient RAG evaluation system, saving time and manpower costs, improving system stability and development efficiency, providing directions for improvement, and enhancing the system's practicality and reliability.
Smart Images
Figure CN121560690A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of RAG system technology, specifically relating to an automated evaluation method and system for improving the stability of RAG systems. Background Technology
[0002] With the rapid development of artificial intelligence and big data technologies, a new generation of AI technologies based on large language models is undergoing revolutionary changes. Against this backdrop, how to leverage large language models to restructure and upgrade traditional products and businesses has become a highly anticipated research area.
[0003] Retrieval-Augmented Generation (RAG) systems are among the most popular applications in natural language processing today. These systems can generate responses based on user input to meet user needs. However, due to the high diversity of human natural language expression, traditional methods often face numerous difficulties in handling these tasks. In contrast, large language models, with their powerful emergent capabilities, demonstrate a broad understanding of natural language, cross-domain adaptability, and extremely high flexibility. Therefore, RAG systems based on large language models have become a research hotspot, a typical application of leveraging the capabilities of large models, and have been widely adopted, with various techniques continuously emerging to improve RAG performance.
[0004] However, comprehensively and accurately evaluating RAG systems remains a challenge. Traditional evaluation methods have many limitations, such as the inability to effectively evaluate long text responses, difficulty in identifying error sources in the retrieval and generation modules, and low relevance to human judgment. To overcome these problems, experts or human evaluators can be invited to review the results generated by RAG and assess the quality of the generated answers according to predefined criteria, such as accuracy, coherence, and relevance. While this method can provide high-quality feedback, it may also require significant time and human resources. In practical applications, RAG technology often faces the following challenges:
[0005] Easy to get started but less effective: Although the basic setup of the RAG system is relatively easy, in actual use, its performance often fails to meet expectations and the results are not ideal.
[0006] Lack of clear improvement path: When the RAG system performs poorly, there is often a lack of clear direction and methods for improvement, and researchers are unsure which technology to choose to optimize the system.
[0007] Lack of an effective evaluation system: Although the system may appear to perform well after simple testing, there is a lack of systematic evaluation methods to accurately judge its actual performance and application value. Summary of the Invention
[0008] To address the problems of existing technologies, this invention provides an automated evaluation method and tool aimed at improving the stability of RAG systems. This evaluation framework, specifically designed for RAG systems, offers a powerful and flexible tool capable of comprehensively evaluating and diagnosing RAG systems. The method can clearly demonstrate the complex interactions between the retrieval and generation modules in a RAG system, as well as the impact of different design choices on system performance. This not only helps researchers gain a deeper understanding and improve existing RAG systems but also points the way for future optimization and innovation.
[0009] The purpose of this invention is to provide an efficient automated evaluation method and system for improving the stability of RAG systems, and to provide a powerful and effective evaluation tool, thereby enhancing the practicality and reliability of RAG systems.
[0010] To achieve the above objectives, the present invention provides the following technical solution:
[0011] An automated evaluation method for improving the stability of a RAG system includes the following steps:
[0012] S1. Receive request data, which includes the question, the recalled knowledge base slices, the model generation results, and the standard answer;
[0013] S2. Calculate evaluation metrics, obtain knowledge base slices for the question and recall, perform recall quality assessment, and calculate recall quality metrics. Obtain knowledge base slices for recall and model generation results, perform ranking quality assessment, and calculate ranking quality metrics. Obtain knowledge base slices for the question and recall, model generation results, and standard answers, perform generation quality assessment, and calculate generation quality metrics. Obtain recall quality metrics, ranking quality metrics, and generation quality metrics;
[0014] S3. Based on the recall quality indicators, ranking quality indicators and generated quality indicators, automatically label the cases requesting evaluation, identify the health status of the cases, perform attribution analysis on unhealthy cases, conduct A / B experiments, compare with the standard answer, and analyze the case situation.
[0015] S4. Based on the requested data, evaluation indicators, and case annotations, perform visualization output and output the final evaluation results.
[0016] Preferably, the request data in step S1 includes two types: single-instance request evaluation data and batch request data. Single-instance requests can be initiated using the single-instance request mode, and batch request data can be initiated using the batch request mode.
[0017] Preferably, the request data in step S1 comes from an evaluation database, and the dataset examples in the evaluation database include questions, answers, the fragment containing the answer, and the document containing the fragment.
[0018] Preferably, the evaluation metrics in step S2 include recall quality metrics, ranking quality metrics, and generation quality metrics.
[0019] Preferably, the specific contents of the evaluation quality indicators are as follows:
[0020] The recall quality metric primarily measures the proportion of standard answers contained in the recalled knowledge base slices, reflecting...
[0021] The degree of matching between the content retrieved in the RAG system and the standard answer is measured by metrics such as ContextPrecision, ContextRecall, and ContextEntitiesRecall.
[0022] Ranking quality metrics primarily measure the position of a relevant knowledge base slice among all ranked slices. Metrics include HitRate (hit rate), MRR (mean reciprocal ranking), and NDCG (normalized discount cumulative gain).
[0023] The quality metrics primarily measure the informational consistency between the generated results and the given knowledge base slices retrieved. These metrics include Answer Relevance, Answer Correctness, and Faithfulness.
[0024] Preferably, the cases requesting evaluation in step S3 are automatically labeled, and a set of systematic labeling rules is designed to ensure the standardization and consistency of the labeling process.
[0025] Preferably, the specific content of the systematic annotation rules is as follows:
[0026] The systematic annotation rules are as follows: First, calculate the keyword coverage rate of the standard answer. If there is full coverage, the case status is marked as normal. If the keyword coverage rate is not 1, it is judged as incomplete recall, the case is attributed to recall, and the case status is automatically marked as bad. Next, calculate the `rouge_score_rerank` value after the knowledge base slice re-ranking. If the `rouge_score_rerank` value is less than a predetermined threshold, the case is judged as not containing the expected answer in the re-ranking, attributed to reranking, and the case status is automatically marked as bad. Finally, calculate the `blue_score_finalanswer` value of the case. If the `blue_score_finalanswer` value is less than a predetermined threshold, the case is judged as text redundancy, the case is attributed to chunking, and the case status is automatically marked as bad.
[0027] Preferably, in step S3, the A / B experiment comparison involves importing the A / B experiments into the evaluation system, calculating the keyword scores and overall generation quality scores for both experiments, and analyzing the iteration quality of the RAG system and the impact of different strategies and parameters on the performance of the RAG system model by obtaining the experimental configuration parameters of the A / B experiments.
[0028] Preferably, the experimental configuration parameters include the following:
[0029] Experimental configuration parameters include chunk parameters, rewrite parameters, recall model selection, rerank model selection, and generation model selection.
[0030] Preferably, in step S4, the request data, evaluation metrics, and case annotations will be visualized, while providing information on executing the RAG process. To this end, an HTML-based visual interface is designed, allowing users to view various information related to the evaluation data. This information includes, but is not limited to: user information, request cases, parameter configurations, time information, and evaluation metrics.
[0031] Compared with existing technologies, the automated evaluation method for improving the stability of RAG systems proposed in this invention has the following beneficial technical effects:
[0032] 1) This invention, through the design of a complete evaluation process, including multi-angle evaluation indicators, systematic annotation rules, and A / B experiments, can comprehensively evaluate and diagnose RAG systems, provide high-quality feedback, save a lot of time and manpower costs, and greatly improve the efficiency of developing and optimizing RAG systems.
[0033] 2) This invention designs a visual interactive interface to clearly demonstrate the complex interaction between the retrieval module and the generation module in the RAG system, helping researchers to understand and improve existing RAG systems more deeply and providing direction for the iterative optimization of RAG systems.
[0034] The present invention also provides an automated evaluation system for improving the stability of a RAG system, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the automated evaluation method for improving the stability of the RAG system. Attached Figure Description
[0035] Figure 1 This is a flowchart of the automated evaluation method for the stability of the RAG system in Embodiment 1 of the present invention;
[0036] Figure 2 This is a flowchart of the automated evaluation method for the stability of the RAG system in Embodiment 2 of the present invention. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0039] Terminology Explanation:
[0040] Retrieval-Augmented Generation (RAG): Retrieval enhancement refers to the process of improving the efficiency and accuracy of information retrieval systems by applying advanced information retrieval technologies and artificial intelligence methods. In this field, technological innovation aims to enable retrieval systems to better understand the intent of user queries. Through technologies such as automatic learning, natural language processing, and data mining, it conducts deeper and more precise analysis of text and multimedia content to provide more relevant and personalized search results. The goal of retrieval enhancement is to optimize the information retrieval process, enabling users to obtain the information they need more quickly and accurately. It is applicable to various application scenarios, including search engines, document management systems, and knowledge base retrieval.
[0041] Example 1
[0042] like Figure 1 As shown in this embodiment, an automated evaluation method for improving the stability of a RAG system includes the following steps:
[0043] S1, Request Receiving Case.
[0044] The received request cases are samples of user evaluations of the RAG system. If a user is only concerned with the performance of a specific module in the RAG process, they can make a single request, which allows for faster evaluation and quicker identification of problems in that module. Alternatively, users can make batch request cases, which allows for a comprehensive evaluation after the development of many modules.
[0045] The case study must include the question, the recalled knowledge base slices, the model-generated results, and the standard answer. The question is a sample from the evaluation database; the recalled knowledge base slices are fragments of knowledge base documents divided according to rules, and are a list of fragments related to the question content; the model-generated results are the predicted answers to the actual question returned after the text fragments have been processed by a large language model.
[0046] S2. Calculate the various evaluation indicators based on the received request cases.
[0047] The recall quality metric primarily measures the proportion of standard answers contained in the recalled knowledge base slices, reflecting...
[0048] The degree of matching between the content retrieved in the RAG system and the standard answer is measured by metrics such as ContextPrecision, ContextRecall, and ContextEntitiesRecall.
[0049]
[0050] Where Precision@k represents the precision at the k-th position, i.e., the proportion of correct results among the first k results, v k It is a binary relevance indicator, indicating whether the k-th search result is relevant. Total Relevant Items is the total number of all relevant items in the first K results.
[0051] Context precision is calculated, which measures how many of the K recalled chunks are relevant to the question and the true answer. A precision score is then calculated based on this. It requires the question, the basic facts, and the context.
[0052]
[0053] Here, Number of Ground Truth Sentences Retrieved represents the number of ground truth sentences that can be traced back from the retrieved context, and Total Number of Ground Truth Sentences represents the total number of ground truth sentences.
[0054] Context recall measures the consistency between the retrieved context and the underlying facts. Claims are extracted from the underlying facts, and each claim is then determined to be inferred from the retrieved context. The number of inferred claims and the total number of claims are then calculated. This process requires the question, the underlying facts, and the context.
[0055]
[0056] Here, Number of Ground Truth Entities Retrieved is a list of entities present in the retrieved context. Number of Ground Truth Entities Retrieved is the number of entities in the ground truth. The ratio of the two is used to obtain an entity recall rate.
[0057] Ranking quality metrics primarily measure the position of a relevant knowledge base slice among all ranked slices. Metrics include HitRate (hit rate), MRR (mean reciprocal ranking), and NDCG (normalized discount cumulative gain).
[0058] The quality metrics primarily measure the informational consistency between the generated results and the given knowledge base slices retrieved. These metrics include Answer Relevance, Answer Correctness, and Faithfulness.
[0059] S3. Automatically label the request cases, identify the health status of the cases, and attribute the status to causes.
[0060] In this embodiment, cases requesting evaluation are automatically labeled using a systematic labeling rule to ensure the standardization and consistency of the labeling process. The systematic labeling rule first calculates the keyword coverage rate of the standard answer. If full coverage is achieved, the case status is labeled as normal. If neither the keyword coverage rate nor the context keyword coverage rate is 1, the case is considered incompletely recalled, attributed to recall, and automatically labeled as bad. Next, the `rouge_score_rerank` value after knowledge base slice re-ranking is calculated. If the `rouge_score_rerank` value is less than a predetermined threshold, the case is considered to have not included the expected answer in the re-ranking, attributed to reranking, and automatically labeled as bad. Finally, the `blue_score_finalanswer` value of the case is calculated. If the `blue_score_finalanswer` value is less than a predetermined threshold, the case is considered textually redundant, attributed to a chunk, and automatically labeled as bad.
[0061] Furthermore, a standard answer keyword dictionary is first established, where the dictionary size is the number of samples in the evaluation dataset. Each key in the dictionary represents a question, and the value corresponds to a list of standard answer keywords. Each list contains at least 1-3 keywords. The keywords are selected from the standard answers based on a large language model and in conjunction with a specific prompt.
[0062] Furthermore, if the keyword coverage rate is 1, meaning that all the keywords in the standard answer are included in the generated predicted answer, then the case status will be marked as normal.
[0063] Furthermore, if the keyword coverage rate is not 1, the context keyword coverage rate of the recall needs to be calculated. If the context keyword coverage rate is not 1, it means that the relevant content was not fully recalled or only part of it was recalled. In this case, the case is attributed to recall, and the case status will be automatically marked as bad.
[0064] Furthermore, if the context keyword coverage rate is 1 but the rogue_score_rerank is less than a predetermined threshold, the case will be attributed to rerank, and the case status will be automatically marked as bad.
[0065] Furthermore, if the keyword coverage rate is 1 and `rouge_score_rerank` is greater than a predetermined threshold but `blue_score_finalanswer` is less than a predetermined threshold, then this case is judged as text redundancy, attributed to a chunk, and the case status will be automatically marked as bad. The calculations for `rouge_score_rerank` and `blue_score_finalanswer` are as follows:
[0066]
[0067] in, BP is a conciseness penalty considering the length of the generated text compared to the reference text. lc is the length of the generated text, lr is the length of the reference text, and is the i-gram precision between the generated and reference texts, calculated by dividing the number of matched i-grams in the generated text by the total number of i-grams in the generated text. Pn is the i-gram precision considering the generated and reference texts. N is the maximum order of the i-gram, usually taken as 4, which calculates the geometric mean from 1-gram to 4-gram. Wn is the weight, usually taken as for all n.
[0068] S4. The configuration information, evaluation indicators, intermediate process information and health status of each request case are visualized and output.
[0069] In this embodiment, the request data, evaluation metrics, and case annotations will be visualized, while information on executing the RAG process will also be provided. For this purpose, FastAPI and ag-grid are selected for data visualization, and an HTML-based visualization interface is designed where users can view various information related to the evaluation data. This information includes, but is not limited to, user information, request cases, parameter configurations, time information, and evaluation metrics, making the evaluation results more intuitive and easier to understand.
[0070] Example 2
[0071] like Figure 2 As shown in the figure, the automated evaluation method for improving the stability of RAG system in this embodiment has the following specific steps for an evaluation process:
[0072] Initiate an evaluation request. For a completed RAG system, performance testing is required. The dataset used for the evaluation comes from an evaluation database, which consists of internal and public data. Select the number of evaluation samples based on the actual situation, generally between 50 and 100. Configure the parameters of each module according to the RAG module to be evaluated.
[0073] Furthermore, the evaluation system needs to interact with the RAG system. That is, the evaluation system can obtain information about each module in the RAG process in real time, start the RAG process, and record module information in real time. After the RAG process is completed, the evaluation system immediately obtains the results, including questions, generated answers, standard answers, recall slice information, etc.
[0074] Furthermore, if an A / B comparison is required, the current experiment is designated as experiment A, and an existing experiment is selected from the experiment list as experiment B. The keyword score and generation quality score of the overall A / B experiment are then calculated.
[0075] Furthermore, if A / B testing is not required, the recall quality index, ranking quality index, and generation quality index of the cases can be calculated.
[0076] Furthermore, based on the current development task, it is determined whether to mark the health status of the case. If not, the output is directly displayed on the visualization interface. If so, the keyword coverage, blue_score_finalanswer value, and blue_score_finalanswer are calculated according to the systematic marking rules.
[0077] Furthermore, the health status of a case is determined based on its keyword coverage, blue_score_finalanswer value, and blue_score_finalanswer. If the case is deemed healthy according to the rules, its status is marked as normal. If it is unhealthy, case attribution is required. Several factors that may lead to unhealthy status include chunking, recall, reranking, and model generation.
[0078] Furthermore, after the evaluation is completed, the system displays the evaluation results on a visual interface, providing full-link visualization tools to show the data flow, processing procedures, and result analysis during the evaluation process. This helps users analyze the evaluation results more intuitively, identify performance bottlenecks, and provide a basis for optimization decisions.
[0079] Example 3
[0080] An automated evaluation system for improving the stability of a RAG system is provided, comprising a processor and a memory storing a plurality of computer instructions, which, when executed by the processor, achieve the improvement.
[0081] The steps of an automated method for assessing the stability of a RAG system.
[0082] For specific limitations on automated evaluation systems for improving RAG system stability, please refer to the limitations on automated evaluation methods for improving RAG system stability mentioned above, which will not be repeated here.
[0083] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the method in the embodiments of the present invention by running the computer program stored in the memory.
[0084] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.
[0085] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.
[0086] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. An automated evaluation method for improving the stability of a RAG system, characterized in that, Includes the following steps: S1. Receive request cases. The specific content of the request cases includes the question, the recalled knowledge base slices, the model generation results, and the standard answer. S2. Calculate evaluation metrics, obtain knowledge base slices of questions and recalls, evaluate recall quality, and calculate recall quality metrics; obtain knowledge base slices of recalls and model generation results, evaluate ranking quality, and calculate ranking quality metrics; obtain knowledge base slices of questions and recalls, model generation results, and standard answers, evaluate generation quality, and calculate generation quality metrics to obtain recall quality metrics, ranking quality metrics, and generation quality metrics. S3. Based on the recall quality indicators, ranking quality indicators and generated quality indicators, automatically label the cases requesting evaluation, identify the health status of the cases, perform attribution analysis on unhealthy cases, conduct A / B experiments, compare with the standard answer, and analyze the case situation. S4. Based on the requested cases, evaluation indicators, and case annotations, perform visualization output and output the final evaluation results.
2. The method according to claim 1, characterized in that, Step S1 also includes request data, which includes two types: singleton request data and batch request data. Singleton request data can be initiated through singleton request mode, and batch request data can be initiated through batch request mode.
3. The method according to claim 2, characterized in that, The request data in step S1 comes from the evaluation database, and the dataset examples in the evaluation database include questions, answers, the fragment containing the answer, and the document containing the fragment.
4. The method according to claim 1, characterized in that, The recall quality metrics include ContextPrecision, ContextRecall, and ContextEntitiesRecall; the ranking quality metrics include Hit Rate, MRR (Mean Reciprocal Ranking), and NDCG (Normalized Discount Cumulative Gain); and the generation quality metrics include Answer Relevance, AnswerCorrectness, and Faithfulness.
5. The method according to claim 1, characterized in that, The cases requesting evaluation in step S3 are automatically labeled using systematic labeling rules, ensuring the standardization and consistency of the labeling process.
6. The method according to claim 5, characterized in that, The systematic annotation rules first calculate the keyword coverage rate of the standard answer. If there is full coverage, the case status is marked as normal. If the keyword coverage rate is not 1, it is judged as incomplete recall, the case is attributed to recall, and the case status is automatically marked as bad. Next, the `rouge_score_rerank` value after knowledge base slice re-ranking is calculated. If the `rouge_score_rerank` value is less than a predetermined threshold, the case is judged as not containing the expected answer in the re-ranking, attributed to rerank, and the case status is automatically marked as bad. Finally, the `blue_score_finalanswer` value of the case is calculated. If the `blue_score_finalanswer` value is less than a predetermined threshold, the case is judged as text redundancy, the case is attributed to chunk, and the case status is automatically marked as bad.
7. The method according to claim 1, characterized in that, The A / B test comparison includes importing the A / B tests into the evaluation system, calculating the keyword scores and overall generation quality scores of the two tests respectively, and analyzing the iteration quality of the RAG system and the impact of different strategies and parameters on the performance of the RAG system model by obtaining the experimental configuration parameters of the A / B tests.
8. The method according to claim 7, characterized in that, The experimental configuration parameters include chunk parameters, rewrite parameters, recall model selection, rerank model selection, and generation model selection.
9. The method according to claim 1, characterized in that, The process involves visualizing the request cases, evaluation metrics, and case annotations to output the final evaluation results. This includes visually displaying the request cases, evaluation metrics, and case annotations, while also providing information on the execution of the RAG process. The visualization includes an HTML-based interface where users can view various information related to the evaluation data, including user information, request cases, parameter configurations, time information, and evaluation metrics.
10. An automated evaluation system for improving the stability of a RAG system, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the automated evaluation method for improving the stability of the RAG system as described in any one of claims 1 to 9.