Method and system for evaluating quality of RAG knowledge base driven by large language model

Abstract

The invention discloses a big language model-driven RAG knowledge base quality evaluation method and system. The method comprises the following steps of: constructing QAR triples of different problem types by utilizing a big language model; performing review optimization on the triple by utilizing a large language model; inputting the constructed questions into an RAG knowledge base question-answering system, and recording replies of the question-answering system, retrieved reference text segments and reordering model scoring for each question; according to a recording result, automatically calculating scores of various preset indexes; and according to the question type, weighting and calculating a comprehensive score. According to the method, the external information source can be flexibly adjusted according to user requirements, and the dynamic QAR set containing a plurality of problem types is automatically created, so that the flexibility and individuation of an evaluation scene are realized. Besides, a set of complete evaluation indexes and weighting rules is established to comprehensively consider the retrieval accuracy, the generation quality, the user preference and the requirements in practical application, so that the evaluation result can comprehensively and objectively reflect the performance of the RAG knowledge base question-answering system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a RAG knowledge base quality evaluation method and system driven by a large language model. Background Art

[0002] By retrieving relevant information from external sources, Retrieval Augmented Generation (RAG) can significantly improve the accuracy of large language model (LLM) questions and answers, reduce hallucinations, and thus improve the reliability and richness of the response content. This method of integrating retrieval and large language model generation capabilities makes the generated responses not only semantically appropriate in the context, but also traceable based on the latest and most accurate information currently available. These advantages have made RAG technology widely used in building knowledge base question-answering systems.

[0003] However, due to the dependence of RAG technology on dynamic external information and the complex interaction between the retrieval and generation components in its own structure, it is quite challenging to conduct a comprehensive and objective quality evaluation of the knowledge base question answering system based on RAG technology. Traditional question answering system evaluation relies on fixed external datasets and manually constructed question-answer sets (QA sets), and uses absolute numerical indicators to calculate the similarity between the knowledge base replies and the groundtruth end-to-end. Although these methods are simple and easy to use, they have the following shortcomings when evaluating the knowledge base question answering system based on RAG technology:

[0004] (1) Fixed evaluation scenarios caused by fixed documents and QA sets: The traditional evaluation method based on static data sets relies on manually constructed QA sets and cannot reflect the RAG system's ability to handle dynamically changing information. Given the original design intention of the RAG system, its evaluation method also needs to be able to adapt to this dynamic nature to avoid the evaluation process limiting the full display of system performance.

[0005] (2) The evaluation caused by absolute numerical indicators is too one-sided: The end-to-end absolute numerical indicators only focus on the similarity between the final response and the ground truth, and do not deeply consider multiple important aspects of the retrieval and generation process in the RAG system and the complex interaction between retrieval accuracy and generation quality. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a RAG knowledge base quality evaluation method and system driven by a large language model.

[0007] The technical solution provided by the present invention is as follows:

[0008] A large language model driven RAG knowledge base quality evaluation method comprises the following steps:

[0009] S1. Generate QAR triples based on the preset question type using a large language model based on user documents<Question,Answer,Reference> , Question is a question, Answer is an answer, and Reference is a reference; the question types include factual questions and answers, comparison, information summary, and multi-hop reasoning;

[0010] S2, review and optimize QAR triples through large language models, including supplementing missing content, reviewing redundant content, adding missing references, and deleting irrelevant content;

[0011] S3. Send the generated Question to the RAG knowledge base question-answering system to be evaluated, and obtain the response of the question-answering system, the retrieved reference text segment Relevant chunk and the rerank score of the reranking model; Relevant chunk refers to the original document text block highly relevant to the question obtained by retrieval and reranking in the question-answering system, and the rerank score refers to the score of the reranking model on the reference text segment;

[0012] S4, calculate the average similarity score of the reference text segment of each question based on Question and Rerank score;

[0013] S5. Calculate the ranking accuracy of the reference text segment of each question based on Question, Reference, Relevant chunk and Rerank score;

[0014] S6. Calculate the relevance of the reference text segment for each question based on the Question and Relevant chunks;

[0015] S7. Calculate the response loyalty of each question based on Question, Response and Relevant chunks;

[0016] S8. Calculate the response relevance of each question based on Question and Response;

[0017] S9, calculate the similarity between the reply and the answer of each question based on Question, Answer and Response;

[0018] S10. Weight the scores of various indicators calculated in S4 to S9 according to the question type, and finally obtain a comprehensive score for evaluating the quality of the RAG knowledge base.

[0019] Further, in step S2:

[0020] For QAR triples<Question,Answer,Reference> If the reference Reference contains content that does not appear in the answer Answer, it will be supplemented accordingly according to the question Question; conversely, if the answer Answer contains content that does not appear in the reference Reference, first check whether there is an ignored reference text segment in the user document. If so, add it to the Reference, update the QAR triple, and keep the answer Answer unchanged, otherwise remove irrelevant content from the answer Answer.

[0021] Furthermore, in step S4, the reference text segment Relevant chunk corresponding to each question is sorted in descending order according to the Rerank score of the reranking model, and then the average Rerank score of the first K Relevant chunks is calculated and used as the average similarity score of the reference text segment of each question.

[0022] Further, in step S5:

[0023] First, use the Rerank score to rearrange the reference text segment Relevant chunk in descending order;

[0024] For a question, its reference is recorded as [R1,...,R p ], the sorted reference text segment is denoted as [C1,...,C q ], for [C1,...,C q Each reference text segment C in ] k (1≤k≤q), traverse [R1,...,R p Each reference R in ] l (1≤l≤p) performs field-by-field precise matching and records C k R l Content accounts for R l Ratio of total content like If it is greater than the preset threshold, then C k Contains R l ;

[0025] The reference text segment containing the reference is recorded as a true RAG reference text segment, which is represented by [TC1,...,TC w ], each true RAG references the text segment TC i The weight corresponding to (1≤i≤w) is On the contrary, the remaining reference text segments are recorded as pseudo RAG reference text segments, denoted as [FC1,...,FC e ], q = w + e; record each pseudo RAG reference text segment in the sorted reference text segment [C1,...,C q ], denoted as [Y1,...,Y e ]; the reference text segment ranking score is calculated as:

[0026]

[0027] Where f represents the penalty factor, f>1; the score is normalized to the interval [0,1], and the reference text segment ranking accuracy precision = (score+1) / 2 is obtained.

[0028] Further, in step S6, the tuple consisting of Question and Relevant chunk is<Question,Relevant chunk> The relevant chunks are fed into the large language model one by one, and the large language model outputs a relevance score. If the relevant chunk provides more context related to the question or can provide context for more parts of the question, its relevance score will be higher. Finally, the relevance score will be normalized to the interval [0,1].

[0029] Further, in step S7:

[0030] The large language model splits the response of the knowledge base question-answering system into several statement points, and determines point by point whether the statement point is supported by the corresponding reference text segment Relevant chunk. The higher the proportion of supported statement points in the response, the higher its loyalty is and the lower the possibility of hallucination is.

[0031] Further, in step S8, the tuple consisting of Question and Response<Question,Response> The responses are fed into a large language model one group at a time, which then outputs a response relevance score. Finally, the response relevance score is normalized to the range [0,1].

[0032] Further, in step S9,

[0033] The large language model splits the response and answer of the knowledge base question-answering system into several argument statements, and traverses the argument statements of the response and the answer to find argument pairs with highly similar semantics. The higher the proportion of argument pairs contained in the statement in the answer, the higher the similarity between the response and the answer.

[0034] Furthermore, in step S10, for different types of questions, the weighting rules of the evaluation indicators are different:

[0035] When the question type is factual question answering, the weights of the average similarity score of the reference text segment, the ranking accuracy, and the similarity between the response and the answer of the knowledge base question answering system are 0.1, 0.1, and 0.8 respectively;

[0036] When the question type is comparison, the weights of the average similarity score of the reference text segment, ranking accuracy, relevance, reply loyalty of the knowledge-based question-answering system, reply relevance, and similarity with the answer are 0.1, 0.16, 0.16, 0.16, 0.16, and 0.26, respectively;

[0037] When the question type is information summary, the weights of the average similarity score of the reference text segment, ranking accuracy, relevance, reply loyalty of the knowledge-based question-answering system, reply relevance, and similarity with the answer are 0.1, 0.2, 0.2, 0.2, 0.1, and 0.2, respectively;

[0038] When the question type is multi-hop reasoning, the weights of the average similarity score of the reference text segment, ranking accuracy, relevance, reply loyalty of the knowledge base question-answering system, reply relevance, and similarity to the answer are 0.05, 0.2, 0.15, 0.2, 0.1, and 0.3, respectively.

[0039] A RAG knowledge base quality evaluation system based on the above method includes a QAR data set generation module and an injection evaluation module, wherein:

[0040] The QAR dataset generation module is used to generate QAR triples based on user documents using a large language model according to preset question types.<Question,Answer,Reference> And the generated QAR triples are reviewed and optimized through a large language model;

[0041] The injection evaluation module is used to send the Question in the QAR triple to the RAG knowledge base question-answering system to be evaluated, obtain the reply Response of the question-answering system, the retrieved reference text segment Relevant chunk and the rerank score of the reranking model, and calculate various evaluation indicators based on the obtained content, and finally obtain a comprehensive score for evaluating the quality of the RAG knowledge base through weighting;

[0042] The injection evaluation module also includes a reference text segment evaluation module, a knowledge base reply evaluation module and a weighted setting module. The reference text segment evaluation module is used to calculate the average similarity score of the reference text segment, the ranking accuracy of the reference text segment and the relevance of the reference text segment. The knowledge base reply evaluation module is used to calculate the reply loyalty, reply relevance and the similarity between the reply and the answer. The weighted setting module is used to set the weighted values ​​of various evaluation indicators.

[0043] Compared with the prior art, the present invention has at least the following beneficial effects:

[0044] The present invention can flexibly adjust external information sources according to user needs, automatically create dynamic QAR sets containing several types of questions, and thus realize the flexibility and personalization of evaluation scenarios. In addition, a complete set of evaluation indicators and weighting rules are established to comprehensively consider retrieval accuracy, generation quality, user preferences, and needs in practical applications, thereby ensuring that the evaluation results can comprehensively and objectively reflect the performance of the knowledge base question-answering system based on RAG technology in the user-set scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0046] Figure 1 It is a framework diagram of a RAG knowledge base quality evaluation method provided by an embodiment of the present invention;

[0047] Figure 2 is a flow chart of calculating the accuracy of sorting reference text segments provided by an embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the knowledge base reply loyalty calculation process provided by an embodiment of the present invention;

[0049] Figure 4 It is a schematic diagram of a knowledge base and answer similarity calculation process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without making creative work are all within the scope of protection of the present invention.

[0051] This embodiment provides a RAG knowledge base quality evaluation method driven by a large language model. Figure 1 As shown, it mainly includes the following steps:

[0052] S1. Generate QAR triples based on the preset question type using a large language model and prompt sentences based on user documents<Question,Answer,Reference> , i.e. question-answer-reference (reference here refers to the actual reference text segment of the QA pair, which is distinguished from the relevant chunk of reference text segment retrieved by RAG below). Question types include factual questions and answers, comparisons, information summaries, and multi-hop reasoning, and prompts include detailed instructions and some examples.

[0053] S2. Based on user documents, the QAR triples are reviewed and optimized through a large language model.

[0054] Review optimization specifically includes supplementing missing content, reviewing redundant content, adding missing references, and deleting irrelevant content. Specifically, for QAR triples<Question,Answer,Reference> If the reference contains content that has not appeared in the answer, it will be supplemented accordingly according to the question; conversely, if the answer contains content that has not appeared in the reference, first check whether there is an ignored reference text segment in the user document. If so, add it to the Reference, update the QAR triple, and keep the answer unchanged. Otherwise, remove irrelevant content from the answer. This can solve the possible hallucination problem in the generation of QAR triples, ensure that the final answer is accurate, and is supported by the reference.

[0055] S3: Send the Question collection to the RAG knowledge base question-answering system based on user documents, record the response of the question-answering system, the retrieved reference text segment Relevant chunk and the rerank score, and construct a quadruple<Question,Response,Relevant chunk,Rerank score> The retrieved reference text segment Relevant chunk refers to the original document text block highly relevant to the question obtained through retrieval and reranking in the RAG system. The reference text segment reranking model score Rerank score refers to the score of the reference text segment by the reranking model. The higher the score, the higher the relevance of the reference text segment Relevant chunk to the question.

[0056] S4. According to<Question,Rerank score> Calculate the average similarity score of the reference text segments for each question. Due to the top K parameter setting in the RAG system, each question will correspond to K reference text segments.

[0057] S5. According to<Question,Reference,Relevant chunk,Rerank score> Calculate the ranking precision of the reference text segment for each question.

[0058] like Figure 2 As shown in the figure, the Rerank score is used to re-rank the reference chunks in descending order. For a question, its real reference is denoted as [R1,...,R p ], the sorted reference text segment Relevantchunk is denoted as [C1,...,C q For [C1,...,C q ] in each reference text segment C k (1≤k≤q), traverse [R1,...,R p ] for each real reference R l (1≤l≤p) performs field-by-field precise matching and records C k R l Content accounts for R l Ratio of total content like If it is greater than the preset threshold (here the threshold is 0.2 to reduce the impact of some keyword duplication), then C k Contains R lIn this way, the reference text segment containing the true reference is recorded as the true RAG reference text segment, which is represented by [TC1,...,TC w ], each true RAG references the text segment TC i The weight corresponding to (1≤i≤w) is On the contrary, the remaining reference text segments are recorded as pseudo RAG reference text segments, denoted as [FC1,...,FC e ], q=w+e. Correspondingly, record each pseudo RAG reference text segment in the sorted reference text segment [C1,...,C q ], denoted as [Y1,...,Y e It is generally believed that the true RAG text segment is in [C1,...,C q The lower the ranking in the text segment, the more severe the penalty. Therefore, the ranking score of the reference text segment is calculated as:

[0059]

[0060] Where f is the penalty factor, f>1, here we take 1.1. Obviously, the more true RAG reference text segments there are, the higher the ranking is and the larger the score is. Finally, the score is normalized to the interval [0,1], and the precision of the reference text segment ranking is obtained as: precision=(score+1) / 2.

[0061] S6. According to<Question,Relevant chunk> Calculate the reference text segment relevance for each question.

[0062] The tuple<Question,Relevant chunk> The large language model is fed with each chunk, and the detailed instructions and examples are adjusted. The large language model outputs a relevance score to quantitatively evaluate the relevance of the content of the reference text segment Relevant chunk to the question. Specifically, if the Relevant chunk provides more context related to the question or can provide context for more parts of the question, its relevance score should increase, which means that the Relevant chunk can provide more detailed and comprehensive information for the question; Relevant chunks that are completely irrelevant to the question should get 0 points; Relevant chunks that are partially related to the question should get 2 points, 3 points, or 4 points, and the higher the score, the higher the relevance; Relevant chunks that are mostly related to the question should get 5 points, 6 points, 7 points, or 8 points, and the higher the score, the higher the relevance; Relevant chunks that are related to the entire question can get 9 points or 10 points; Relevant chunks that are completely relevant and helpful in answering the question should get 10 points. Finally, the relevance score is normalized to the interval [0,1].

[0063] S7. According to<Question,Response,Relevant chunk> Calculate the knowledge base response fidelity for each question. Figure 3 As shown in the figure, the detailed instructions and examples are adjusted, and the large language model splits the knowledge base response into several argument statements. The detailed instructions and examples are adjusted again, and the large language model determines point by point whether the argument statement is supported by the corresponding reference text segment Relevant chunk. The higher the proportion of supported argument statements in a knowledge base response, the higher its loyalty and the lower the possibility of hallucination. It can quantitatively evaluate whether the knowledge base response is loyal to the retrieved reference text segment Relevant chunk.

[0064] S8. According to<Question,Response> Calculate the knowledge base response relevance for each question.

[0065] The tuple<Question,Response> Feed each group into the large language model, adjust the detailed instructions and examples, and the large language model outputs the relevance score to quantitatively evaluate the directness, completeness, accuracy, and exhaustiveness of the knowledge base response. Specifically, responses of different lengths should be treated equally; the relevance score should increase as the response can provide more contextual information for the question; responses that are not related to the question should get 0 points; responses that are partially related to the question should get 2 points, 3 points, or 4 points, and the higher the score, the higher the relevance; responses that are related to most of the question should get 5 points, 6 points, 7 points, or 8 points, and the higher the score, the higher the relevance; responses that are related to the entire question can get 9 points or 10 points; responses that are related and fully answer the question should get 10 points; responses that are obviously wrong and only seem to be related should get 0 points. Finally, the relevance score is normalized to the interval [0,1].

[0066] S9. According to<Question,Answer,Response> Calculate the end-to-end similarity between the knowledge base response and the true answer for each question. Figure 4 As shown, the detailed instructions and examples are adjusted, and the large language model splits the knowledge base response Response and the real answer Answer into several argument statements, and then the detailed instructions and examples are adjusted, and the large language model traverses the argument statement of the knowledge base response Response and the argument statement of the real answer Answer to find argument pairs with highly similar semantics. The higher the proportion of argument pairs contained in the argument statement in the real answer Answer, the higher the similarity between the knowledge base response Response and the real answer Answer. In addition, traditional absolute numerical indicators such as BLEU and ROUGE will also be used to intuitively measure the similarity between the knowledge base response Response and the real answer Answer.

[0067] S10. For each question, the calculated scores of each indicator are weighted according to the question type to obtain a comprehensive score. For different question types, the weighting rules of many evaluation indicators are different, as shown in the following table.

[0068] Table 1 Multi-index weighting rules

[0069]

[0070] Corresponding to the above method, this embodiment also provides a RAG knowledge base quality evaluation system driven by a large language model, which mainly includes a QAR data set generation module and an injection evaluation module, wherein the QAR data set generation module is used to generate QAR triples based on user documents using a large language model according to preset question types.<Question,Answer,Reference> And the generated QAR triples are reviewed and optimized through a large language model.

[0071] The injection evaluation module is used to send the Question in the QAR triplet into the RAG knowledge base question-answering system to be evaluated, obtain the response Response of the question-answering system, the retrieved reference text segment Relevant chunk and the rerank score of the reranking model, and calculate various evaluation indicators based on the obtained content. Finally, a comprehensive score for evaluating the quality of the RAG knowledge base is obtained through weighted analysis.

[0072] The injection evaluation module further includes a reference text segment evaluation module, a knowledge base reply evaluation module and a weighted setting module. The reference text segment evaluation module is used to calculate the average similarity score of the reference text segment, the reference text segment ranking accuracy and the reference text segment relevance. The knowledge base reply evaluation module is used to calculate the reply loyalty, reply relevance and the similarity between the reply and the answer. The weighted setting module is used to set the weighted values ​​of various evaluation indicators.

[0073] The above system can execute a large language model driven RAG knowledge base quality evaluation method as described above, and has corresponding functional modules and beneficial effects. For technical details not described in detail, please refer to the RAG knowledge base quality evaluation method provided in the above content.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A large language model driven RAG knowledge base quality evaluation method, characterized in that: Includes steps: S1. Generate QAR triples based on the preset question type using a large language model based on user documents<Question,Answer,Reference> , Question is a question, Answer is an answer, and Reference is a reference; the question types include factual questions and answers, comparison, information summary, and multi-hop reasoning; S2, review and optimize QAR triples through large language models, including supplementing missing content, reviewing redundant content, adding missing references, and deleting irrelevant content; S3. Send the generated Question to the RAG knowledge base question-answering system to be evaluated, and obtain the response of the question-answering system, the retrieved reference text segment Relevant chunk and the rerank score of the reranking model; Relevant chunk refers to the original document text block highly relevant to the question obtained by retrieval and reranking in the question-answering system, and the rerank score refers to the score of the reranking model on the reference text segment; S4, calculate the average similarity score of the reference text segment of each question based on Question and Rerank score; S5. Calculate the ranking accuracy of the reference text segment of each question based on Question, Reference, Relevant chunk and Rerank score; S6. Calculate the relevance of the reference text segment for each question based on the Question and Relevant chunks; S7. Calculate the response loyalty of each question based on Question, Response and Relevant chunks; S8. Calculate the response relevance of each question based on Question and Response; S9, calculate the similarity between the reply and the answer of each question based on Question, Answer and Response; S10. Weight the scores of various indicators calculated in S4 to S9 according to the question type, and finally obtain a comprehensive score for evaluating the quality of the RAG knowledge base.

2. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S2: For QAR triples<Question,Answer,Reference> If the reference Reference contains content that does not appear in the answer Answer, it will be supplemented accordingly according to the question Question; conversely, if the answer Answer contains content that does not appear in the reference Reference, first check whether there is an ignored reference text segment in the user document. If so, add it to the Reference, update the QAR triple, and keep the answer Answer unchanged, otherwise remove irrelevant content from the answer Answer.

3. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S4, the reference text segment Relevant chunk corresponding to each question is sorted in descending order according to the Rerank score of the reranking model, and then the average Rerank score of the first K Relevant chunks is calculated and used as the average similarity score of the reference text segment for each question.

4. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S5: First, use the Rerank score to rearrange the reference text segment Relevant chunk in descending order; For a question, its reference is recorded as [R1,...,R p ], the sorted reference text segment is denoted as [C1,...,C q ], for [C1,...,C q Each reference text segment C in ] k (1≤k≤q), traverse [R1,...,R p Each reference R in ] l (1≤l≤p) performs field-by-field precise matching and records C k R l Content accounts for R l Ratio of total content like If it is greater than the preset threshold, then C k Contains R l ; The reference text segment containing the reference is recorded as a true RAG reference text segment, which is represented by [TC1,...,TC w ], each true RAG references the text segment TC i The weight corresponding to (1≤i≤w) is On the contrary, the remaining reference text segments are recorded as pseudo RAG reference text segments, denoted as [FC1,...,FC e ], q = w + e; record each pseudo RAG reference text segment in the sorted reference text segment [C1,...,C q ], denoted as [Y1,...,Y e ]; the reference text segment ranking score is calculated as: Where f represents the penalty factor, f>1; the score is normalized to the interval [0,1], and the reference text segment ranking accuracy precision = (score+1) / 2 is obtained.

5. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S6, the bigram consisting of Question and Relevant chunk is<Question,Relevant chunk> The chunks are fed into the large language model one by one, and the large language model outputs a relevance score. If the relevant chunk provides more context related to the Question or can provide context for more parts of the Question, its relevance score will be higher. Finally, the relevance score will be normalized to the interval [0,1].

6. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S7: The large language model splits the response of the knowledge base question-answering system into several statement points, and determines point by point whether the statement point is supported by the corresponding reference text segment Relevant chunk. The higher the proportion of supported statement points in the response, the higher its loyalty is and the lower the possibility of hallucination is.

7. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S8, the tuple consisting of Question and Response is<Question,Response> The responses are fed into a large language model one group at a time, which then outputs a response relevance score. Finally, the response relevance score is normalized to the range [0,1].

8. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S9, The large language model splits the response and answer of the knowledge base question-answering system into several argument statements, and traverses the argument statements of the response and the answer to find argument pairs with highly similar semantics. The higher the proportion of argument pairs contained in the statement in the answer, the higher the similarity between the response and the answer.

9. The RAG knowledge base quality evaluation method according to claim 1, characterized in that: In step S10, for different types of questions, the weighting rules of each evaluation indicator are different: When the question type is factual question answering, the weights of the average similarity score of the reference text segment, the ranking accuracy, and the similarity between the response and the answer of the knowledge base question answering system are 0.1, 0.1, and 0.8 respectively; When the question type is comparison, the weights of the average similarity score of the reference text segment, ranking accuracy, relevance, reply loyalty of the knowledge-based question-answering system, reply relevance, and similarity with the answer are 0.1, 0.16, 0.16, 0.16, 0.16, and 0.26, respectively; When the question type is information summary, the weights of the average similarity score of the reference text segment, ranking accuracy, relevance, reply loyalty of the knowledge-based question-answering system, reply relevance, and similarity with the answer are 0.1, 0.2, 0.2, 0.2, 0.1, and 0.2, respectively; When the question type is multi-hop reasoning, the weights of the average similarity score of the reference text segment, ranking accuracy, relevance, reply loyalty of the knowledge base question-answering system, reply relevance, and similarity to the answer are 0.05, 0.2, 0.15, 0.2, 0.1, and 0.3, respectively.

10. A RAG knowledge base quality evaluation system based on the method according to any one of claims 1 to 9, characterized in that: It includes QAR dataset generation module and injection evaluation module, where: The QAR dataset generation module is used to generate QAR triples based on user documents using a large language model according to preset question types.<Question,Answer,Reference> And the generated QAR triples are reviewed and optimized through a large language model; The injection evaluation module is used to send the Question in the QAR triple to the RAG knowledge base question-answering system to be evaluated, obtain the reply Response of the question-answering system, the retrieved reference text segment Relevant chunk and the rerank score of the reranking model, and calculate various evaluation indicators based on the obtained content, and finally obtain a comprehensive score for evaluating the quality of the RAG knowledge base through weighting; The injection evaluation module also includes a reference text segment evaluation module, a knowledge base reply evaluation module and a weighted setting module. The reference text segment evaluation module is used to calculate the average similarity score of the reference text segment, the ranking accuracy of the reference text segment and the relevance of the reference text segment. The knowledge base reply evaluation module is used to calculate the reply loyalty, reply relevance and the similarity between the reply and the answer. The weighted setting module is used to set the weighted values ​​of various evaluation indicators.

Images