An evaluation method and device for retrieval enhancement generation (RAG) system

Abstract

This application provides an evaluation method and apparatus for a retrieval enhancement generation (RAG) system, relating to the field of data processing technology. The method specifically involves: inputting a preset question-and-answer text into the target retrieval enhancement generation (RAG) system to obtain a first response text and retrieval context text; comparing the standard statement set corresponding to the obtained standard response text, the first statement set corresponding to the first response text, and the retrieval context statement set corresponding to the retrieval context text; analyzing and calculating the correct statement subset, incorrect statement subset, missing correct statement subset, and irrelevant statement subset in the generated first statement set to obtain a first performance analysis result; and generating a target comprehensive performance evaluation result for the target RAG system based on multiple first performance analysis results. This application can evaluate the performance of the RAG system from multiple aspects, providing strong support for the wider and more efficient application of RAG systems in the field of artificial intelligence.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and specifically to an evaluation method and apparatus for a search-enhanced generation RAG system. Background Technology

[0002] In the field of artificial intelligence, especially with the widespread application of large language models, Retrieval Augmentation (RAG) systems have become a prime example of leveraging the capabilities of large models. By cleverly combining retrieval and generation modules, RAG systems demonstrate the potential to provide richer and more accurate content in information responses. They can quickly locate relevant information from massive amounts of data using the retrieval module, and then integrate this information into logical and targeted responses using the generation module, providing users with more valuable services.

[0003] However, there are currently many challenges in comprehensively and accurately evaluating the performance of RAG systems. Traditional evaluation methods struggle to effectively measure the quality of long text responses. Because long text responses often contain complex structures and rich information, traditional evaluation methods lack effective means to deeply analyze their accuracy, logic, and completeness, easily overlooking details within long texts, resulting in incomplete and inaccurate quality assessments. Furthermore, traditional evaluation metrics are often based on simple text matching or surface features, failing to fully consider deeper factors such as semantic understanding and logical coherence that humans focus on when comprehending and judging information, leading to significant discrepancies between evaluation results and actual human perception.

[0004] Therefore, existing technologies are insufficient for in-depth and detailed analysis of RAG systems, failing to meet the need for comprehensive and accurate performance evaluation, thus limiting the further optimization and expansion of RAG systems in practical applications. Therefore, a new evaluation framework is urgently needed to overcome these problems and achieve effective performance evaluation of RAG systems. Summary of the Invention

[0005] In view of this, embodiments of this application provide an evaluation method and apparatus for a retrieval-enhanced generation RAG system, which can evaluate the performance of the RAG system from multiple aspects, providing strong support for the wider and more efficient application of the RAG system in the field of artificial intelligence.

[0006] In a first aspect, embodiments of this application provide an evaluation method for a retrieval-enhanced generation (RAG) system, including:

[0007] Obtain multiple preset question-and-answer texts and the standard response text corresponding to each preset question-and-answer text;

[0008] The preset question and answer text is input into the target retrieval enhancement generation RAG system to obtain the first response text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first response text;

[0009] Obtain the standard statement set corresponding to the standard response text, the first statement set corresponding to the first response text, and the search context statement set corresponding to the search context text;

[0010] By combining the semantics of each statement in the standard statement set, the first statement set, and the retrieval context statement set, a statement comparison is performed to generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set;

[0011] Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset, an analysis and calculation are performed to obtain a first performance analysis result. Based on the first performance analysis result among multiple preset question and answer texts, multiple first response texts, and multiple search context texts, a target comprehensive performance evaluation result of the target RAG system is generated.

[0012] As an optional implementation of this application, obtaining the standard statement set corresponding to the standard response text, the first statement set corresponding to the first response text, and the search context statement set corresponding to the search context text includes:

[0013] The standard response text, the first response text, and the retrieval context text are extracted using the first major language model to obtain the standard declaration set, the first declaration set, and the retrieval context declaration set.

[0014] As an optional implementation of this application, the step of comparing the semantics of each statement in the standard statement set, the first statement set, and the retrieval context statement set to generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set includes:

[0015] Obtain the semantics corresponding to each statement in the first statement set, the semantics corresponding to each statement in the standard statement set, and the semantics corresponding to each statement in the retrieval context statement set;

[0016] Based on the semantics corresponding to each statement in the first statement set and the standard statement set, a first proportion parameter of the statements in the first statement set in the standard statement set is obtained; the first proportion parameter is used to characterize the correctness of the first response text.

[0017] Based on the semantics corresponding to each statement in the first statement set and the standard statement set, a second proportion parameter is obtained for the statement in the standard statement set in the first statement set; the second proportion parameter is used to characterize the completeness of the first response text.

[0018] Based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, a third proportion parameter of the statements in the first statement set in the retrieval context statement set is obtained; the third proportion parameter is used to characterize the fidelity of the first response text.

[0019] Based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, a fourth proportion parameter of the statement in the standard statement set in the retrieval context statement set is obtained; the fourth proportion parameter is used to characterize the quality of the first response text.

[0020] The declaration comparison result is generated based on the first percentage parameter, the second percentage parameter, the third percentage parameter, and the fourth percentage parameter.

[0021] As an optional implementation of this application, before performing analysis and calculation based on the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, and the irrelevant declaration subset to obtain the first performance analysis result, the method further includes:

[0022] Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statements, the relevant context statement subset in the retrieval context statement set, and the number of statements in each subset of the irrelevant retrieval context statement subset, the overall performance index set of the target RAG system, the index set of the retrieval module, and the index set of the response generation module are obtained. As an optional implementation of this application embodiment, the method further includes:

[0023] The number of declarations in the correct declaration subset and the incorrect declaration subset are summed to obtain a first sum value, and the ratio of the number of declarations in the correct declaration subset to the first sum value is obtained to generate an overall precision value;

[0024] The number of declarations in the correct declaration subset and the missing declaration subset are summed to obtain a second sum value, and the ratio of the number of declarations in the correct declaration subset to the second sum value is obtained to generate an overall recall value;

[0025] The overall performance index set is generated based on the overall precision value and the overall recall value.

[0026] As an optional implementation of this application, the method further includes:

[0027] The number of declarations in the relevant search context declaration subset and the irrelevant search context declaration subset are summed to obtain a third sum value, and the ratio of the number of declarations in the relevant search context declaration subset to the third sum value is obtained to generate a search context precision value;

[0028] The correct statements and missing correct statements in the relevant context statement subset are summed to obtain a fourth sum value, and the ratio of the fourth sum value to the second sum value is obtained to generate a retrieval context precision value;

[0029] The set of performance metrics for the retrieval module is generated based on the retrieval context precision value and the retrieval context precision value.

[0030] As an optional implementation of this application, the method further includes:

[0031] Obtain a first value of the correct declaration in the relevant context declaration subset, and obtain the ratio of the first value to the fourth sum value to generate context utilization;

[0032] Obtain a second value of the erroneous declarations belonging to the union of the relevant context declaration subset and the unrelated context declaration subset, and obtain the ratio of the second value to the second sum value to generate noise sensitivity;

[0033] Obtain a third value for erroneous statements that do not belong to the union of the relevant context statement subset and the unrelated context statement subset, and obtain the ratio of the third value to the second sum value to generate the probability of hallucination occurrence;

[0034] Obtain a fourth value for correct statements that do not belong to the union of the relevant context statement subset and the unrelated context statement subset, and obtain the ratio of the fourth value to the second sum value to generate its own knowledge utilization.

[0035] The number of correctly declared statements in the irrelevant context declaration subset is summed with the number of incorrectly declared statements in the union of the relevant context declaration subset and the irrelevant context declaration subset to obtain a fifth value. The ratio of the fifth value to the second sum value is then obtained to generate a fidelity score.

[0036] Based on the context utilization, noise sensitivity, probability of hallucination, utilization of self-knowledge, and fidelity, a set of indicators is generated for the response generation module.

[0037] Secondly, embodiments of this application provide an evaluation apparatus for a retrieval enhancement generation RAG system, comprising:

[0038] The first acquisition unit is used to acquire multiple preset question and answer texts and the standard response text corresponding to each preset question and answer text.

[0039] The second acquisition unit is used to input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first response text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first response text;

[0040] The third acquisition unit is used to acquire the standard declaration set corresponding to the standard response text, the first declaration set corresponding to the first response text, and the search context declaration set corresponding to the search context text.

[0041] The comparison unit is used to perform a semantic comparison of each statement in the standard statement set, the first statement set, and the retrieval context statement set, and generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set.

[0042] The analysis unit is used to perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset to obtain a first performance analysis result, and to generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result among multiple preset question and answer texts, multiple first response texts, and multiple search context texts.

[0043] As an optional implementation of this application, the acquisition unit is specifically used to extract declarations from the standard response text, the first response text, and the retrieval context text using a first large language model, to obtain the standard declaration set, the first declaration set, and the retrieval context declaration set.

[0044] As an optional implementation of this application, the comparison unit is specifically used for: obtaining the semantics corresponding to each statement in the first statement set, the semantics corresponding to each statement in the standard statement set, and the semantics corresponding to each statement in the retrieval context statement set; obtaining a first proportion parameter of the statements in the first statement set in the standard statement set based on the semantics corresponding to each statement in the first statement set and the standard statement set; the first proportion parameter is used to characterize the correctness of the first response text; obtaining a second proportion parameter of the statements in the standard statement set in the first statement set based on the semantics corresponding to each statement in the first statement set and the standard statement set; the second proportion parameter... The first response text is characterized by a number representing its completeness. Based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, a third percentage parameter is obtained for the statements in the first statement set within the retrieval context statement set. This third percentage parameter represents the fidelity of the first response text. Based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, a fourth percentage parameter is obtained for the statements in the standard statement set within the retrieval context statement set. This fourth percentage parameter represents the quality of the first response text. The statement comparison result is generated based on the first percentage parameter, the second percentage parameter, the third percentage parameter, and the fourth percentage parameter.

[0045] As an optional implementation of this application, the analysis unit is further configured to obtain the overall performance index set of the target RAG system, the index set of the retrieval module, and the index set of the response generation module based on the number of declarations in each subset of the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, the irrelevant declarations, the relevant context declaration subset in the retrieval context declaration set, and the irrelevant context declaration subset. As an optional implementation of this application, the analysis unit is further configured to sum the number of declarations in the correct declaration subset and the incorrect declaration subset to obtain a first sum value, and obtain the ratio of the number of declarations in the correct declaration subset to the first sum value to generate an overall accuracy value.

[0046] The number of declarations in the correct declaration subset and the missing declaration subset are summed to obtain a second sum value, and the ratio of the number of declarations in the correct declaration subset to the second sum value is obtained to generate an overall recall value; the overall performance index set is generated based on the overall precision value and the overall recall value.

[0047] As an optional implementation of this application, the analysis unit is further configured to sum the number of declarations in the relevant search context declaration subset and the irrelevant search context declaration subset to obtain a third sum value, and obtain the ratio of the number of declarations in the relevant search context declaration subset to the third sum value to generate a search context precision value; and to sum the correct declarations and missing correct declarations in the relevant context declaration subset to obtain a fourth sum value, and obtain the ratio of the fourth sum value to the second sum value to generate a search recall value;

[0048] The set of performance metrics for the retrieval module is generated based on the retrieval context precision value and the retrieval context precision value.

[0049] As an optional implementation of this application, the analysis unit is further configured to: obtain a first value of the correct statements in the relevant context statement subset, and obtain the ratio of the first value to the fourth sum value to generate context utilization; obtain a second value of the incorrect statements belonging to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain the ratio of the second value to the second sum value to generate noise sensitivity; obtain a third value of the incorrect statements not belonging to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain the ratio of the third value to the second sum value to generate the probability of hallucination occurrence. ; Obtain a fourth value for correct statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain the ratio of the fourth value to the second sum value to generate self-knowledge utilization; Sum the number of correct statements in the irrelevant context statement subset with the number of incorrect statements in the union of the relevant context statement subset and the irrelevant context statement subset to obtain a fifth value, and obtain the ratio of the fifth value to the second sum value to generate fidelity; Based on the context utilization, the noise sensitivity, the probability of hallucination, the self-knowledge utilization, and the fidelity, generate the indicator set of the response generation module.

[0050] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to implement the evaluation method of the retrieval enhancement generation RAG system described in any of the above embodiments when executing the computer program.

[0051] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the evaluation method for the retrieval enhancement generation RAG system described in any of the above embodiments.

[0052] The evaluation method for a Retrieval Enhancement Generation (RAG) system provided in this application obtains preset question-and-answer text, standard response text, first response text, and retrieval context text, and further extracts corresponding declaration sets. This achieves comprehensive utilization of multiple aspects of data from the RAG system. Furthermore, it performs a declaration-level comparison of the RAG system's output (i.e., the first response text), the standard response text corresponding to the preset question text, and the context information used by the RAG system to generate answers (i.e., the retrieval context text). This delves into the semantic level of the text, obtaining the correct declaration subset, incorrect declaration subset, missing correct declaration subset, and irrelevant declaration subset from the first declaration set. This clarifies which declarations are correct, which are incorrect, which key information is missing, and which are irrelevant. Simultaneously, based on the first performance analysis results between multiple preset question-and-answer texts and their corresponding related texts, a comprehensive performance evaluation result is generated, enabling a holistic understanding of the RAG system's performance and providing a clear direction for its optimization. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings that need to be called in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 One of the flowcharts of the evaluation method for the retrieval enhancement generation RAG system provided in the embodiments of this application;

[0056] Figure 2 The second flowchart of the evaluation method for the retrieval enhancement generation RAG system provided in the embodiments of this application;

[0057] Figure 3 A schematic diagram of the declaration relationships for the evaluation method of the retrieval enhancement generation RAG system provided in the embodiments of this application;

[0058] Figure 4A schematic diagram of the structure of the evaluation apparatus for the retrieval enhancement generation RAG system provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0061] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0062] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0063] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] This application provides an evaluation method for a retrieval-enhanced RAG generation system, referring to... Figure 1 As shown, the evaluation method for this retrieval enhancement generation RAG system includes the following steps S101-S105:

[0065] S101. Obtain multiple preset question and answer texts and the standard response text corresponding to each preset question and answer text.

[0066] In this embodiment, the multiple preset question-and-answer texts and the standard response text corresponding to each preset question-and-answer text can be pre-set by staff to build basic data resources for subsequent evaluation. The acquired preset question-and-answer texts and standard response texts are key criteria for determining whether the performance of the target retrieval enhancement generation (RAG) system is up to standard. It should be noted that the selection of these preset texts must follow the principles of comprehensiveness and representativeness, covering various scenarios that the RAG system may be applied to, such as questions from different professional fields, with different language styles and varying degrees of complexity, to ensure that the evaluation can accurately reflect the system's capability boundaries in actual use and provide a reliable reference benchmark for subsequent accurate evaluation of system performance.

[0067] Retrieval-Augmented Generation (RAG) systems are a technical architecture in the field of artificial intelligence that combines retrieval and generation modules. They play a crucial role in large-scale language model applications, aiming to overcome the limitations of traditional generative models by introducing external knowledge sources to improve the quality and accuracy of generated content. In practical applications, when a user asks a question, the RAG system first uses its retrieval module to search for relevant information from a large amount of text data (such as document libraries and knowledge bases), forming the retrieval context. Subsequently, the generation module generates an answer based on the retrieved context and its own language generation capabilities. For example, in intelligent customer service scenarios, when a customer inquires about product usage, the RAG system can quickly retrieve relevant product manual content and, combined with a language model, generate accurate and detailed answers, providing users with more targeted services.

[0068] For example, when evaluating the target retrieval enhancement generation RAG system, 10 preset question-and-answer texts and corresponding standard answer texts can be set. The target retrieval enhancement generation RAG system can then retrieve and generate corresponding answer texts from these 10 preset answer texts. These 10 answer texts are then compared with the corresponding standard answer texts in subsequent steps to evaluate the performance of the target retrieval enhancement generation RAG system.

[0069] S102. Input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first response text and retrieval context text.

[0070] The retrieval context text refers to the context text retrieved by the target retrieval enhancement generation RAG system when generating the first response text.

[0071] In some embodiments, inputting the preset question-and-answer text into the target retrieval enhancement generation (RAG) system to obtain a first response text and retrieval context text simulates the actual operation of the current RAG system. The first response text obtained after inputting the preset question-and-answer text into the target RAG system is the direct output of the target RAG system under the current input. The retrieval context text records the knowledge source information relied upon by the target RAG system when generating the response. These two elements are crucial for in-depth analysis of the system's retrieval and generation mechanisms, realistically demonstrating how the system utilizes its retrieval capabilities to acquire information and generate answers when processing questions, providing direct data support for subsequent evaluation of system performance from multiple dimensions.

[0072] S103. Obtain the standard statement set corresponding to the standard response text, the first statement set corresponding to the first response text, and the search context statement set corresponding to the search context text.

[0073] It should be noted that a statement is a basic unit of information extracted from complex text, possessing independent semantic and judgmental value. It can be a statement, an opinion, a factual description, etc., capable of independently expressing a certain meaning and whose accuracy can be verified. For example, in a text about historical events, statements such as "a certain battle occurred in a specific year" or "a certain historical figure had a specific identity" can be considered statements.

[0074] In this embodiment, for a preset question-and-answer text, after obtaining its corresponding standard answer text, the first answer text output by the target RAG system, and the retrieved search context text, declaration extraction is performed on these three texts respectively to obtain the standard declaration set corresponding to the standard answer text, the first declaration set corresponding to the first answer text, and the search context declaration set corresponding to the search context text. This is because declarations, as information fragments with independent semantics and judgment value in the text, transform complex text into a quantifiable and comparable set of declarations by extracting declarations from the first answer text, standard answer text, and search context text. This allows for subsequent comparative analysis between multiple declarations, thereby more deeply exploring the characteristics and problems of the system in the information processing and generation process.

[0075] Specifically, in the application embodiment, the standard response text, the first response text, and the retrieval context text can be extracted using a first major language model to obtain the standard declaration set, the first declaration set, and the retrieval context declaration set.

[0076] Among them, the first Large Language Model (LLM), with its powerful language understanding and processing capabilities, can perform semantic analysis on the input text. For the first response text, the standard response text, and the retrieval context text, LLM can identify information segments with independent semantic and judgmental value and extract them as statements. For example, for a response text containing multiple sentences, LLM can extract each sentence or phrase that expresses an independent viewpoint or fact as a statement based on factors such as sentence structure and semantic logic, thus forming a corresponding set of statements.

[0077] Then, the first response text, the standard response text, and the retrieval context text are extracted using the first major language model to obtain the first declaration set, the standard declaration set, and the retrieval context declaration set.

[0078] S104. By combining the semantics of each statement in the standard statement set, the first statement set, and the retrieval context statement set, a statement comparison is performed to generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set.

[0079] It is understandable that comparing the declarations using the first declaration set, the standard declaration set, and the retrieval context declaration set is equivalent to comparing the semantics between the first response text, the standard response text, and the retrieval context text. Through this comparison, the correctness, completeness, fidelity, and other aspects of the response generated by the target RAG system can be obtained. At the same time, the quality of the context information obtained by the retrieval module and the generator's utilization of the retrieval information can also be evaluated.

[0080] Wherein, the correct declaration subset is the declaration in the first declaration set that matches the standard declaration set; the incorrect declaration subset includes declarations in the first declaration set that contradict the semantics of the declarations in the standard declaration set; the missing correct declaration subset is the declaration in the first declaration set that does not lack any of the declarations in the standard declaration set; and the irrelevant declaration subset is the declaration in the first declaration set corresponding to the first response text that has no semantic connection with any of the declarations in the search context declaration set.

[0081] Specifically, the correct declaration subset is obtained as follows: for each declaration in the first declaration set, it is precisely matched or semantically similar to the declarations in the standard declaration set (algorithms based on word vectors, semantic understanding, etc. can be used); when a declaration in the first declaration set is highly consistent with a declaration in the standard declaration set in terms of semantics and logic (for example, the matching degree in terms of meaning, topic, key information, etc. reaches a set threshold), then the declaration is determined to be a correct declaration and is added to the correct declaration subset.

[0082] The method for obtaining the subset of incorrect statements is as follows: For each statement in the first set of statements, perform an exact match or semantic similarity calculation with the statements in the standard set of statements (algorithms based on word vectors, semantic understanding, etc. can be used); when a statement in the first set of statements differs significantly from a statement in the standard set of statements in terms of semantics and logic, such as inconsistent key information, semantic contradiction, or inability to establish a reasonable logical connection, then the statement is identified as an incorrect statement and added to the subset of incorrect statements; if semantic analysis shows that the core content expressed by the two statements is consistent, then the statement in the first set of statements belongs to the subset of correct statements; if the first set of statements contains "the moon is the central celestial body of the solar system", then it clearly contradicts the standard statement and belongs to the subset of incorrect statements.

[0083] The missing subset of correct statements is obtained by checking each statement in the standard statement set one by one to see if there is a corresponding statement in the first statement set. If a statement exists in the standard statement set but has no matching or similar statement in the first statement set (this can also be determined using a semantic similarity algorithm), then this statement constitutes the missing subset of correct statements.

[0084] The irrelevant statement subset is obtained by comparing each statement in the first statement set with the statements in the retrieval context statement set. If a statement in the first statement set cannot be found to have any related statement in the retrieval context statement set (whether directly or indirectly related, which can be determined according to pre-defined association rules, such as topic relevance, semantic coherence, etc.), then the statement is considered an irrelevant statement and added to the irrelevant statement subset.

[0085] S105. Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset, perform analysis and calculation to obtain a first performance analysis result, and generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result among multiple preset question and answer texts, multiple first response texts, and multiple search context texts.

[0086] In this embodiment of the application, the first performance analysis result may include the performance of the target RAG system under the preset question text, and may include various parameters and indicators that can reflect the correctness, completeness, fidelity and quality of retrieval context information of the first response text, so as to more clearly demonstrate the performance of the target RAG system in a quantitative way.

[0087] Then, by integrating the results of comparisons between multiple pre-set question-and-answer texts and the corresponding first response texts, the system's performance under different problem scenarios is comprehensively evaluated, thereby generating a comprehensive performance evaluation result for the target RAG system. This result can intuitively present the system's level in key performance indicators such as accuracy, completeness, and reliability, and also provides key evidence for judging the improvement direction and practical application value of the RAG system. It is the core output of the entire evaluation process.

[0088] It should be noted that testing with multiple different pre-set question-and-answer texts ensures that the evaluation results better reflect the system's performance in real-world applications. Whether dealing with simple, common questions or complex, professional ones, the system receives thorough testing and evaluation.

[0089] Furthermore, after obtaining the subset of correct statements, the subset of incorrect statements, the subset of missing correct statements, and the subset of irrelevant statements, the specific implementation method for performing analysis and calculation based on the subset of correct statements, the subset of incorrect statements, the subset of missing correct statements, and the subset of irrelevant statements to obtain the first performance analysis result includes the following steps S1051 to S1055:

[0090] S1051. Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain the first proportion parameter of the statements in the first statement set in the standard statement set.

[0091] The first percentage parameter is used to characterize the correctness of the first response text.

[0092] Specifically, semantic understanding models or algorithms in natural language processing can be used to obtain the semantics corresponding to each statement, so as to obtain subsequent parameters based on the semantics of the statement. This transforms the textual statements into a semantic form that can be processed and compared by a computer, laying the foundation for subsequent calculation and comparison of percentage parameters.

[0093] Each statement in the first set of statements is semantically matched with the statements in the standard set of statements. Semantic similarity algorithms, such as cosine similarity or edit distance (calculated in semantic vector space), can be used to determine the degree of similarity between the statements in the first set and those in the standard set. If the similarity reaches a certain threshold (which can be set according to specific application scenarios and requirements), the two are considered to match. Then, the proportion of the number of matched statements to the total number of statements in the first set is counted to obtain the first proportion parameter. For example, if the first set of statements has 10 statements, and after semantic matching, 7 statements have a high similarity to statements in the standard set, then the first proportion parameter is 7 / 10 = 0.7.

[0094] This percentage parameter intuitively reflects the proportion of statements in the first response text generated by the system that are semantically consistent with the standard answer. A higher percentage parameter indicates that the system can provide more correct information when answering questions, and has a higher accuracy rate; conversely, a lower parameter suggests that the system may have more errors or inaccurate responses, requiring further improvement in its knowledge base, reasoning ability, or generation algorithm.

[0095] S1052. Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain the second proportion parameter of the statement in the standard statement set in the first statement set.

[0096] The second percentage parameter is used to characterize the completeness of the first response text.

[0097] Also based on semantic matching technology, this time it checks the matching status of each declaration in the standard declaration set within the first declaration set. The number of declarations in the standard declaration set that can find a semantic match in the first declaration set is counted, and then divided by the total number of declarations in the standard declaration set to obtain the second proportion parameter. For example, if the standard declaration set has 8 declarations, and 6 of them have a semantic match in the first declaration set, then the second proportion parameter is 6 / 8 = 0.75.

[0098] The second parameter measures the extent to which the system-generated response text covers the information in the standard answer. A higher parameter indicates that the system-generated response is more complete and covers most of the key information in the standard answer; while a lower parameter means that the system may have missed some important information, and the response is not comprehensive enough. It is necessary to optimize its information retrieval and integration capabilities to ensure that it can provide a more complete answer.

[0099] S1053. Based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, obtain the third proportion parameter of the statements in the first statement set in the retrieval context statement set.

[0100] The third percentage parameter is used to characterize the fidelity of the first response text.

[0101] For the first set of declarations and the context set of declarations, semantic association analysis is used to determine the degree of association between the declarations in the first set and the declarations in the context set. Methods based on knowledge graphs, semantic networks, etc., can be employed to determine whether the declarations in the first set can find reasonable semantic support in the context set. The proportion of semantically related declarations to the total number of declarations in the first set is counted to obtain the third proportion parameter. For example, if the first set has 12 declarations, and 9 of them are semantically related to declarations in the context set, then the third proportion parameter is 9 / 12 = 0.75.

[0102] This parameter reflects the degree to which the system's generated response relies on and adheres to the retrieved contextual information. A higher third percentage indicates that the system makes good use of the information provided by the retrieval context when generating the response, remaining faithful to the retrieval results and producing a more reliable response. Conversely, a lower percentage may suggest that the system does not fully utilize contextual information during the generation process, or that it generates content irrelevant to the context, potentially leading to "illusions" (where the model generates content that does not conform to reality). In such cases, improvements are needed in how the system processes and utilizes contextual information.

[0103] S1054. Based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, obtain the fourth proportion parameter of the statement in the standard statement set in the retrieval context statement set.

[0104] The fourth percentage parameter is used to characterize the quality of the first response text.

[0105] Semantic matching analysis is performed on the standard declaration set and the context declaration set. The proportion of the number of declarations in the standard declaration set that have semantically matching statements in the context declaration set is calculated to obtain the fourth proportion parameter. For example, if the standard declaration set has 10 declarations, and 8 of them have semantically related declarations in the context declaration set, then the fourth proportion parameter is 8 / 10 = 0.8.

[0106] The fourth parameter assesses the relationship between the quality of the retrieved contextual information and the standard answer from another perspective. A higher parameter indicates that the contextual information obtained by the retrieval module is highly relevant to the standard answer and can provide good support for generating the correct answer; conversely, a lower parameter may indicate that the retrieval module is insufficient in obtaining relevant information and needs to optimize the retrieval strategy to improve the role of the retrieved contextual information in generating a high-quality response.

[0107] S1055. Generate the declaration comparison result based on the first proportion parameter, the second proportion parameter, the third proportion parameter, and the fourth proportion parameter.

[0108] Based on the calculated first, second, third, and fourth percentage parameters, a claim comparison result can be constructed. These parameters, as key indicators, comprehensively reflect the first response text's accuracy, completeness, fidelity, and the quality of retrieval context information. For example, these parameters can be organized into a data structure, such as a structure or object containing four fields (corresponding to the four percentage parameters), for subsequent system performance evaluation and analysis. Furthermore, by combining the specific values ​​and trends of these parameters, a more detailed claim comparison report can be generated, including explanations of each parameter, comparative analysis with expected standards, and preliminary diagnoses of potential system performance problems, providing rich information for further in-depth evaluation and optimization of the RAG system.

[0109] By acquiring the semantics of the declarations and calculating the relevant proportion parameters to generate declaration comparison results, an effective method based on semantic understanding and quantitative analysis is provided for comprehensively and accurately evaluating the performance of RAG systems. This helps to gain a deeper understanding of the system's performance in different aspects and to make targeted improvements and optimizations.

[0110] The evaluation method for a Retrieval Enhancement Generation (RAG) system provided in this application obtains preset question-and-answer text, standard response text, first response text, and retrieval context text, and further extracts corresponding declaration sets. This achieves comprehensive utilization of multiple aspects of data from the RAG system. Furthermore, it performs a declaration-level comparison of the RAG system's output (i.e., the first response text), the standard response text corresponding to the preset question text, and the context information used by the RAG system to generate answers (i.e., the retrieval context text). This delves into the semantic level of the text, obtaining the correct declaration subset, incorrect declaration subset, missing correct declaration subset, and irrelevant declaration subset from the first declaration set. This clarifies which declarations are correct, which are incorrect, which key information is missing, and which are irrelevant. Simultaneously, based on the first performance analysis results between multiple preset question-and-answer texts and their corresponding related texts, a comprehensive performance evaluation result is generated, enabling a holistic understanding of the RAG system's performance and providing a clear direction for its optimization.

[0111] As an extension and refinement of the above embodiments, in order to analyze and calculate based on the subset of correct statements, the subset of incorrect statements, the subset of missing correct statements, and the subset of irrelevant statements to obtain a first performance analysis result, and to generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result among multiple preset question-and-answer texts, multiple first response texts, and multiple search context texts, the following steps are also required:

[0112] Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statements, the relevant search context statement subset in the search context statement set, and the number of statements in each subset of the irrelevant search context statement subset, the overall performance index set, the index set of the retrieval module, and the index set of the response generation module of the target RAG system are obtained. In this embodiment, by calculating the overall performance index set, the index set of the retrieval module, and the index set of the response generation module of the target retrieval enhancement generation RAG system, the performance of the target RAG system can be evaluated from three aspects: the overall system, retrieval, and response; thus obtaining the overall performance index set, the index set of the retrieval module, and the index set of the response generation module. This allows for a comprehensive and detailed evaluation of the RAG system's performance during the retrieval and generation processes, providing clear directions and basis for system improvement and optimization from multiple perspectives, helping to improve the overall quality and service level of the system, and better meeting user needs in different scenarios.

[0113] Specifically, let A be the subset of correct statements, B be the subset of incorrect statements, C be the subset of missing correct statements, and D be the subset of irrelevant statements; let E be the subset of relevant context statements and F be the subset of irrelevant context statements, referring to... Figure 3 As shown, the relationships between the multiple subsets obtained based on the preset question-and-answer text, the first response text, and the retrieval context text are as follows: Figure 3 As shown, the standard response text includes a subset A of correct statements and a subset C of missing correct statements; the first response text output by the target RAG system includes a subset A of correct statements and a subset B of incorrect statements; the relevant context statement subset E includes a subset A of correct statements, a subset B of incorrect statements, a subset C of missing correct statements, and a set D of irrelevant statements; the irrelevant context statement subset F includes a subset B of incorrect statements and a set D of irrelevant statements.

[0114] The method for calculating the overall performance index set of the target retrieval enhancement generation RAG system includes the following steps 1 and 3:

[0115] Step 1: Sum the number of declarations in the correct declaration subset and the incorrect declaration subset to obtain a first sum value, and obtain the ratio of the number of declarations in the correct declaration subset to the first sum value to generate an overall accuracy value.

[0116] The accuracy value represents the correctness of the target RAG system's output response, i.e., the proportion of correctly stated statements in the response. For example, in a question-and-answer scenario, higher accuracy indicates that the system can accurately understand the user's question and provide the correct answer, reducing the output of incorrect information. The method for calculating the accuracy value is as follows:

[0117]

[0118] Step 2: Sum the number of declarations in the correct declaration subset and the missing declaration subset to obtain a second sum value, and obtain the ratio of the number of declarations in the correct declaration subset to the second sum value to generate an overall recall value.

[0119] The recall value represents the completeness of the output responses from the target RAG system, specifically the proportion of standard answer statements mentioned in the responses. A high recall rate means the system can cover more key information from standard answers, providing users with comprehensive responses. The method for calculating the recall value is as follows:

[0120]

[0121] Step 3: Generate the set of overall performance indicators based on the overall precision value and the overall recall value.

[0122] Furthermore, the precision and recall values ​​can be used to determine the system's performance in terms of accuracy and completeness. Low precision may indicate insufficient knowledge or reasoning ability of the generator, requiring improvements to the generation algorithm or expansion of training data. Low recall may indicate that the retrieval module has failed to acquire sufficiently comprehensive information, necessitating optimization of the retrieval strategy or expansion of the retrieval scope to specifically improve the overall system performance.

[0123] The method for calculating the set of performance indicators for the retrieval module of the target retrieval enhancement generation RAG system includes the following steps 4 and 6:

[0124] Step 4: Sum the number of declarations in the relevant search context declaration subset and the irrelevant search context declaration subset to obtain a third sum value, and obtain the ratio of the number of declarations in the relevant search context declaration subset to the third sum value to generate a search context precision value.

[0125] Specifically, the retrieval context precision value represents the precision of the retrieval context obtained by the target RAG system, measuring the retrieval module's ability to reduce noise and obtain high-quality relevant information. High context precision reduces the time and effort users spend filtering irrelevant information, improving system efficiency. For example, in an information retrieval system, users can locate useful information faster, enhancing system usability and user experience. The method for calculating the recall value is as follows:

[0126]

[0127] Where |E|+|F| is the first sum value. It should be noted that the retrieval context precision value calculated according to the above formula is directly proportional to the numerical value; alternatively, the retrieval context precision value can be generated by calculating the ratio of the number of declarations in the subset of irrelevant retrieval context declarations to the third sum value, in which case the retrieval context precision value is inversely proportional to the numerical value.

[0128] Step 5: Sum the correct statements and missing correct statements in the relevant context statement subset to obtain a fourth sum value, and obtain the ratio of the fourth sum value to the second sum value to generate a retrieval recall value.

[0129] The retrieval recall value represents the proportion of standard response claims covered by the retrieval context retrieved by the target RAG system. A high metric indicates that the retrieval module can effectively find information related to the correct answer, laying the foundation for generating accurate responses. For example, in search engines, a high claim recall rate ensures that search results contain the key information needed by the user, improving the relevance and usefulness of search results; the method for calculating the claim recall value is as follows:

[0130]

[0131] Where |E|+|F| is the first sum value, and |A∈E|+|C∈E| is the second sum value. It should be noted that the retrieval recall value calculated according to the above formula is directly proportional to the numerical value; alternatively, a fourth sum value can be obtained by summing the correct statements and missing correct statements in the irrelevant context statement subset. The retrieval recall value is then generated by the ratio of this fourth sum value to the second sum value, in which case the retrieval recall value is inversely proportional to the numerical value.

[0132] Step 6: Generate the set of performance indicators for the retrieval module based on the retrieval context precision value and the retrieval context precision value.

[0133] Furthermore, based on the retrieval recall and retrieval context precision, the performance of the retrieval module of the target RAG system is evaluated. If the declared recall is low, the matching mechanism of the retrieval algorithm can be improved or the retrieval data source can be increased. If the context precision is low, the retrieval filter needs to be optimized to remove more irrelevant information, thereby improving the performance of the retrieval module and making the retrieved context information more accurate and useful.

[0134] The method for calculating the set of performance metrics for the response generation module of the target retrieval enhanced generation (RAG) system includes the following steps 7 and 11:

[0135] Step 7: Obtain the first value of the correct declaration in the relevant context declaration subset, and obtain the ratio of the first value to the fourth sum value to generate context utilization.

[0136] The context utilization rate represents the extent to which the response generator of the target RAG system effectively utilizes relevant information in the context. A high context utilization rate indicates that the generator is adept at using retrieved information to generate responses, making the responses more fact-based and context-aware, thus enhancing the credibility and logic of the responses. The method for calculating the context utilization rate is as follows:

[0137]

[0138] Where |A∈E| is the first value, and |A∈E|+|C∈E| is the second sum value. It should be noted that the context utilization rate calculated according to the above formula is directly proportional to the value; alternatively, another first value can be obtained from the correct declarations in the subset of irrelevant context declarations, and then the ratio of the first value to the fourth sum value can be obtained to generate the context utilization rate, in which case the context utilization rate is inversely proportional to the value.

[0139] Step 8: Obtain the second value of the erroneous statement belonging to the union of the relevant context statement subset and the unrelated context statement subset, and obtain the ratio of the second value to the second sum value to generate noise sensitivity.

[0140] The noise sensitivity refers to the degree to which the generator of the target RAG system is affected by noise in relevant and irrelevant text blocks. Low noise sensitivity indicates that the generator can resist noise interference and stably generate high-quality responses. For example, in social media information processing, the system can still generate accurate and useful responses in the face of a large amount of noisy information; therefore, the method for calculating noise sensitivity is as follows:

[0141]

[0142] Where |B∈(E∪F)| is the second value, and |A|+|C| is the third sum value.

[0143] Step 9: Obtain the third value of the error statements that do not belong to the union of the relevant context statement subset and the unrelated context statement subset, and obtain the ratio of the third value to the second sum value to generate the probability of hallucination.

[0144] The probability of hallucination occurrence is used to represent the situation where the evaluation of the target RAG system generates erroneous information that is not in context. A low hallucination rate ensures that the responses generated by the system are truthful and reliable, avoiding the generation of fictitious or misleading content. This is particularly important in fields such as medicine and finance, as it can prevent serious consequences caused by erroneous information; the method for calculating the probability of hallucination occurrence is as follows:

[0145]

[0146] in, The third value is |A|+|C|, which is the third sum value.

[0147] Step 10: Obtain the fourth value of the correct statements that do not belong to the union of the relevant context statement subset and the unrelated context statement subset, and obtain the ratio of the fourth value to the second sum value to generate its own knowledge utilization.

[0148] The self-knowledge utilization rate is used to represent the degree to which the generated response uses the model's own knowledge rather than solely relying on the retrieved context. Appropriate use of self-knowledge can supplement insufficient contextual information, improving the flexibility and comprehensiveness of the response. In some emerging fields or scenarios with rapid knowledge updates, the application of self-knowledge can enable the system to provide more cutting-edge and comprehensive answers; therefore, the method for calculating the self-knowledge utilization rate is as follows:

[0149]

[0150] in, The fourth value is |A|+|C|, and the third sum value is |A|+|C|.

[0151] Step 11: Sum the number of correctly declared statements in the irrelevant context declaration subset with the number of incorrectly declared statements in the union of the relevant context declaration subset and the irrelevant context declaration subset to obtain a fifth value, and obtain the ratio of the fifth value to the second sum value to generate fidelity.

[0152] The fidelity score itself, or knowledge utilization, represents the degree to which the generated representation uses the retrieval context. High fidelity ensures that the generated response is closely related to the retrieval context, logically consistent, and helps users understand the basis of the response, thus enhancing their trust in the system. The fidelity score is calculated using the following formula:

[0153]

[0154] Among them, among them, The fifth value is |A| + |C|, and the third sum value is |A| + |C|.

[0155] Step 12: Based on the context utilization rate, the noise sensitivity, the probability of hallucination, the utilization rate of self-knowledge, and the fidelity, generate the indicator set of the response generation module.

[0156] These metrics provide a comprehensive understanding of the generator's performance characteristics. Low context utilization allows for improvements in how the generator extracts and integrates contextual information; high noise sensitivity necessitates optimizing the generation algorithm to enhance its anti-interference capabilities; a high illusion rate requires adjusting the training strategy to reduce erroneous information generation; and by balancing the generator's knowledge and fidelity performance, the generator's knowledge application and context dependency mechanisms can be optimized, thereby continuously improving the performance of the generation module and generating higher-quality responses.

[0157] Furthermore, the target comprehensive performance evaluation result of the target RAG system is obtained through the method provided in the above embodiments.

[0158] This comprehensive evaluation provides clear direction for system optimization. Whether it's the retrieval module, the generation module, or the overall system architecture, specific improvement measures can be determined based on the evaluation results. For example, if the system frequently produces incorrect or missing correct statements when handling a specific type of problem, it may be necessary to expand or optimize the knowledge data in the relevant domain. If there are many irrelevant statements, it may be necessary to improve the filtering mechanism of the retrieval context or the way the generation module utilizes the context, thereby achieving precise optimization of the RAG system and improving system performance.

[0159] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides an evaluation device for a retrieval-enhanced RAG generation system. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the evaluation device for a retrieval-enhanced RAG generation system in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0160] This application provides an evaluation apparatus for a retrieval-enhanced generation RAG system. Figure 4 This is a schematic diagram of the evaluation device for the retrieval enhancement generation RAG system, as shown below. Figure 4 As shown, the evaluation apparatus 400 for the retrieval enhancement generation RAG system includes:

[0161] The first acquisition unit 401 is used to acquire multiple preset question and answer texts and standard response texts corresponding to each preset question and answer text.

[0162] The second acquisition unit 402 is used to input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first response text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first response text;

[0163] The third acquisition unit 403 is used to acquire the standard declaration set corresponding to the standard response text, the first declaration set corresponding to the first response text, and the search context declaration set corresponding to the search context text.

[0164] The comparison unit 404 is used to perform a semantic comparison of each statement in the standard statement set, the first statement set, and the retrieval context statement set, and generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set;

[0165] Analysis unit 405 is used to perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset to obtain a first performance analysis result, and to generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result among multiple preset question and answer texts, multiple first response texts, and multiple search context texts.

[0166] As an optional implementation of this application, the first acquisition unit 401 is specifically used to extract declarations from the standard response text, the first response text, and the search context text using a first large language model, and to obtain the standard declaration set, the first declaration set, and the search context declaration set.

[0167] As an optional implementation of this application, the comparison unit 404 is specifically used for: obtaining the semantics corresponding to each statement in the first statement set, the semantics corresponding to each statement in the standard statement set, and the semantics corresponding to each statement in the retrieval context statement set; obtaining a first proportion parameter of the statements in the first statement set in the standard statement set based on the semantics corresponding to each statement in the first statement set and the standard statement set; the first proportion parameter is used to characterize the correctness of the first response text; obtaining a second proportion parameter of the statements in the standard statement set in the first statement set based on the semantics corresponding to each statement in the first statement set and the standard statement set; the second proportion... The parameters are used to characterize the completeness of the first response text; based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, a third proportion parameter is obtained for the statements in the first statement set in the retrieval context statement set; the third proportion parameter is used to characterize the fidelity of the first response text; based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, a fourth proportion parameter is obtained for the statements in the standard statement set in the retrieval context statement set; the fourth proportion parameter is used to characterize the quality of the first response text; and the statement comparison result is generated according to the first proportion parameter, the second proportion parameter, the third proportion parameter, and the fourth proportion parameter.

[0168] As an optional implementation of this application embodiment, the analysis unit 405 is further configured to obtain the overall performance index set of the target RAG system, the index set of the retrieval module, and the index set of the response generation module based on the number of declarations in each subset of the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, the irrelevant declarations, the relevant context declaration subset in the retrieval context declaration set, and the irrelevant context declaration subset. As an optional implementation of this application embodiment, the analysis unit 405 is further configured to sum the number of declarations in the correct declaration subset and the incorrect declaration subset to obtain a first sum value, and obtain the ratio of the number of declarations in the correct declaration subset to the first sum value to generate an overall accuracy value;

[0169] The number of declarations in the correct declaration subset and the missing declaration subset are summed to obtain a second sum value, and the ratio of the number of declarations in the correct declaration subset to the second sum value is obtained to generate an overall recall value; the overall performance index set is generated based on the overall precision value and the overall recall value.

[0170] As an optional implementation of this application, the analysis unit 405 is further configured to sum the number of declarations in the relevant search context declaration subset and the irrelevant search context declaration subset to obtain a third sum value, and obtain the ratio of the number of declarations in the relevant search context declaration subset to the third sum value to generate a search context precision value; and to sum the correct declarations and missing correct declarations in the relevant context declaration subset to obtain a fourth sum value, and obtain the ratio of the fourth sum value to the second sum value to generate a search recall value;

[0171] The set of performance metrics for the retrieval module is generated based on the retrieval context precision value and the retrieval context precision value.

[0172] As an optional implementation of this application, the analysis unit 405 is further configured to: obtain a first value of the correct statements in the relevant context statement subset, and obtain the ratio of the first value to the fourth sum value to generate context utilization; obtain a second value of the incorrect statements belonging to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain the ratio of the second value to the second sum value to generate noise sensitivity; obtain a third value of the incorrect statements not belonging to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain the ratio of the third value to the second sum value to generate hallucination occurrence. The probability is calculated as follows: A fourth value is obtained for correct statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset; the ratio of the fourth value to the second sum value is obtained to generate self-knowledge utilization; a fifth value is obtained by summing the number of correct statements in the irrelevant context statement subset with the number of incorrect statements in the union of the relevant context statement subset and the irrelevant context statement subset; the ratio of the fifth value to the second sum value is obtained to generate fidelity; based on the context utilization, the noise sensitivity, the probability of hallucination, the self-knowledge utilization, and the fidelity, a set of indicators for the response generation module is generated.

[0173] Based on the same inventive concept, this disclosure also provides an electronic device. Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 5 As shown, the electronic device provided in this embodiment includes a memory 501 and a processor 502. The memory 501 is used to store a computer program; the processor 502 is used to execute the audio data processing method provided in the above embodiment when executing the computer program.

[0174] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the computing device to implement the evaluation method of the retrieval enhancement generation RAG system provided in the above embodiments.

[0175] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0176] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0177] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0178] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An evaluation method for a retrieval-enhanced generation RAG system, characterized in that, include: Obtain multiple preset question-and-answer texts and the standard response text corresponding to each preset question-and-answer text; The preset question and answer text is input into the target retrieval enhancement generation RAG system to obtain the first response text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first response text; Obtain the standard statement set corresponding to the standard response text, the first statement set corresponding to the first response text, and the search context statement set corresponding to the search context text; By combining the semantics of each statement in the standard statement set, the first statement set, and the retrieval context statement set, a statement comparison is performed to generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set; Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset, an analysis and calculation are performed to obtain a first performance analysis result. Based on the first performance analysis result among multiple preset question and answer texts, multiple first response texts, and multiple search context texts, a target comprehensive performance evaluation result of the target retrieval enhancement generation RAG system is generated.

2. The method according to claim 1, characterized in that, The step of obtaining the standard statement set corresponding to the standard response text, the first statement set corresponding to the first response text, and the search context statement set corresponding to the search context text includes: The standard response text, the first response text, and the retrieval context text are extracted using the first major language model to obtain the standard declaration set, the first declaration set, and the retrieval context declaration set.

3. The method according to claim 1, characterized in that, The step of comparing the semantics of each statement in the standard statement set, the first statement set, and the retrieval context statement set to generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set includes: Based on the semantics corresponding to each statement in the first statement set and the standard statement set, a first proportion parameter of the statements in the first statement set in the standard statement set is obtained; the first proportion parameter is used to characterize the correctness of the first response text. Based on the semantics corresponding to each statement in the first statement set and the standard statement set, a second proportion parameter is obtained for the statement in the standard statement set in the first statement set; the second proportion parameter is used to characterize the completeness of the first response text. Based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, a third proportion parameter of the statements in the first statement set in the retrieval context statement set is obtained; the third proportion parameter is used to characterize the fidelity of the first response text. Based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, a fourth proportion parameter of the statement in the standard statement set in the retrieval context statement set is obtained; the fourth proportion parameter is used to characterize the quality of the first response text. Based on the first proportion parameter, the second proportion parameter, the third proportion parameter, and the fourth proportion parameter, a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements are generated in the first set of statements.

4. The method according to claim 1, characterized in that, Before performing analysis and calculation based on the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, and the irrelevant declaration subset to obtain the first performance analysis result, the method further includes: Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statement, the relevant search context statement subset in the search context statement set, and the number of statements in each subset of the irrelevant search context statement subset, the overall performance index set of the target search enhancement generation RAG system, the index set of the search module, and the index set of the response generation module are obtained.

5. The method according to claim 4, characterized in that, The method further includes: The number of declarations in the correct declaration subset and the incorrect declaration subset are summed to obtain a first sum value, and the ratio of the number of declarations in the correct declaration subset to the first sum value is obtained to generate an overall precision value; The number of declarations in the correct declaration subset and the missing correct declaration subset are summed to obtain a second sum value, and the ratio of the number of declarations in the correct declaration subset to the second sum value is obtained to generate an overall recall value; The overall performance index set is generated based on the overall precision value and the overall recall value.

6. The method according to claim 5, characterized in that, The method further includes: The number of declarations in the relevant search context declaration subset and the irrelevant search context declaration subset are summed to obtain a third sum value, and the ratio of the number of declarations in the relevant search context declaration subset to the third sum value is obtained to generate a search context precision value; The correct statements and missing correct statements in the relevant retrieval context statement subset are summed to obtain a fourth sum value, and the ratio of the fourth sum value to the second sum value is obtained to generate a retrieval recall value; Based on the retrieval context precision value and the retrieval recall value, a set of metrics for the retrieval module is generated.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the first value of the correct declarations in the relevant search context declaration subset, and obtain the ratio of the first value to the fourth sum value to generate the search context utilization rate; Obtain a second value of the erroneous claims belonging to the union of the relevant retrieval context claim subset and the irrelevant retrieval context claim subset, and obtain the ratio of the second value to the second sum value to generate noise sensitivity; Obtain a third value of the erroneous statements that do not belong to the union of the relevant search context statement subset and the irrelevant search context statement subset, and obtain the ratio of the third value to the second sum value to generate the probability of hallucination occurrence; Obtain the fourth value of the correct statements that do not belong to the union of the relevant search context statement subset and the irrelevant search context statement subset, and obtain the ratio of the fourth value to the third sum value to generate its own knowledge utilization. The fifth value is obtained by summing the number of correctly declared statements in the irrelevant search context declaration subset with the number of incorrect statements in the union of the relevant search context declaration subset and the irrelevant search context declaration subset, and the ratio of the fifth value to the third sum value is obtained to generate the fidelity score. Based on the context utilization, noise sensitivity, probability of hallucination, utilization of self-knowledge, and fidelity, a set of indicators is generated for the response generation module.

8. An evaluation apparatus for a retrieval enhancement generation RAG system, characterized in that, include: The first acquisition unit is used to acquire multiple preset question and answer texts and the standard response text corresponding to each preset question and answer text. The second acquisition unit is used to input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first response text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first response text; The third acquisition unit is used to acquire the standard declaration set corresponding to the standard response text, the first declaration set corresponding to the first response text, and the search context declaration set corresponding to the search context text. The comparison unit is used to perform a semantic comparison of each statement in the standard statement set, the first statement set, and the retrieval context statement set, and generate a subset of correct statements, a subset of incorrect statements, a subset of missing correct statements, and a subset of irrelevant statements in the first statement set. The analysis unit is used to perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset to obtain a first performance analysis result, and to generate a target comprehensive performance evaluation result of the target retrieval enhancement generation RAG system based on the first performance analysis result among multiple preset question and answer texts, multiple first response texts, and multiple retrieval context texts.

9. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program; the processor being used to cause the electronic device to implement the evaluation method of the retrieval enhancement generation RAG system according to any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the evaluation method of the retrieval enhancement generation RAG system according to any one of claims 1-7.

Images