Evaluation method and device for RAG (Retrieval Enhanced Generation) system

Abstract

The embodiment of the invention provides an evaluation method and device for a retrieval enhancement generation (RAG) system, and relates to the technical field of data processing.The method specifically comprises the steps that a preset question and answer text is input into a target retrieval enhancement generation (RAG) system to obtain a first answer text and a retrieval context text; performing declaration comparison on a standard declaration set corresponding to the obtained standard reply text, a first declaration set corresponding to the obtained first reply text and a retrieval context declaration set corresponding to the obtained retrieval context text; and analyzing and calculating the correct declaration subset, the wrong declaration subset, the missing correct declaration subset and the irrelevant declaration subset in the generated first declaration set to obtain first performance analysis results, and generating a target comprehensive performance evaluation result of the target RAG system based on a plurality of first performance analysis results. According to the method, the RAG system can be subjected to performance evaluation from multiple aspects, and powerful support is provided for wider and more efficient application of the RAG system in the field of artificial intelligence.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an evaluation method and device for a retrieval enhancement generation RAG system. Background Art

[0002] In the field of artificial intelligence, especially with the widespread application of large language models, the Retrieval Augmented Generation (RAG) system has become a typical representative of utilizing the power of large models. By cleverly combining the retrieval and generation modules, the RAG system has shown the potential to provide richer and more accurate content in information replies. It can use the retrieval module to quickly locate relevant information from massive data, and then use the generation module to integrate this information into logical and targeted replies, providing users with more valuable services.

[0003] However, there are many challenges in comprehensively and accurately evaluating the performance of RAG systems. Traditional evaluation methods are difficult to effectively measure the quality of long text replies. Since long text replies often contain complex structures and rich information, traditional evaluation methods lack effective means to deeply analyze the accuracy, logic and completeness of the replies, and are prone to ignore the details in the long text, resulting in an incomplete and inaccurate evaluation of its quality. At the same time, traditional evaluation indicators are often based on simple text matching or surface features, and cannot fully consider the deep-level factors such as semantic understanding and logical coherence that humans pay attention to when understanding and judging information, resulting in a large deviation between the evaluation results and the actual human experience.

[0004] Therefore, it is difficult for the existing technology to conduct in-depth and detailed analysis of the RAG system, and it cannot meet the needs of comprehensive and accurate evaluation of system performance, thus limiting the further optimization and expansion of the RAG system in practical applications. Therefore, a new evaluation framework is urgently needed to overcome these problems and realize effective evaluation of RAG system performance. Summary of the invention

[0005] In view of this, the embodiments of the present application provide an evaluation method and device for a retrieval enhancement generation RAG system, which can evaluate the performance of the RAG system from multiple aspects and provide strong support for the wider and more efficient application of the RAG system in the field of artificial intelligence.

[0006] In a first aspect, an embodiment of the present application provides an evaluation method for a retrieval enhancement generation RAG system, comprising:

[0007] Obtaining a plurality of preset question and answer texts and a standard answer text corresponding to each of the preset question and answer texts;

[0008] Input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain a first answer text and a retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first answer text;

[0009] Obtaining a standard statement set corresponding to the standard reply text, a first statement set corresponding to the first reply text, and a retrieval context statement set corresponding to the retrieval context text;

[0010] Perform statement comparison based on the semantics of each statement in the standard statement set, the first statement set, and the search context statement set to generate a correct statement subset, an incorrect statement subset, a missing correct statement subset, and an irrelevant statement subset in the first statement set;

[0011] Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset and the irrelevant statement subset, analysis and calculation are performed to obtain a first performance analysis result, and based on the first performance analysis result between multiple preset question and answer texts, multiple first reply texts, and multiple retrieval context texts, a target comprehensive performance evaluation result of the target RAG system is generated.

[0012] As an optional implementation of the embodiment of the present application, the obtaining of the standard statement set corresponding to the standard reply text, the first statement set corresponding to the first reply text, and the retrieval context statement set corresponding to the retrieval context text includes:

[0013] The standard reply text, the first reply text and the search context text are subjected to statement extraction through a first large language model to obtain the standard statement set, the first statement set and the search context statement set.

[0014] As an optional implementation of the embodiment of the present application, the semantics of each statement in the standard statement set, the first statement set and the search context statement set are combined to perform statement comparison to generate a correct statement subset, an incorrect statement subset, a missing correct statement subset and an irrelevant statement subset in the first statement set. , including:

[0015] Obtaining the semantics corresponding to each statement in the first statement set, the semantics corresponding to each statement in the standard statement set, and the semantics corresponding to each statement in the search context statement set;

[0016] Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtaining a first proportion parameter of the statements in the first statement set in the standard statement set; the first proportion parameter is used to characterize the correctness of the first reply text;

[0017] Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtaining a second proportion parameter of the statements in the standard statement set in the first statement set; the second proportion parameter is used to characterize the completeness of the first reply text;

[0018] Based on the semantics corresponding to each statement in the first statement set and the search context statement set, obtaining a third proportion parameter of the statements in the first statement set in the search context statement set; the third proportion parameter is used to characterize the fidelity of the first reply text;

[0019] Based on the semantics corresponding to each statement in the standard statement set and the search context statement set, obtaining a fourth proportion parameter of the statements in the standard statement set in the search context statement set; the fourth proportion parameter is used to characterize the quality of the first reply text;

[0020] The declaration comparison result is generated according to the first proportion parameter, the second proportion parameter, the third proportion parameter, and the fourth proportion parameter.

[0021] As an optional implementation of the embodiment of the present application, before performing analysis and calculation based on the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, and the irrelevant declaration subset to obtain a first performance analysis result, the method further includes:

[0022] Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statement, the relevant previous retrieval context statement subset in the retrieval context statement set, and the number of statements in each subset of the irrelevant retrieval context statement subset, the overall performance indicator set of the target RAG system, the indicator set of the retrieval module, and the indicator set of the reply generation module are obtained. As an optional implementation of the embodiment of the present application, the method also includes:

[0023] Adding the number of statements in the correct statement subset and the number of statements in the incorrect statement subset to obtain a first sum value, and obtaining a ratio of the number of statements in the correct statement subset to the first sum value to generate an overall accuracy value;

[0024] Adding the number of statements in the correct statement subset and the missing statement subset to obtain a second sum value, and obtaining a ratio of the number of statements in the correct statement subset to the second sum value to generate an overall recall value;

[0025] The overall performance indicator set is generated based on the overall precision value and the overall recall value.

[0026] As an optional implementation of the embodiment of the present application, the method further includes:

[0027] Adding the number of statements in the relevant retrieval context statement subset and the number of statements in the irrelevant retrieval context statement subset to obtain a third total value, and obtaining a ratio of the number of statements in the relevant retrieval context statement subset to the third total value to generate a retrieval context precision value;

[0028] Adding the correct statements and the missing correct statements in the relevant context statement subset to obtain a fourth sum value, and obtaining a ratio of the fourth sum value to the second sum value to generate a retrieval context precision value;

[0029] The retrieval module performance indicator set is generated based on the retrieval context precision value and the retrieval context accuracy value.

[0030] As an optional implementation of the embodiment of the present application, the method further includes:

[0031] Obtaining a first value of the correct statement in the relevant context statement subset, and obtaining a ratio of the first value to the fourth sum value to generate a context utilization rate;

[0032] Obtaining a second value of the erroneous statements in a union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the second value to the second sum value to generate a noise sensitivity;

[0033] Obtaining a third value of erroneous statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the third value to the second sum value to generate a hallucination occurrence probability;

[0034] Obtaining a fourth value of a correct statement that does not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the fourth value to the second sum value to generate own knowledge utilization;

[0035] Adding the number of correct statements in the irrelevant context statement subset and the number of incorrect statements in the union of the relevant context statement subset and the irrelevant context statement subset to obtain a fifth value, and obtaining a ratio of the fifth value to the second sum value to generate fidelity;

[0036] Based on the context utilization, the noise sensitivity, the probability of hallucination occurrence, the own knowledge utilization, and the fidelity, a set of indicators for the answer generation module is generated.

[0037] In a second aspect, an embodiment of the present application provides an evaluation device for a retrieval enhancement generation RAG system, including:

[0038] A first acquisition unit is used to acquire a plurality of preset question and answer texts and a standard reply text corresponding to each of the preset question and answer texts;

[0039] The second acquisition unit is used to input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first answer text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first answer text;

[0040] A third acquisition unit is used to acquire a standard statement set corresponding to the standard reply text, a first statement set corresponding to the first reply text, and a retrieval context statement set corresponding to the retrieval context text;

[0041] a comparison unit, configured to perform statement comparison based on the semantics of each statement in the standard statement set, the first statement set, and the search context statement set, and generate a correct statement subset, an incorrect statement subset, a missing correct statement subset, and an irrelevant statement subset in the first statement set;

[0042] An analysis unit is used to perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset and the irrelevant statement subset to obtain a first performance analysis result, and generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result between multiple preset question and answer texts, multiple first reply texts, and multiple retrieval context texts.

[0043] As an optional implementation of the embodiment of the present application, the acquisition unit is specifically used to perform statement extraction on the standard reply text, the first reply text and the retrieval context text through a first large language model to obtain the standard statement set, the first statement set, and the retrieval context statement set.

[0044] As an optional implementation manner of the embodiment of the present application, the comparison unit is specifically used to: obtain the semantics corresponding to each statement in the first statement set, the semantics corresponding to each statement in the standard statement set, and the semantics corresponding to each statement in the retrieval context statement set; based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain a first proportion parameter of the statements in the first statement set in the standard statement set; the first proportion parameter is used to characterize the correctness of the first reply text; based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain a second proportion parameter of the statements in the standard statement set in the first statement set; the second proportion parameter The number is used to characterize the integrity of the first reply text; based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, the third proportion parameter of the statements in the first statement set in the retrieval context statement set is obtained; the third proportion parameter is used to characterize the fidelity of the first reply text; based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, the fourth proportion parameter of the statements in the standard statement set in the retrieval context statement set is obtained; the fourth proportion parameter is used to characterize the quality of the first reply text; according to the first proportion parameter, the second proportion parameter, the third proportion parameter and the fourth proportion parameter, the statement comparison result is generated.

[0045] As an optional implementation of the embodiment of the present application, the analysis unit is further used to obtain the overall performance indicator set of the target RAG system, the indicator set of the retrieval module, and the indicator set of the reply generation module based on the number of statements in each of the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statement, the relevant upper retrieval context statement subset in the retrieval context statement set, and the irrelevant retrieval context statement subset. As an optional implementation of the embodiment of the present application, the analysis unit is further used to add the number of statements in the correct statement subset and the incorrect statement subset to obtain a first sum value, and obtain the ratio of the number of statements in the correct statement subset to the first sum value to generate an overall accuracy value;

[0046] Add the number of statements in the correct statement subset and the missing statement subset to obtain a second total value, and obtain the ratio of the number of statements in the correct statement subset to the second total value to generate an overall recall value; generate the overall performance indicator set based on the overall precision value and the overall recall value.

[0047] As an optional implementation of the embodiment of the present application, the analysis unit is also used to add the number of statements in the relevant retrieval context statement subset and the irrelevant retrieval context statement subset to obtain a third sum value, and obtain the ratio of the number of statements in the relevant retrieval context statement subset to the third sum value to generate a retrieval context precision value; add the correct statements and the missing correct statements in the relevant context statement subset to obtain a fourth sum value, and obtain the ratio of the fourth sum value to the second sum value to generate a retrieval recall value;

[0048] The retrieval module performance indicator set is generated based on the retrieval context precision value and the retrieval context accuracy value.

[0049] As an optional implementation of the embodiment of the present application, the analysis unit is also used to obtain a first value of the correct statement in the relevant context statement subset, and obtain a ratio of the first value to the fourth total value to generate a context utilization rate; obtain a second value of the erroneous statement in the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the second value to the second total value to generate noise sensitivity; obtain a third value of the erroneous statement that does not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the third value to the second total value to generate a probability of hallucination occurrence. ; Obtain a fourth value of correct statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain the ratio of the fourth value to the second total value to generate own knowledge utilization; Add the number of correct statements in the irrelevant context statement subset and the number of incorrect statements in the relevant context statement subset and the union of the irrelevant context statement subset to obtain a fifth value, and obtain the ratio of the fifth value to the second total value to generate fidelity; Based on the context utilization, the noise sensitivity, the probability of hallucination, the own knowledge utilization, and the fidelity, generate a set of indicators for the answer generation module.

[0050] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the evaluation method of the retrieval enhancement generation RAG system described in any of the above embodiments when executing the computer program.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the evaluation method of the retrieval enhancement generation RAG system described in any of the above embodiments.

[0052] The evaluation method for the retrieval-enhanced generation of RAG system provided in the embodiment of the present application realizes the comprehensive utilization of various data of the RAG system by obtaining the preset question and answer text, the standard reply text, the first reply text and the retrieval context text, and further extracting the corresponding statement set, and further compares the output of the RAG system (i.e., the first reply text), the standard reply text corresponding to the preset question text, and the context information (i.e., the retrieval context text) on which the RAG system relies when generating the answer at the statement level, so as to go deep into the semantic level of the text, obtain the correct statement subset, the wrong statement subset, the missing correct statement subset and the irrelevant statement subset in the first statement set; and then clarify which statements are correct, which are wrong, which key information is missing, and which are irrelevant content. At the same time, the target comprehensive performance evaluation result is generated based on the first performance analysis result between multiple preset question and answer texts and their corresponding related texts, which can grasp the performance status of the RAG system as a whole and provide a clear direction for the optimization of the RAG system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 One of the step flow charts of the evaluation method of the retrieval enhancement generation RAG system provided in the embodiment of the present application;

[0056] Figure 2 The second step flow chart of the evaluation method of the retrieval enhancement generation RAG system provided in the embodiment of the present application;

[0057] Figure 3 A schematic diagram of the declaration relationship of the evaluation method of the retrieval enhancement generation RAG system provided in an embodiment of the present application;

[0058] Figure 4A schematic diagram of the structure of an evaluation device of a retrieval enhancement generation RAG system provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0062] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0063] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0064] The present application embodiment provides an evaluation method for a retrieval enhancement generation RAG system, referring to Figure 1 As shown, the evaluation method of the retrieval enhancement generation RAG system includes the following steps S101-S105:

[0065] S101. Obtain a plurality of preset question and answer texts and a standard reply text corresponding to each of the preset question and answer texts.

[0066] In an embodiment of the present application, the multiple preset question and answer texts and the standard reply text corresponding to each of the preset question and answer texts can be pre-set by the staff to build basic data resources for subsequent evaluation. The obtained preset question and answer texts and standard reply texts are the key basis for determining whether the performance of the target retrieval enhancement generation RAG system is qualified or not; it should be noted that the selection of these preset texts must follow the principles of comprehensiveness and representativeness, and must cover various scenarios in which the RAG system may be applied, such as different professional fields, different language styles, and problems of varying complexity, etc., to ensure that the evaluation can accurately reflect the system's capability boundaries in actual use, and provide a reliable reference benchmark for the subsequent precise evaluation of system performance.

[0067] Among them, the Retrieval-Augmented Generation (RAG) system is a technical architecture that combines retrieval and generation modules in the field of artificial intelligence. It plays an important role in large-scale language model applications. It aims to solve the limitations of traditional generation models and improve the quality and accuracy of generated content by introducing external knowledge sources. In practical applications, when a user asks a question, the RAG system first uses the retrieval module to find information related to the question from a large amount of text data (such as document libraries, knowledge bases, etc.). The retrieved information constitutes the retrieval context. Subsequently, the generation module generates answers based on the retrieved context and its own language generation capabilities. For example, in an intelligent customer service scenario, when a customer asks a question about the use of a product, the RAG system can quickly retrieve the content of the relevant product manual, and combine the language model to generate accurate and detailed answers to provide users with more targeted services.

[0068] Exemplarily, when evaluating the target retrieval enhancement generation RAG system, 10 preset question and answer texts and standard reply texts corresponding to the 10 preset questions and answers can be set, and then the target retrieval enhancement generation RAG system is allowed to retrieve the 10 preset reply texts to generate corresponding reply texts, and then use these 10 reply texts to compare with the corresponding standard reply texts in subsequent steps to evaluate the performance of the target retrieval enhancement generation RAG system.

[0069] S102. Input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first reply text and the retrieval context text.

[0070] Among them, the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first reply text.

[0071] In some embodiments, the inputting of the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first reply text and the retrieval context text is to simulate the actual operation process of the current RAG system. The first reply text obtained after the preset question and answer text is input into the target RAG system is the direct output result of the target RAG system under the current input; the retrieval context text records the knowledge source information that the target RAG system relies on when generating the reply. Both are crucial for in-depth analysis of the retrieval and generation mechanism of the system, and can truly show how the system uses its own retrieval capabilities to obtain information and generate answers when dealing with problems, providing direct data support for the subsequent evaluation of system performance from multiple dimensions.

[0072] S103, obtaining a standard statement set corresponding to the standard reply text, a first statement set corresponding to the first reply text, and a retrieval context statement set corresponding to the retrieval context text.

[0073] It should be noted that a statement is a basic information unit with independent semantics and judgment value that is decomposed from a complex text. It can be a statement, an opinion, a factual description, etc., which can independently express a certain meaning and its accuracy can be verified. For example, in a text about a historical event, "a certain battle took place in a specific year" or "a certain historical figure has a specific identity" can all be regarded as a statement.

[0074] In an embodiment of the present application, for a preset question and answer text, after obtaining its corresponding standard reply text, the first reply text output by the target RAG system, and the retrieved search context text, statement extraction is performed on these three texts respectively to obtain the standard statement set corresponding to the standard reply text, the first statement set corresponding to the first reply text, and the search context statement set corresponding to the search context text; this is because the statement is an information fragment with independent semantics and judgment value in the text, and by performing statement extraction on the first reply text, the standard reply text, and the search context text, the complex text is converted into a statement set form that can be quantified and compared. This enables the subsequent comparative analysis between multiple statements to more deeply explore the characteristics and problems of the system in the information processing and generation process.

[0075] Specifically, in the application embodiment, the first large language model can be used to perform statement extraction on the standard reply text, the first reply text and the retrieval context text to obtain the standard statement set, the first statement set, and the retrieval context statement set.

[0076] Among them, the Large Language Model (LLM) can perform semantic analysis on the input text with its powerful language understanding and processing capabilities. For the first reply text, standard reply text and search context text, LLM can identify information fragments with independent semantics and judgment value and extract them as statements. For example, for a reply text containing multiple sentences, LLM can extract each sentence or phrase that can express an independent opinion or fact as a statement based on factors such as sentence structure and semantic logic, thereby forming a corresponding statement set.

[0077] Furthermore, the first reply text, the standard reply text and the search context text are respectively subjected to statement extraction through the first large language model to obtain the first statement set, the standard statement set and the search context statement set.

[0078] S104, performing statement comparison based on the semantics of each statement in the standard statement set, the first statement set, and the search context statement set, and generating a correct statement subset, an incorrect statement subset, a missing correct statement subset, and an irrelevant statement subset in the first statement set.

[0079] It can be understood that the statement comparison using the first statement set, the standard statement set, and the retrieval context statement set is to make a semantic comparison between the first reply text, the standard reply text, and the retrieval context text. Through this comparison, the correctness, completeness, fidelity and other aspects of the reply generated by the target RAG system can be obtained. At the same time, the quality of the context information obtained by the retrieval module and the use of the retrieval information by the generator can also be evaluated.

[0080] Among them, the correct statement subset is the statements in the first statement set that hit the standard statement set; the incorrect statement subset includes the statements in the first statement set that are semantically inconsistent with the statements in the standard statement set; the missing correct statement subset is the statements in the first statement set that are not missing from the standard statement set; the irrelevant statement subset is the statements in the first statement set corresponding to the first reply text that have no semantic connection with any statement in the retrieval context statement set.

[0081] Specifically, the correct statement subset is obtained as follows: for each statement in the first statement set, it is precisely matched or semantic similarity is calculated with the statements in the standard statement set (algorithms based on word vectors, semantic understanding and other technologies may be used); when a statement in the first statement set is highly consistent with a statement in the standard statement set in terms of semantics and logic (for example, the degree of match in terms of meaning, theme, key information, etc. reaches a set threshold), the statement is judged to be a correct statement and is added to the correct statement subset.

[0082] The method for obtaining the incorrect statement subset is: for each statement in the first statement set, it is precisely matched with the statements in the standard statement set or the semantic similarity is calculated (algorithms based on word vectors, semantic understanding and other technologies can be used); when a statement in the first statement set and a statement in the standard statement set have obvious differences in semantics and logic, such as inconsistent key information, contradictory semantics, or inability to establish a reasonable logical association, then the statement is determined to be an incorrect statement and is added to the incorrect statement subset; through semantic analysis, it can be seen that the core content expressed by the two is consistent, so the statement in the first statement set belongs to the correct statement subset; if the first statement set contains "the moon is the central celestial body of the solar system", it is obviously contrary to the standard statement and belongs to the incorrect statement subset.

[0083] The missing correct statement subset is obtained by checking each statement in the standard statement set one by one to see whether it has a corresponding statement in the first statement set. If there is a statement in the standard statement set and there is no matching or similar statement in the first statement set (which can also be determined based on the semantic similarity algorithm), then this statement constitutes the missing correct statement subset.

[0084] The irrelevant statement subset is obtained by comparing each statement in the first statement set with the statements in the search context statement set. If a statement in the first statement set cannot find any statement associated with it in the search context statement set (whether directly or indirectly associated, which can be determined according to a pre-set association rule, such as based on topic relevance, semantic coherence, etc.), then the statement is regarded as an irrelevant statement and added to the irrelevant statement subset.

[0085] S105. Perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset to obtain a first performance analysis result, and generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result between multiple preset question and answer texts, multiple first reply texts, and multiple retrieval context texts.

[0086] In an embodiment of the present application, the first performance analysis result may include the performance of the target RAG system under the preset question text, and may include various parameters and indicators that can reflect the correctness, completeness, fidelity and retrieval context information quality of the first reply text, so as to more clearly demonstrate the performance of the target RAG system in a quantitative manner.

[0087] Then, by integrating the statement comparison results of multiple preset question and answer texts and the corresponding first reply texts, the system's performance in different problem scenarios is comprehensively considered, thereby generating the target comprehensive performance evaluation results of the target RAG system. This result can intuitively present the level of the system in key performance indicators such as accuracy, completeness, and reliability, and also provides a key basis for the improvement direction of the RAG system and the judgment of its actual application value, which is the core output link of the entire evaluation process.

[0088] It should be noted that the system is fully tested and evaluated by using multiple different preset question and answer texts, so that the evaluation results can better reflect the performance of the system in actual applications. Whether it is simple and common questions or complex and professional questions, the system can be fully tested and evaluated.

[0089] Further, after obtaining the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, and the irrelevant declaration subset, the specific implementation method of performing analysis and calculation based on the correct declaration subset, the incorrect declaration subset, the missing correct declaration subset, and the irrelevant declaration subset to obtain the first performance analysis result includes the following steps S1051 to S1055:

[0090] S1051. Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain a first proportion parameter of the statements in the first statement set in the standard statement set.

[0091] The first proportion parameter is used to characterize the correctness of the first reply text.

[0092] Specifically, the semantic understanding model or algorithm in natural language processing technology can be used to obtain the semantics corresponding to each statement, so as to obtain subsequent parameters through the semantics of the statement. The textual statement is converted into a semantic form that can be processed and compared by a computer, laying the foundation for the subsequent calculation and comparison of the proportion parameters.

[0093] Semantically match each statement in the first statement set with the statements in the standard statement set. A semantic similarity calculation algorithm, such as cosine similarity, edit distance and other methods (calculated in the semantic vector space), can be used to determine the degree of similarity between the statements in the first statement set and the statements in the standard statement set. If the similarity reaches a certain threshold (which can be set according to the specific application scenario and requirements), the two are considered to match. Then, the ratio of the number of matched statements to the total number of statements in the first statement set is counted to obtain the first proportion parameter. For example, there are 10 statements in the first statement set. After semantic matching, 7 statements have a high similarity with the statements in the standard statement set, then the first proportion parameter is 7 / 10=0.7.

[0094] This ratio parameter intuitively reflects the proportion of statements in the first reply text generated by the system that are semantically consistent with the standard answer. A higher first ratio parameter indicates that the system can provide more correct information when answering questions, and the accuracy is higher; conversely, a lower parameter indicates that the system may have more errors or inaccurate reply content, and needs to further improve its knowledge reserve, reasoning ability or generation algorithm.

[0095] S1052: Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain a second proportion parameter of the statements in the standard statement set in the first statement set.

[0096] The second proportion parameter is used to characterize the completeness of the first reply text.

[0097] Based on the same semantic matching technology, this time, each statement in the standard statement set is checked for matching in the first statement set. The number of statements in the standard statement set that can find semantic matching in the first statement set is counted, and then divided by the total number of statements in the standard statement set to obtain the second proportion parameter. For example, if the standard statement set has 8 statements, 6 of which have semantic matching statements in the first statement set, then the second proportion parameter is 6 / 8=0.75.

[0098] The second parameter measures the coverage of the standard answer information by the system-generated reply text. If this parameter is high, it means that the reply generated by the system is relatively complete and can cover most of the key information in the standard answer; while a low parameter means that the system may have missed some important information and the reply is not comprehensive enough, and it needs to optimize its information retrieval and integration capabilities to ensure that it can provide a more complete answer.

[0099] S1053. Based on the semantics corresponding to each statement in the first statement set and the search context statement set, obtain a third proportion parameter of the statements in the first statement set in the search context statement set.

[0100] The third ratio parameter is used to characterize the fidelity of the first reply text.

[0101] For the first statement set and the context statement set, the semantic association analysis is used to determine the degree of association between the statements in the first statement set and the statements in the context statement set. Methods based on knowledge graphs, semantic networks and other technologies can be used to determine whether the statements in the first statement set can find reasonable semantic support in the context statement set. The proportion of the number of statements with semantic associations to the total number of statements in the first statement set is counted to obtain the third proportion parameter. For example, the first statement set has 12 statements, of which 9 statements are semantically associated with the statements in the context statement set, then the third proportion parameter is 9 / 12=0.75.

[0102] This parameter reflects the degree to which the system-generated responses rely on and follow the retrieved context information. A higher third-ratio parameter indicates that the system makes better use of the information provided by the retrieval context when generating responses, is faithful to the retrieval results, and generates more reliable responses; whereas a lower parameter may imply that the system does not fully utilize context information during the generation process, or generates content that is irrelevant to the context, which is prone to "hallucinations" (hallucinations refer to content generated by the model that does not conform to the facts), and needs to improve its processing and utilization of context information.

[0103] S1054: Based on the semantics corresponding to each statement in the standard statement set and the search context statement set, obtain a fourth proportion parameter of the statements in the standard statement set in the search context statement set.

[0104] The fourth proportion parameter is used to characterize the quality of the first reply text.

[0105] Perform semantic matching analysis on the standard statement set and the context statement set, calculate the ratio of the number of semantically matching statements in the standard statement set to the total number of statements in the standard statement set, and obtain the fourth proportion parameter. For example, if the standard statement set has 10 statements, 8 of which have semantically related statements in the context statement set, then the fourth proportion parameter is 8 / 10=0.8.

[0106] The fourth ratio parameter evaluates the relationship between the quality of the retrieved context information and the standard answer from another perspective. If this parameter is high, it means that the context information obtained by the retrieval module is highly correlated with the standard answer and can provide good support for generating the correct answer; conversely, a low parameter may indicate that the retrieval module is insufficient in obtaining relevant information and needs to optimize the retrieval strategy to improve the help of the retrieved context information in generating high-quality answers.

[0107] S1055. Generate the declaration comparison result according to the first proportion parameter, the second proportion parameter, the third proportion parameter, and the fourth proportion parameter.

[0108] According to the calculated first proportion parameter, second proportion parameter, third proportion parameter and fourth proportion parameter, the statement comparison result can be constructed. As key indicators, these parameters can comprehensively reflect the correctness, completeness, fidelity and quality of the retrieval context information of the first reply text. For example, these parameters can be organized into a data structure, such as a structure or object containing four fields (corresponding to four proportion parameters), for subsequent use in system performance evaluation and analysis. At the same time, a more detailed statement comparison report can be generated in combination with the specific values ​​and change trends of these parameters, including an explanation of each parameter, a comparative analysis with the expected standard, and a preliminary diagnosis of possible problems in system performance, etc., to provide rich information basis for further in-depth evaluation and optimization of the RAG system.

[0109] By obtaining the declaration semantics and calculating the relevant proportion parameters to generate the declaration comparison results, an effective method based on semantic understanding and quantitative analysis is provided to comprehensively and accurately evaluate the performance of the RAG system, which helps to gain a deep understanding of the performance of the system in different aspects and make targeted improvements and optimizations.

[0110] The evaluation method for the retrieval-enhanced generation of RAG system provided in the embodiment of the present application realizes the comprehensive utilization of various data of the RAG system by obtaining the preset question and answer text, the standard reply text, the first reply text and the retrieval context text, and further extracting the corresponding statement set, and further compares the output of the RAG system (i.e., the first reply text), the standard reply text corresponding to the preset question text, and the context information (i.e., the retrieval context text) on which the RAG system relies when generating the answer at the statement level, so as to go deep into the semantic level of the text, obtain the correct statement subset, the wrong statement subset, the missing correct statement subset and the irrelevant statement subset in the first statement set; and then clarify which statements are correct, which are wrong, which key information is missing, and which are irrelevant content. At the same time, the target comprehensive performance evaluation result is generated based on the first performance analysis result between multiple preset question and answer texts and their corresponding related texts, which can grasp the performance status of the RAG system as a whole and provide a clear direction for the optimization of the RAG system.

[0111] As an extension and refinement of the above embodiment, in the step of performing analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset to obtain a first performance analysis result, and generating a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result between the plurality of preset question and answer texts, the plurality of the first reply texts, and the plurality of the retrieval context texts, the following steps need to be performed:

[0112] Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statement, the relevant previous and next statement subset in the retrieval context statement set, and the number of statements in each subset of the irrelevant retrieval context statement subset, the overall performance indicator set of the target RAG system, the indicator set of the retrieval module, and the indicator set of the reply generation module are obtained. In an embodiment of the present application, the overall performance indicator set of the target retrieval enhancement generation RAG system, the indicator set of the retrieval module, and the indicator set of the reply generation module can be calculated to perform corresponding indicator calculations on the performance of the target RAG system from the three aspects of the system as a whole, retrieval, and reply; to obtain the overall performance indicator set, the indicator set of the retrieval module, and the indicator set of the reply generation module. In this way, the performance of the RAG system in the retrieval and generation process can be comprehensively and meticulously evaluated, providing clear directions and basis for the improvement and optimization of the system from multiple perspectives, which is helpful to improve the overall quality and service level of the system and better meet the needs of users in different scenarios.

[0113] Specifically, let the correct statement subset be set A, the incorrect statement subset be set B, the missing correct statement subset be set C, and the irrelevant statement be set D; let the relevant context statement subset be set E, and the irrelevant context statement subset be set F. Figure 3 As shown, the relationship between the above-mentioned multiple subsets obtained according to the preset question and answer text, the first reply text, and the search context text is as follows Figure 3 As shown, the standard reply text includes a correct statement subset A and a missing correct statement subset C, and the first reply text output by the target RAG system includes a correct statement subset A and an incorrect statement subset B; the relevant upper retrieval context statement subset E includes a correct statement subset as set A, an incorrect statement subset as set B, a missing correct statement subset as set C, and an irrelevant statement as set D; the irrelevant retrieval context statement subset is a set F, including an incorrect statement subset as set B and an irrelevant statement as set D.

[0114] The method for calculating the overall performance index set of the target retrieval enhancement generation RAG system includes the following steps 1 and 3:

[0115] Step 1: Add the number of declarations in the correct declaration subset and the number of declarations in the incorrect declaration subset to obtain a first total value, and obtain a ratio of the number of declarations in the correct declaration subset to the first total value to generate an overall accuracy value.

[0116] The precision value is used to indicate the correctness of the response output by the target RAG system, that is, the proportion of correct statements in the response. For example, in a question-and-answer scenario, if the precision is high, it means that the system can accurately understand the user's question and provide the correct answer, reducing the output of erroneous information; the method for calculating the precision value is referred to the following formula:

[0117]

[0118] Step 2: Add the number of statements in the correct statement subset and the missing statement subset to obtain a second total value, and obtain the ratio of the number of statements in the correct statement subset to the second total value to generate an overall recall value.

[0119] The recall value is used to indicate the completeness of the response output by the target RAG system, that is, the proportion of standard answer statements mentioned in the response. A high recall rate means that the system can cover more key information in the standard answer and provide users with a comprehensive answer; the method for calculating the recall value is referred to the following formula:

[0120]

[0121] Step 3: Generate the overall performance indicator set based on the overall precision value and the overall recall value.

[0122] Furthermore, the accuracy and completeness of the system can be determined through the precision and recall values. If the precision is low, it may be that the knowledge reserve or reasoning ability of the generator is insufficient, and the generation algorithm needs to be improved or the training data needs to be expanded; if the recall is low, it may be that the retrieval module fails to obtain sufficient and comprehensive information, and the retrieval strategy needs to be optimized or the retrieval scope needs to be expanded, so as to improve the overall performance of the system in a targeted manner.

[0123] The method for calculating the target retrieval enhancement generation RAG system retrieval module performance indicator set includes the following steps 4 and 6:

[0124] Step 4: Add the number of statements in the relevant retrieval context statement subset and the number of statements in the irrelevant retrieval context statement subset to obtain a third total value, and obtain the ratio of the number of statements in the relevant retrieval context statement subset to the third total value to generate a retrieval context precision value.

[0125] Specifically, the retrieval context accuracy value is used to represent the accuracy of the retrieval context retrieved by the target RAG system, so as to measure the ability of the retrieval module to reduce noise and obtain high-quality relevant information. High context accuracy can reduce the time and energy of users to screen irrelevant information and improve system efficiency. For example, in an information retrieval system, users can locate useful information more quickly, enhancing the usability and user experience of the system. The method for calculating the recall value is referred to the following formula:

[0126]

[0127] Among them, |E|+|F| is the first sum value. It should be noted that the retrieval context precision value calculated according to the above formula is proportional to the numerical value; the retrieval context precision value can also be generated by calculating the ratio of the number of statements in the irrelevant retrieval context statement subset to the third sum value, and the retrieval context precision value is inversely proportional to the numerical value.

[0128] Step 5: Add the correct statements and the missing correct statements in the relevant context statement subset to obtain a fourth sum value, and obtain a ratio of the fourth sum value to the second sum value to generate a retrieval recall value.

[0129] The retrieval recall value is used to indicate the proportion of standard answer statements covered by the retrieval context retrieved by the target RAG system. A high indicator indicates that the retrieval module can effectively find information related to the correct answer, laying the foundation for generating accurate responses. For example, in a search engine, a high statement recall rate ensures that search results contain the key information required by users, improving the relevance and usefulness of search results; the method for calculating the statement recall value is referred to the following formula:

[0130]

[0131] Among them, |E|+|F| is the first sum value, and |A∈E|+|C∈E| is the second sum value. It should be noted that the retrieval recall value calculated according to the above formula is proportional to the numerical value; it is also possible to add the correct statements and the missing correct statements in the irrelevant context statement subset to obtain another fourth sum value, and generate a retrieval recall value by the ratio of the fourth sum value to the second sum value, and the retrieval recall value is inversely proportional to the numerical value.

[0132] Step 6: Generate the retrieval module performance indicator set based on the retrieval context accuracy value and the retrieval context precision value.

[0133] Furthermore, the performance of the retrieval module of the target RAG system is further evaluated based on the retrieval recall value and the retrieval context precision value. If the statement recall is low, it is possible to consider improving the matching mechanism of the retrieval algorithm or increasing the retrieval data source. If the context precision is low, it is necessary to optimize the retrieval filter and remove more irrelevant information, thereby improving the performance of the retrieval module and making the retrieved context information more accurate and useful.

[0134] The method for calculating the performance indicator set of the answer generation module of the target retrieval enhancement generation RAG system includes the following steps 7 and 11:

[0135] Step 7: Obtain a first value of the correct statement in the relevant context statement subset, and obtain a ratio of the first value to the fourth sum value to generate a context utilization rate.

[0136] The context utilization rate is used to indicate the degree to which the answer generator of the target RAG system effectively uses the relevant information in the context. A high context utilization rate indicates that the generator is good at using the retrieved information to generate a reply, making the reply more fact-based and context-based, and enhancing the credibility and logic of the reply; the method for calculating the context utilization rate is referred to the following formula:

[0137]

[0138] Among them, |A∈E| is the first value, and |A∈E|+|C∈E| is the second sum value. It should be noted that the context utilization calculated according to the above formula is proportional to the value; another first value of the correct statement in the irrelevant context statement subset can also be obtained, and then the ratio of the first value to the fourth sum value is obtained to generate the context utilization, and the context utilization is inversely proportional to the value.

[0139] Step 8: Obtain a second value of the erroneous statement in the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the second value to the second sum value to generate noise sensitivity.

[0140] The noise sensitivity is used to indicate the degree to which the target RAG system's embodiment generator is affected by noise in relevant and irrelevant text blocks. Low noise sensitivity indicates that the generator can resist noise interference and stably generate high-quality responses. For example, in social media information processing, in the face of a large amount of noisy information, the system can still generate accurate and useful responses; the method for calculating noise sensitivity is referred to the following formula:

[0141]

[0142] Among them, |B∈(E∪F)| is the second value, and |A|+|C| is the third sum value.

[0143] Step 9: Obtain a third value of erroneous statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the third value to the second sum value to generate a probability of hallucination occurrence.

[0144] The hallucination probability is used to represent the situation of erroneous information that is not in context generated by the evaluation of the target RAG system. A low hallucination rate ensures that the responses generated by the system are authentic and reliable, avoiding fictitious or misleading content. This is particularly important in the fields of medicine and finance to prevent serious consequences caused by erroneous information; the method for calculating the hallucination probability is referred to the following formula:

[0145]

[0146] in, is the third numerical value, and |A|+|C| is the third sum value.

[0147] Step 10: Obtain a fourth value of the correct statement that does not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the fourth value to the second total value to generate self-knowledge utilization.

[0148] The self-knowledge utilization is used to indicate the degree to which the evaluation generator uses the model's own knowledge rather than relying solely on the retrieval context content. Reasonable use of self-knowledge can supplement the lack of context information and improve the flexibility and comprehensiveness of the response. In some emerging fields or scenarios where knowledge is updated quickly, the use of self-knowledge can enable the system to provide more cutting-edge and comprehensive answers; the method for calculating the self-knowledge utilization is referred to the following formula:

[0149]

[0150] in, is the fourth value, and |A|+|C| is the third sum value.

[0151] Step 11: Add the number of correct statements in the irrelevant context statement subset and the number of incorrect statements in the union of the relevant context statement subset and the irrelevant context statement subset to obtain a fifth value, and obtain the ratio of the fifth value to the second total value to generate fidelity.

[0152] The fidelity self-knowledge utilization is used to indicate the degree to which the generated representation generator uses the retrieval context. High fidelity ensures that the generated response is closely related to the retrieval context and is logical, which helps users understand the basis of the response and enhances their trust in the system. The method for calculating fidelity is referred to the following formula:

[0153]

[0154] Among them, among them, is the fifth value, and |A|+|C| is the third sum value.

[0155] Step 12: Generate a set of indicators for the answer generation module based on the context utilization, the noise sensitivity, the probability of hallucination, the own knowledge utilization, and the fidelity.

[0156] Through these indicators, we can fully understand the performance characteristics of the generator. If the context utilization rate is low, we can improve the generator's extraction and integration of context information; if the noise sensitivity is high, we need to optimize the generation algorithm to enhance the anti-interference ability; if the hallucination rate is high, we need to adjust the training strategy to reduce the generation of erroneous information; according to the performance of its own knowledge and fidelity, we can balance the relationship between the two, optimize the generator's knowledge application and context dependency mechanism, so as to continuously improve the performance of the generation module and generate better quality reply content.

[0157] Then, the target comprehensive performance evaluation result of the target RAG system is obtained by the method provided in the above embodiment.

[0158] This comprehensive evaluation result provides a clear direction for system optimization. Whether it is the retrieval module, the generation module or the overall system architecture, specific improvement measures can be determined based on the evaluation results. For example, if it is found that the system often makes incorrect statements or lacks correct statements when dealing with a certain type of problem, it may be necessary to expand or optimize the knowledge data in the relevant field in a targeted manner; if there are many irrelevant statements, it may be necessary to improve the screening mechanism of the retrieval context or the way the generation module uses the context, so as to achieve accurate optimization of the RAG system and improve system performance.

[0159] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides an evaluation device for a retrieval enhancement RAG generation system. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that an evaluation device for a retrieval enhancement RAG generation system in this embodiment can correspond to all the contents of the aforementioned method embodiment.

[0160] The embodiment of the present application provides an evaluation device for a retrieval enhancement generation RAG system. Figure 4 A schematic diagram of the structure of the evaluation device for generating the RAG system for the retrieval enhancement is shown in FIG. Figure 4 As shown, the evaluation device 400 of the retrieval enhancement generation RAG system includes:

[0161] The first acquisition unit 401 is used to acquire a plurality of preset question and answer texts and a standard reply text corresponding to each of the preset question and answer texts;

[0162] The second acquisition unit 402 is used to input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain a first answer text and a retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first answer text;

[0163] A third acquisition unit 403 is used to acquire a standard statement set corresponding to the standard reply text, a first statement set corresponding to the first reply text, and a search context statement set corresponding to the search context text;

[0164] A comparison unit 404 is used to compare the semantics of each statement in the standard statement set, the first statement set and the search context statement set, and generate a correct statement subset, an incorrect statement subset, a missing correct statement subset and an irrelevant statement subset in the first statement set;

[0165] The analysis unit 405 is used to perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset and the irrelevant statement subset to obtain a first performance analysis result, and generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result between multiple preset question and answer texts, multiple first reply texts, and multiple retrieval context texts.

[0166] As an optional implementation of the embodiment of the present application, the first acquisition unit 401 is specifically used to perform statement extraction on the standard reply text, the first reply text and the retrieval context text through a first large language model to obtain the standard statement set, the first statement set, and the retrieval context statement set.

[0167] As an optional implementation of the embodiment of the present application, the comparison unit 404 is specifically used to: obtain the semantics corresponding to each statement in the first statement set, the semantics corresponding to each statement in the standard statement set, and the semantics corresponding to each statement in the retrieval context statement set; based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain a first proportion parameter of the statements in the first statement set in the standard statement set; the first proportion parameter is used to characterize the correctness of the first reply text; based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtain a second proportion parameter of the statements in the standard statement set in the first statement set; the second proportion The parameter is used to characterize the integrity of the first reply text; based on the semantics corresponding to each statement in the first statement set and the retrieval context statement set, a third proportion parameter of the statements in the first statement set in the retrieval context statement set is obtained; the third proportion parameter is used to characterize the fidelity of the first reply text; based on the semantics corresponding to each statement in the standard statement set and the retrieval context statement set, a fourth proportion parameter of the statements in the standard statement set in the retrieval context statement set is obtained; the fourth proportion parameter is used to characterize the quality of the first reply text; the statement comparison result is generated according to the first proportion parameter, the second proportion parameter, the third proportion parameter and the fourth proportion parameter.

[0168] As an optional implementation of the embodiment of the present application, the analysis unit 405 is also used to obtain the overall performance indicator set of the target RAG system, the indicator set of the retrieval module, and the indicator set of the reply generation module based on the number of statements in each subset of the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statement, the relevant previous retrieval context statement subset in the retrieval context statement set, and the irrelevant retrieval context statement subset. As an optional implementation of the embodiment of the present application, the analysis unit 405 is also used to add the number of statements in the correct statement subset and the incorrect statement subset to obtain a first sum value, and obtain the ratio of the number of statements in the correct statement subset to the first sum value to generate an overall accuracy value;

[0169] Add the number of statements in the correct statement subset and the missing statement subset to obtain a second total value, and obtain the ratio of the number of statements in the correct statement subset to the second total value to generate an overall recall value; generate the overall performance indicator set based on the overall precision value and the overall recall value.

[0170] As an optional implementation of an embodiment of the present application, the analysis unit 405 is also used to add the number of statements in the relevant retrieval context statement subset and the irrelevant retrieval context statement subset to obtain a third sum value, and obtain the ratio of the number of statements in the relevant retrieval context statement subset to the third sum value to generate a retrieval context precision value; add the correct statements and the missing correct statements in the relevant context statement subset to obtain a fourth sum value, and obtain the ratio of the fourth sum value to the second sum value to generate a retrieval recall value;

[0171] The retrieval module performance indicator set is generated based on the retrieval context precision value and the retrieval context accuracy value.

[0172] As an optional implementation of the embodiment of the present application, the analysis unit 405 is also used to obtain a first value of the correct statement in the relevant context statement subset, and obtain a ratio of the first value to the fourth total value to generate a context utilization rate; obtain a second value of the erroneous statement in the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the second value to the second total value to generate noise sensitivity; obtain a third value of the erroneous statement that does not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtain a ratio of the third value to the second total value to generate a hallucination appearance. probability; obtaining a fourth numerical value of correct statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the fourth numerical value to the second total value to generate own knowledge utilization; adding the number of correct statements in the irrelevant context statement subset and the number of incorrect statements in the relevant context statement subset and the union of the irrelevant context statement subset to obtain a fifth numerical value, and obtaining a ratio of the fifth numerical value to the second total value to generate fidelity; based on the context utilization, the noise sensitivity, the probability of hallucination, the own knowledge utilization, and the fidelity, generate a set of indicators for the answer generation module.

[0173] Based on the same inventive concept, an embodiment of the present disclosure also provides an electronic device. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as Figure 5 As shown, the electronic device provided in this embodiment includes: a memory 501 and a processor 502, wherein the memory 501 is used to store a computer program; and the processor 502 is used to execute the audio data processing method provided in the above embodiment when executing the computer program.

[0174] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the computing device implements the evaluation method of the retrieval enhancement generation RAG system provided in the above embodiment.

[0175] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0176] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0177] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0178] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating a retrieval enhancement generation (RAG) system, characterized in that: include: Obtaining a plurality of preset question and answer texts and a standard answer text corresponding to each of the preset question and answer texts; Input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain a first answer text and a retrieval context text; the retrieval context text is the context text retrieved by the target RAG system when generating the first answer text; Obtaining a standard statement set corresponding to the standard reply text, a first statement set corresponding to the first reply text, and a retrieval context statement set corresponding to the retrieval context text; Perform statement comparison based on the semantics of each statement in the standard statement set, the first statement set, and the search context statement set to generate a correct statement subset, an incorrect statement subset, a missing correct statement subset, and an irrelevant statement subset in the first statement set; Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset and the irrelevant statement subset, analysis and calculation are performed to obtain a first performance analysis result, and based on the first performance analysis result between multiple preset question and answer texts, multiple first reply texts, and multiple retrieval context texts, a target comprehensive performance evaluation result of the target RAG system is generated.

2. The method according to claim 1, characterized in that The obtaining of the standard statement set corresponding to the standard reply text, the first statement set corresponding to the first reply text, and the search context statement set corresponding to the search context text includes: The standard reply text, the first reply text and the search context text are subjected to statement extraction through a first large language model to obtain the standard statement set, the first statement set and the search context statement set.

3. The method according to claim 1, characterized in that The combining the semantics of each statement in the standard statement set, the first statement set, and the search context statement set to perform statement comparison to generate a correct statement subset, an incorrect statement subset, a missing correct statement subset, and an irrelevant statement subset in the first statement set includes: Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtaining a first proportion parameter of the statements in the first statement set in the standard statement set; the first proportion parameter is used to characterize the correctness of the first reply text; Based on the semantics corresponding to each statement in the first statement set and the standard statement set, obtaining a second proportion parameter of the statements in the standard statement set in the first statement set; the second proportion parameter is used to characterize the completeness of the first reply text; Based on the semantics corresponding to each statement in the first statement set and the search context statement set, obtaining a third proportion parameter of the statements in the first statement set in the search context statement set; the third proportion parameter is used to characterize the fidelity of the first reply text; Based on the semantics corresponding to each statement in the standard statement set and the search context statement set, obtaining a fourth proportion parameter of the statements in the standard statement set in the search context statement set; the fourth proportion parameter is used to characterize the quality of the first reply text; The declaration comparison result is generated according to the first proportion parameter, the second proportion parameter, the third proportion parameter, and the fourth proportion parameter.

4. The method according to claim 1, characterized in that Before performing analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, and the irrelevant statement subset to obtain a first performance analysis result, the method further includes: Based on the correct statement subset, the incorrect statement subset, the missing correct statement subset, the irrelevant statements, the relevant previous retrieval context statement subset in the retrieval context statement set, and the number of statements in each subset of the irrelevant retrieval context statement subset, the overall performance indicator set of the target RAG system, the indicator set of the retrieval module, and the indicator set of the reply generation module are obtained.

5. The method according to claim 4, characterized in that The method further comprises: Adding the number of statements in the correct statement subset and the number of statements in the incorrect statement subset to obtain a first sum value, and obtaining a ratio of the number of statements in the correct statement subset to the first sum value to generate an overall accuracy value; Adding the number of statements in the correct statement subset and the missing statement subset to obtain a second sum value, and obtaining a ratio of the number of statements in the correct statement subset to the second sum value to generate an overall recall value; The overall performance indicator set is generated based on the overall precision value and the overall recall value.

6. The method according to claim 5, characterized in that The method further comprises: Adding the number of statements in the relevant search context statement subset and the irrelevant search context statement subset to obtain a third sum value, and obtaining a ratio of the number of statements in the relevant search context statement subset to the third sum value to generate a search context precision value; Adding the correct statements and the missing correct statements in the relevant context statement subset to obtain a fourth sum value, and obtaining a ratio of the fourth sum value to the second sum value to generate a retrieval recall value; The retrieval module performance indicator set is generated based on the retrieval context precision value and the retrieval context accuracy value.

7. The method according to any one of claim 6, characterized in that: The method further comprises: Obtaining a first value of the correct statement in the relevant context statement subset, and obtaining a ratio of the first value to the fourth sum value to generate a retrieval context utilization rate; Obtaining a second value of the erroneous statements in a union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the second value to the second sum value to generate a noise sensitivity; Obtaining a third value of erroneous statements that do not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the third value to the second sum value to generate a hallucination occurrence probability; Obtaining a fourth value of a correct statement that does not belong to the union of the relevant context statement subset and the irrelevant context statement subset, and obtaining a ratio of the fourth value to the third sum value to generate own knowledge utilization; Adding the number of correct statements in the irrelevant context statement subset and the number of incorrect statements in the union of the relevant context statement subset and the irrelevant context statement subset to obtain a fifth value, and obtaining a ratio of the fifth value to the third sum value to generate fidelity; Based on the context utilization, the noise sensitivity, the probability of hallucination occurrence, the own knowledge utilization, and the fidelity, a set of indicators for the answer generation module is generated.

8. An evaluation device for a retrieval enhancement generation RAG system, characterized in that: include: A first acquisition unit is used to acquire a plurality of preset question and answer texts and a standard reply text corresponding to each of the preset question and answer texts; The second acquisition unit is used to input the preset question and answer text into the target retrieval enhancement generation RAG system to obtain the first answer text and the retrieval context text; the retrieval context text is the context text retrieved by the target retrieval enhancement generation RAG system when generating the first answer text; A third acquisition unit is used to acquire a standard statement set corresponding to the standard reply text, a first statement set corresponding to the first reply text, and a retrieval context statement set corresponding to the retrieval context text; a comparison unit, configured to perform statement comparison based on the semantics of each statement in the standard statement set, the first statement set, and the search context statement set, and generate a correct statement subset, an incorrect statement subset, a missing correct statement subset, and an irrelevant statement subset in the first statement set; An analysis unit is used to perform analysis and calculation based on the correct statement subset, the incorrect statement subset, the missing correct statement subset and the irrelevant statement subset to obtain a first performance analysis result, and generate a target comprehensive performance evaluation result of the target RAG system based on the first performance analysis result between multiple preset question and answer texts, multiple first reply texts, and multiple retrieval context texts.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the evaluation method of the retrieval enhancement generation RAG system as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the evaluation method of the retrieval enhancement generation RAG system according to any one of claims 1 to 7.

Images