Method for carrying out RAG index evaluation on RAG system put into production

Abstract

A method for carrying out RAG index evaluation on an RAG system put into production comprises the steps that a test set is prepared in advance, and the content of the test set is lt; an original question and a correct answer gt; a two-tuple list; the RAG index evaluation method specifically comprises the following steps of 1, calling an RAG module interface of a production environment by using an original problem in a test set, and obtaining a model reply and a reference text list; 2, the original question, the model reply, the reference text list and the correct answer form evaluation data according to an input parameter format of an evaluation framework; and 3, calling an evaluation data interface of an evaluation framework by using the evaluation data obtained in the step 2, and generating an evaluation result by the evaluation framework. According to the method, RAG evaluation can be carried out under the condition that a model and a database interface used in a production environment are not provided.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval enhancement generation evaluation, and in particular to a method for performing RAG index evaluation on a RAG system that has been put into production. Background Art

[0002] The RAG system put into production refers to a system that contains RAG modules or functions, such as a knowledge base question and answer application system. RAG is the abbreviation of retrieval-augmented generation, which obtains certain text fragments through retrieval and provides them to a large model (also known as a large language model, large-scale language model, English: Large Language Model, abbreviated LLM) as supplementary information for generating answers, thereby reducing the probability of hallucinations when generating large models.

[0003] The purpose of using the RAG evaluation framework is to judge the current RAG performance through data, which generally includes indicators such as the correctness of the response, the completeness of the response, the proportion of relevant fragments in the retrieval context, the degree to which the generator effectively uses relevant information in the context, etc. After the evaluation, the query, sorting, filtering, generation and other modules can be optimized according to different indicator scores.

[0004] RAG evaluation usually extracts entities and keywords from questions, correct answers, model answers, and recalled texts (text fragments obtained through retrieval), and combines different mathematical models and statistical methods to derive different indicator scores for each test data.

[0005] There are currently a variety of open source RAG evaluation frameworks, each designed to solve the problem of difficulty in manual evaluation and scoring when evaluating RAG functions. What they have in common is that they require users to provide their own large language model, embedding model, rearrangement model, vector database and other interfaces, as well as test sets to generate evaluation data sets within the framework, and then conduct evaluations based on the evaluation interfaces. The disadvantage of this method is that it is suitable for technical research during development, but it is not friendly to environments that have already integrated RAG into the system, and there are also security risks in opening up the interfaces called within the production environment.

[0006] Therefore, in view of the shortcomings of the existing technology, it is necessary to provide a method for evaluating RAG indicators of the RAG system that has been put into production to overcome the shortcomings of the existing technology. Summary of the invention

[0007] The purpose of the present invention is to avoid the shortcomings of the prior art and provide a method for evaluating RAG indicators of a RAG system that has been put into production, which can perform RAG evaluation without providing a model and database interface used in the production environment.

[0008] The purpose of the present invention is achieved through the following technical measures.

[0009] Provide a method for evaluating RAG indicators of a RAG system that has been put into production, prepare a test set in advance, and the test set content is a list of <original question, correct answer> tuples;

[0010] The method for RAG indicator evaluation is specifically carried out through the following steps:

[0011] Step 1. Use the original question in the test set to call the RAG module interface of the production environment to obtain the "model response" and "reference text list";

[0012] Step 2. Combine the "original question", "model response", "reference text list" and "correct answer" into evaluation data according to the input parameter format of the evaluation framework;

[0013] Step 3. Use the evaluation data obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

[0014] Preferably, the above method for evaluating RAG indicators on a RAG system that has been put into production repeats step 1, uses all or part of the original questions in the test set to call the RAG module interface of the production environment and obtain the corresponding "model response" and "reference text list";

[0015] Step 2. Each "original question", "model response", "reference text list" and "correct answer" obtained is combined into the corresponding evaluation data according to the input parameter format of the evaluation framework, and each evaluation data is summarized into an evaluation data summary table;

[0016] Step 3. Use the evaluation data in the evaluation data summary table obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

[0017] Another preferred method of performing RAG index evaluation on the RAG system that has been put into production is to use an original question in the test set each time, and execute step 1 and step 2 in sequence, respectively, to obtain evaluation data corresponding to each original question each time, and to obtain evaluation data corresponding to all or part of the original questions after executing step 1 and step 2 multiple times, and to form an evaluation data summary table by summarizing each evaluation data;

[0018] Step 3. Use the evaluation data in the evaluation data summary table obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

[0019] Preferably, the above-mentioned method for evaluating RAG indicators of a RAG system that has been put into production further includes step 4. saving the evaluation results in the form of a file or database.

[0020] Preferably, in the above-mentioned method of performing RAG indicator evaluation on a RAG system that has been put into production, the output of the production environment includes model responses and a list of referenced texts or information for obtaining the referenced text content through another query.

[0021] Preferably, in the above-mentioned method for performing RAG indicator evaluation on a RAG system that has been put into production, the RAG evaluation framework input parameters used are composed of the original question, the correct answer, the model response and the reference text list.

[0022] Preferably, in the above-mentioned method for performing RAG indicator evaluation on a RAG system that has been put into production, the evaluation data interface of the RAG evaluation framework used can be called separately.

[0023] The method of the present invention for performing RAG indicator evaluation on a RAG system that has been put into production performs RAG evaluation through an open source RAG evaluation framework or a framework specified by the demander without providing a model and a database interface used in the production environment. It is only required that the system in the production environment can return the knowledge text referenced when performing RAG. The solution of the present invention does not need to provide a model and a database interface used in the production environment, thereby reducing the security risks of internal interface exposure. The solution of the present invention has relatively loose requirements on the RAG evaluation framework, and currently open source frameworks on the Internet such as RAGAS and RAGChecker can be used. The solution of the present invention evaluates the effects of the production environment and can better reflect the actual effect of the system than evaluating it in the development environment. The model used to perform RAG operations through the solution of the present invention can be different from the model used as a referee for the evaluation, so as to avoid the same model's degree of semantic understanding affecting the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0025] Figure 1 It is an overall structural diagram of a RAG evaluation framework for applying a method of the present invention to perform RAG index evaluation on a RAG system that has been put into production.

[0026] Figure 2 It is the overall structure diagram of directly using the RAD evaluation framework in the existing technology.

[0027] Figure 3 It is another overall structural diagram of a RAG evaluation framework for applying a method of the present invention to perform RAG indicator evaluation on a RAG system that has been put into production. DETAILED DESCRIPTION

[0028] The present invention will be further described with reference to the following examples.

[0029] Example 1.

[0030] The present embodiment is a method for performing RAG index evaluation on a RAG system that has been put into production. In order to describe the technical solution of the present invention in detail, the improvement of the method of the present embodiment is compared with the direct use of the RAG evaluation framework in the prior art, and the implementation principles and steps are explained one by one.

[0031] In the prior art, when evaluating the RAG indicators of the RAG system that has been put into production, the RAG evaluation framework is directly used, such as Figure 2 As shown in the figure. Directly using the RAG evaluation framework is actually an evaluation of the capabilities of the large language model, embedding model, and rearrangement model, which requires the interfaces of each model to be exposed. Since the evaluation is conducted with the interfaces of each model exposed, the interfaces called in the production environment are open to the outside world, which poses a security risk.

[0032] The input parameters of each RAG evaluation framework are scored using the four-tuple <original question, correct answer, model response answer, reference text list>. It is feasible to organize evaluation data by combining the test set + production environment output results. The modules for generating evaluation data and scoring evaluation data in the RAG framework are separated, which makes it possible to use externally generated data for evaluation. When performing RAG in the production environment, the retrieved text, user input, and model output are all necessary, and it is feasible to obtain data.

[0033] The method of performing RAG index evaluation on the RAG system that has been put into production in this example optimizes the method of directly using the RAG evaluation framework to perform evaluation in the prior art. Figure 2 As shown, the method of the present invention for performing RAG indicator evaluation on a RAG system that has been put into production performs RAG evaluation through an open source RAG evaluation framework or a framework specified by the demander without providing a model and a database interface used in the production environment. It is only required that the system in the production environment can return the knowledge text referenced when performing RAG. The solution of the present invention does not need to provide a model and a database interface used in the production environment, thereby reducing the security risks of internal interface exposure.

[0034] This embodiment is a method for evaluating RAG indicators on a RAG system that has been put into production. The output of the production environment includes model responses and a list of referenced texts, or information about the content of the referenced texts obtained through another query. The input parameters of the RAG evaluation framework used are composed of the original question, the correct answer, the model response, and the list of referenced texts. The evaluation data interface of the RAG evaluation framework used can be called separately.

[0035] The present invention discloses a method for performing RAG index evaluation on a RAG system that has been put into production. A test set is prepared in advance, and the content of the test set is a list of two-tuples of <original question, correct answer>. The input parameters of each RAG evaluation framework are scored using the four-tuple <original question, correct answer, model reply answer, reference text list>. The test set is used to call the RAG module interface of the production environment through the test set prepared in advance, and the output results of the production environment can be used to organize the evaluation. RAG index evaluation is performed on the RAG system that has been put into production without exposing each model interface.

[0036] The present invention provides a method for evaluating RAG indicators of a RAG system that has been put into production, which is specifically performed through the following steps:

[0037] Step 1. Use the original question in the test set to call the RAG module interface of the production environment to obtain the "model response" and "reference text list";

[0038] Step 2. Combine the "original question", "model response", "reference text list" and "correct answer" into evaluation data according to the input parameter format of the evaluation framework;

[0039] Step 3. Use the evaluation data obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

[0040] The method may further include step 4 of saving the evaluation results in a file or database format.

[0041] It should be noted that the <original question, correct answer> tuple list in the test set usually includes multiple "original questions". You can choose to use all the original questions to perform the above steps 1 to 3 according to actual needs, or you can choose a part of the original questions to perform the above steps 1 to 3 according to needs.

[0042] For example, one solution may be to repeat step 1, use all or part of the original questions in the test set to call the RAG module interface of the production environment and obtain the corresponding "model response" and "reference text list";

[0043] Step 2. Each "original question", "model response", "reference text list" and "correct answer" obtained is combined into the corresponding evaluation data according to the input parameter format of the evaluation framework, and each evaluation data is summarized into an evaluation data summary table;

[0044] Step 3. Use the evaluation data in the evaluation data summary table obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

[0045] The following scheme may also be adopted: each time an original question in the test set is used, and steps 1 and 2 are executed in sequence, and evaluation data corresponding to each original question is obtained each time. After executing steps 1 and 2 multiple times, evaluation data corresponding to all or part of the original questions are obtained, and each evaluation data is summarized to form an evaluation data summary table;

[0046] Step 3. Use the evaluation data in the evaluation data summary table obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

[0047] There is an evaluation framework that produces evaluation results, which can be done through a large language model. This is common knowledge in the field and is not the main invention of this case, so it will not be elaborated here.

[0048] The method of the present invention for performing RAG indicator evaluation on a RAG system that has been put into production performs RAG evaluation through an open source RAG evaluation framework or a framework specified by the demander without providing a model and a database interface used in the production environment. It is only required that the system in the production environment can return the knowledge text referenced when performing RAG. The solution of the present invention does not need to provide a model and a database interface used in the production environment, thereby reducing the security risks of internal interface exposure. The solution of the present invention has relatively loose requirements on the RAG evaluation framework, and currently open source frameworks on the Internet such as RAGAS and RAGChecker can be used. The solution of the present invention evaluates the effects of the production environment and can better reflect the actual effect of the system than evaluating it in the development environment. The model used to perform RAG operations through the solution of the present invention can be different from the model used as a referee for the evaluation, so as to avoid the same model's degree of semantic understanding affecting the evaluation results.

[0049] Example 2.

[0050] A method for evaluating RAG indicators of a RAG system that has been put into production, the other features of which are the same as those of Example 1, except that: the processes of step 1 to step 3 in this embodiment are integrated into the RAG system, such as Figure 3 As shown in the figure, you can evaluate the production system by deploying an open source RAG evaluation framework through a plug-in.

[0051] The method of the present invention for performing RAG indicator evaluation on a RAG system that has been put into production performs RAG evaluation through an open source RAG evaluation framework or a framework specified by the demander without providing a model and a database interface used in the production environment. It is only required that the system in the production environment can return the knowledge text referenced when performing RAG. The solution of the present invention does not need to provide a model and a database interface used in the production environment, thereby reducing the security risks of internal interface exposure. The solution of the present invention has relatively loose requirements on the RAG evaluation framework, and currently open source frameworks on the Internet such as RAGAS and RAGChecker can be used. The solution of the present invention evaluates the effects of the production environment and can better reflect the actual effect of the system than evaluating it in the development environment. The model used to perform RAG operations through the solution of the present invention can be different from the model used as a referee for the evaluation, so as to avoid the same model's degree of semantic understanding affecting the evaluation results.

[0052] Example 3.

[0053] The method of the present invention for evaluating RAG indicators of a RAG system that has been put into production is described in detail with a specific example.

[0054] Preprocessing steps: Prepare the test set as shown in Table 1.

[0055] Table 1

[0056]

[0057] Specific implementation steps:

[0058] 1. Read the test set and record the original question list and the correct answer list respectively.

[0059] question = ["When a shopping mall restaurant kitchen and dining area are separated by a fireproof wall, how many hours of fire resistance is required at least?","A 5-story industrial plant with a building height of 25m is equipped with an indoor fire hydrant system. What is the minimum dynamic pressure of the indoor fire hydrant at the most unfavorable point in the plant?","The design working pressure of the indoor fire hydrant system of a hotel is 1.0MPa. Two fire hydrant pumps are installed (one for use and one for backup). What is the minimum maximum range of the pressure gauge on the outlet pipe of the fire hydrant pump in MPa?"]

[0060] truth = ["2 hours","0.35MPa","2.5MPa"]

[0061] 2. Take a question in question as an input parameter and call the RAG module interface of the production environment to get the model response and reference text information, as follows:

[0062] {

[0063] “content”:”According to Article 6.2.3 of the Code for Fire Protection Design of Buildings GB50016-2014 (2018 Edition), the following parts of a building should be separated from other parts by fire-resistant walls with a fire resistance limit of not less than 2.00h, including \"except for kitchens in suites of residential buildings, public kitchens in dormitories and apartment buildings and kitchens in other buildings\".

[0064] Shopping mall catering kitchens belong to "kitchens in other buildings", so when shopping mall catering kitchens and dining areas are separated by fireproof walls, the fire resistance limit of the fireproof walls must be at least 2.00 hours. ",

[0065] "chunks":['Fire safety management in crowded places GB_T40248-2021\n8.3 Shopping malls\n8.3.2 Warehouses located in shopping malls should be completely separated from business and office areas by partitions with a fire resistance limit of not less than 3.00h, and openings leading to business halls should be equipped with Class A fire doors. ',

[0066] 'Code for Fire Protection Design of Buildings GB50016-2014 (2018 Edition)\n6 Building Structure\n6.5 Fire Doors, Windows and Fire Shutters\nArticle Explanation\n\n6.5.3 This article is a general requirement for fire shutters installed on fire walls, fire partitions and openings on building exterior walls.\n\xa0\xa0\xa0(3) Since the relevant standards do not specify the smoke-tightness performance of fire shutters, this article also stipulates that the gaps around fire shutters should be strictly sealed against fire and smoke according to the use of fire shutters in actual buildings to prevent smoke and fire from spreading through the gaps around the shutters. \n\xa0\xa0\xa0(4) Regarding the fire resistance time of fireproof roller shutters, due to different installation locations, the fire resistance limit requirements of the fire separation locations are different. For example, if a fireproof roller shutter is installed on a firewall or in a location where a firewall is required, the fire resistance limit of the roller shutter must be at least 3.00h; if it is installed on a fireproof partition wall with a fire resistance limit requirement of 2.00h, the fire resistance limit of the roller shutter cannot be less than 2.00h. If a fireproof cooling water curtain is used to protect the fireproof roller shutter, the fire duration of the water curtain system must also be determined according to the above method. \n' ]

[0068] }.

[0069] The "content" part is the model response, and the "chunks" part is the quoted text information.

[0070] 3. Fill in the original question, model response, reference text list and correct answer into the corresponding fields of the assessment data format. Here, we take the RAGAS framework input parameters as an example to form the following content:

[0071] {

[0072] 'question':['When a shopping mall restaurant kitchen and dining area are separated by a fireproof wall, how many hours should the fireproof wall's fire resistance limit be at least? '],

[0073] 'answer':['According to the known information in Article 6.2.3 of the Code for Fire Protection Design of Buildings GB50016-2014 (2018 Edition) , the following parts in a building should be separated from other parts by fire-resistant walls with a fire resistance limit of not less than 2.00h, including \"except for kitchens in suites in residential buildings, public kitchens in dormitories and apartment buildings, and kitchens in other buildings\".

[0074] The catering kitchen in a shopping mall belongs to the category of "kitchens in other buildings". Therefore, when the catering kitchen in a shopping mall is separated from the dining area by a fireproof partition wall, the fire resistance limit of the fireproof partition wall must be at least 2.00 hours. '],

[0075] 'contexts':[['Fire safety management in crowded places GB_T40248-2021\n8.3 Shopping malls\n8.3.2 Warehouses located in shopping malls should be completely separated from business and office areas by partitions with a fire resistance limit of not less than 3.00h, and openings leading to business halls should be equipped with Class A fire doors. ',

[0076] 'Code for Fire Protection Design of Buildings GB50016-2014 (2018 Edition)\n6 Building Structure\n6.5 Fire Doors, Windows and Fire Shutters\nArticle Explanation\n\n6.5.3 This article is a general requirement for fire shutters installed on fire walls, fire partitions and openings on building exterior walls.\n\xa0\xa0\xa0(3) Since the relevant standards do not specify the smoke-tightness performance of fire shutters, this article also stipulates that the gaps around fire shutters should be strictly sealed against fire and smoke according to the use of fire shutters in actual buildings to prevent smoke and fire from spreading through the gaps around the shutters. \n\xa0\xa0\xa0(4) Regarding the fire resistance time of fireproof roller shutters, due to different installation locations, the fire resistance limit requirements of the fire separation locations are different. For example, if a fireproof roller shutter is installed on a firewall or in a location where a firewall is required, the fire resistance limit of the roller shutter must be at least 3.00h; if it is installed on a fireproof partition wall with a fire resistance limit requirement of 2.00h, the fire resistance limit of the roller shutter cannot be less than 2.00h. If a fireproof cooling water curtain is used to protect the fireproof roller shutter, the fire duration of the water curtain system must also be determined according to the above method. \n'

[0077] ]],

[0078] 'ground_truth':['2 hours']

[0079] }.

[0080] 'question', 'answer', 'contexts', and 'ground_truth' correspond to the original question, model response, reference text list, and correct answer, respectively.

[0081] 4. Repeat steps 2 and 3 until each item in the original question list is used as an input parameter to call the RAG interface of the production environment and obtain the corresponding model response and reference text list.

[0082] 5. The constructed evaluation data set is used as the input parameter to call the evaluation data interface of the RAG evaluation framework.

[0083] 6. The evaluation results are shown in Table 2. The evaluation data interface returns parameters that are saved to a file.

[0084] Table 2

[0085]

[0086] Experimental results show that the solution of the present invention does not need to provide models and database interfaces used in the production environment, reducing the security risks of internal interface exposure. The solution of the present invention has relatively loose requirements on the RAG evaluation framework, and currently online open source frameworks such as RAGAS, RAGChecker, etc. can be used. The solution of the present invention evaluates the effects of the production environment and can better reflect the actual effect of the system than evaluating it in the development environment. The model used to perform RAG operations through the solution of the present invention can be different from the model used as a referee for the evaluation, so as to avoid the same model's degree of semantic understanding affecting the evaluation results.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for evaluating RAG indicators of a RAG system that has been put into production, characterized in that: Prepare a test set in advance. The test set content is a list of <original question, correct answer> pairs. The method for RAG indicator evaluation is specifically carried out through the following steps: Step 1. Use the "original question" in the test set to call the RAG module interface of the production environment to obtain the "model response" and "reference text list"; Step 2. Combine the "original question", "model response", "reference text list" and "correct answer" into evaluation data according to the input parameter format of the evaluation framework; Step 3. Use the evaluation data obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

2. The method for evaluating RAG indicators of a RAG system that has been put into production according to claim 1 is characterized in that: Repeat step 1, use all or part of the "original questions" in the test set to call the RAG module interface of the production environment and obtain the corresponding "model response" and "reference text list"; Step 2. Compose each "original question", "model response", "reference text list" and "correct answer" obtained into the corresponding evaluation data according to the input parameter format of the evaluation framework, and summarize each evaluation data into an evaluation data summary table; Step 3. Use the evaluation data in the evaluation data summary table obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

3. The method for evaluating RAG indicators of a RAG system that has been put into production according to claim 1 is characterized in that: Each time, use an "original question" in the test set, and execute step 1 and step 2 in sequence, and obtain the evaluation data corresponding to each original question each time. After executing step 1 and step 2 multiple times, obtain the evaluation data corresponding to all or part of the original questions, and summarize each evaluation data to form an evaluation data summary table; Step 3. Use the evaluation data in the evaluation data summary table obtained in step 2 to call the evaluation data interface of the evaluation framework, and the evaluation framework generates the evaluation results.

4. The method for evaluating RAG indicators of a RAG system that has been put into production according to claim 1, 2 or 3, characterized in that: The method also includes step 4: saving the evaluation results in the form of a file or a database.

5. The method for evaluating RAG indicators of a RAG system that has been put into production according to claim 1, 2 or 3, characterized in that: The output for the production environment includes the model response and a list of referenced texts or information to retrieve the referenced text content through another query.

6. The method for evaluating RAG indicators of a RAG system that has been put into production according to claim 5 is characterized in that: The input parameters of the RAG evaluation framework used consist of the original question, the correct answer, the model response, and a list of reference texts.

7. The method for evaluating RAG indicators of a RAG system that has been put into production according to claim 6 is characterized in that: The evaluation data interface of the RAG evaluation framework used can be called separately.

Images