Information processing system, information processing method, and program

Abstract

To achieve highly accurate performance evaluation of the RAG system without the need to pre-prepare question-and-answer pairs as a dataset. [Solution] An information processing system for evaluating a Search and Aggregate Generator (RAG) system, comprising: a question generation unit that generates evaluation questions based on a reference document; an acquisition unit that acquires at least one of a search document retrieved by the RAG system and an answer generated by the RAG system as the output of the RAG system for the questions; and a performance evaluation unit that evaluates the performance of the RAG system using the output of the RAG system acquired by the acquisition unit and the reference document.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Conventionally, a technique for evaluating the performance of a Retrieval-Augmented Generation (RAG) system is known. For example, in Patent Document 1, Q&A data, which is a pair of a question and an answer related to target data, is generated by a large language model, the question is input into the RAG system to obtain an answer, and a technique for evaluating the accuracy of the RAG answer with respect to the answer in the Q&A data is described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, in order to evaluate the performance of the RAG system, it is necessary to prepare a pair of a question and an answer in advance as a data set, and there is room for improvement in the performance evaluation method. In addition, although a method of automatically generating a model answer using an LLM has been proposed, there is a problem that the risk of hallucination included in the generated model answer itself cannot be eliminated, and the reliability of the evaluation is impaired.

[0005] The present invention has been made in consideration of such circumstances, and one of the objectives is to provide an information processing system, an information processing method, and a program that can realize high-precision performance evaluation of the RAG system without the need to prepare a pair of a question and an answer in advance as a data set.

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing system for evaluating a Search Enhancement Generation (RAG) system, comprising: a question generation unit that generates evaluation questions based on a reference document; an acquisition unit that acquires at least one of a search document retrieved by the RAG system and an answer generated by the RAG system as the output of the RAG system for the questions; and a performance evaluation unit that evaluates the performance of the RAG system using the output of the RAG system acquired by the acquisition unit and the reference document. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to achieve highly accurate performance evaluation of a RAG system without the need to prepare question-and-answer pairs as a dataset in advance. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows the overall processing flow and overview of the RAG evaluation method according to this embodiment. [Figure 2] This block diagram shows an example of the functional configuration of information processing system 100. [Figure 3] This figure shows an example of the data structure of the evaluation dataset 154. [Figure 4] This diagram shows the relationship between evaluation indicators and input / output data as determined by the performance evaluation unit 140. [Figure 5] This figure shows an example of the fidelity assessment prompt template 156A. [Figure 6] This figure shows an example of the accuracy assessment prompt template 156B. [Figure 7] This figure shows an example of prompt template 156C for relevance assessment. [Figure 8] This flowchart shows the flow of the evaluation process by the information processing system 100 according to this embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the information processing system, information processing method, and program of the present invention will be described with reference to the drawings.

[0010] [Challenges in the underlying technology] In recent years, the adoption of chatbots utilizing Large Language Models (LLMs) has been rapidly increasing in corporate customer support and internal help desks. In particular, Retrieval-Augmented Generation (RAG) technology, which generates answers based on organization-specific documents and up-to-date manuals, is extremely useful because it can supplement knowledge that the LLM has not learned. However, regular performance evaluations are essential to maintain and improve the quality of answers generated by RAG systems, and several challenges exist in the methodologies for these evaluations.

[0011] Many conventional evaluation methods involved preparing a static dataset consisting of a set of "questions" and their corresponding "model answers (correct answer data)," and measuring the similarity between the answers output by the RAG system and the model answers. However, the process of preparing these "model answers" itself has a major problem. Creating them manually is extremely costly, and it is difficult to keep up with document updates. On the other hand, methods for automatically generating "model answers" using LLM have also been proposed, but this does not eliminate the risk that the model answers generated for evaluation themselves may contain errors (hallucination).

[0012] The information processing system according to this embodiment uses highly reliable "reference documents (official help pages, etc.)" themselves as the direct basis for correct answers (Ground Truth), thereby enabling highly accurate and efficient evaluation of the search and answer performance of the RAG system without generating uncertain model answers.

[0013] [Overall System Configuration and Operation Overview] Figure 1 is a diagram showing the overall processing flow and overview of the RAG evaluation method according to this embodiment. In this figure, the data flow between the RAG system to be evaluated and the mechanism for evaluating its performance (the evaluation method according to this embodiment) is shown.

[0014] The RAG system is, for example, an interactive AI system used in user support such as electronic payment services. As shown in Figure 1, the RAG system has a function of searching for relevant documents (search documents) from an internal vector store for a question (input) from a user, and using the search documents to generate and output an answer by an LLM for RAG (LLM for answer generation).

[0015] The evaluation process according to this embodiment is roughly composed of "preparation of the evaluation dataset (S1, S2)", "execution of the evaluation (S3, S4)", and the final "performance evaluation".

[0016] First, in S1 (input), a reference document serving as an evaluation criterion is input. The reference document is a help page, guide information, etc. officially created by a service provider, and is a document with guaranteed content accuracy. This reference document becomes the absolute correct answer criterion (Golden Doc) in the subsequent evaluation process.

[0017] Next, in S2 (output), based on the input reference document, an evaluation question is generated by a first large language model (LLM1) for question generation. Here, LLM1 only generates "questions" based on the content of the reference document and does not generate "answers" for those questions. The generated questions are accumulated as an evaluation dataset.

[0018] Next, in S3 (input), the generated evaluation questions are input into the RAG system to be evaluated. Based on the input questions, the RAG system searches for relevant documents from its internal vector store and uses these retrieved documents to generate answers. The dashed arrow from the reference document to the vector store in the figure indicates that, as a preparatory step, the reference document has been indexed in the RAG system's vector store.

[0019] Subsequently, in S4 (acquisition), the output from the RAG system is obtained. The output obtained here includes not only the "answer" generated by the RAG system but also the information of the "retrieved documents" that the RAG system referred to for answer generation.

[0020] Finally, the performance evaluation of the RAG system is executed using a second large language model (LLM2) for performance evaluation. Specifically, the "answer" and "retrieved documents" output by the RAG system are compared and verified with the "reference document" input in step S1 for evaluation.

[0021] As will be described later, this evaluation is carried out by leveraging the advanced inference ability of LLM2. This enables the proposed evaluation method to clearly identify whether the error in the answer is due to "search failure" or "generation failure".

[0022] [Functional Configuration of Information Processing System] Figure 2 is a block diagram illustrating an example of the functional configuration of the information processing system 100 according to this embodiment. This information processing system 100 is the main device that executes the evaluation process (question generation and performance evaluation) described in Figure 1. The information processing system 100 comprises an input / output unit 110, a question generation unit 120, an acquisition unit 130, a performance evaluation unit 140, and a storage unit 150. In Figure 2, the information processing system 100, which is a single device, is shown to execute each function, but the present invention is not limited to such a configuration, and the functions of the information processing system 100 may be distributed across multiple devices (for example, a group of servers on the cloud), and these multiple devices may cooperate to execute each process. These components are functional modules realized by a hardware processor, such as a CPU (Central Processing Unit), reading and executing a program (software) stored in the storage unit 150. Some or all of these components may be implemented by dedicated hardware (including circuitry) such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units).

[0023] The storage unit 150 is a high-capacity storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores the reference document 152, the evaluation dataset 154, and the prompt template 156.

[0024] The reference document 152 is text data that serves as the correct answer criterion for evaluation, and each page or section is assigned a unique reference document ID. The evaluation dataset 154 is a list of questions generated by the question generation unit 120. The prompt template 156 is a template for instructions to have LLM2 perform a rigorous evaluation. Note that the granularity of the reference document ID is not limited to the page level. For example, an entire PDF file may be treated as a single reference document, or IDs may be assigned to specific "sections (chapters / sections)" or "paragraphs" within a page as the smallest unit, and the system may be configured to evaluate search performance at a finer granularity. Furthermore, the reference document is not limited to text data; it may also include figure / table data or image data, in which case the result of OCR processing of text information within the image may be used as the reference document.

[0025] The input / output unit 110 is an interface for data communication with external devices. For example, it is responsible for importing the reference document 152 into LLM1, sending requests to the RAG system 200 API, sending prompts to LLM2, and receiving inference results.

[0026] The question generation unit 120 is a functional unit that generates evaluation questions based on the content of the reference document 152. The question generation unit 120 is an example of the "question generation unit" in the claims. The question generation unit 120 uses LLM1 to input a specific page (e.g., P1) of the reference document 152 and gives instructions such as "Create a question that a user might ask based on the content described on this page." This generates question Q1, for which the answer is certain to be found on page P1. At this time, the ID of question Q1 and page P1 are recorded as a pair.

[0027] The acquisition unit 130 is a functional unit that inputs the generated question into the RAG system 200 and acquires the response result. The acquisition unit 130 is an example of the "acquisition unit" in the claims. The acquisition unit 130 not only receives the answer text but also acquires metadata (search document ID) indicating which documents the RAG system 200 internally searched and used as the basis for the answer. This makes it possible to analyze in detail the behavior of the RAG system 200 as a search engine.

[0028] The performance evaluation unit 140 is a functional unit that evaluates the performance of the RAG system 200 using the information acquired by the acquisition unit 130 and the reference document 152. The performance evaluation unit 140 is an example of the "performance evaluation unit" in the claims. Functionally, the performance evaluation unit 140 is divided into a search performance evaluation unit 140A and a response performance evaluation unit 140B.

[0029] The search performance evaluation unit 140A quantitatively evaluates the accuracy of document retrieval by the RAG system 200. Specifically, it compares the search document IDs retrieved by the RAG system 200 with the correct reference document IDs that were used to generate the queries. Alternatively, the search performance evaluation unit 140A may perform the evaluation based on whether or not the correct reference document ID is included in the set of multiple search documents output by the RAG system 200 (set operation).

[0030] The response performance evaluation unit 140B evaluates the quality of response generation by the RAG system 200. The response performance evaluation unit 140B inputs the question, RAG response, and the correct reference document into LLM2, and determines whether the RAG response is correct or not, considering only the content of the reference document as true.

[0031] [Evaluation data structure] Figure 3 shows an example of the structure of the evaluation dataset 154. The evaluation dataset 154 consists of a record for each question. Each record includes a "Question ID," a "Generated Question," and a "Reference Document ID (Golden Doc ID)." The "Question ID" is an identifier that identifies an individual question (e.g., Q001). The "Generated Question" is a natural language question sentence generated by LLM1 (e.g., "What are the identity verification procedures?"). The "Reference Document ID" is the ID of the reference document referenced when creating the question (e.g., KYC_001).

[0032] The distinguishing feature of this data structure is that it does not contain a "correct answer sentence," but rather a "source of the correct answer (reference document ID)" from which the question sentence was generated. If the "correct answer sentence" were retained, the answer sentence would also have to be updated whenever the content of the reference document was updated, increasing maintenance costs. However, by retaining the "source of the information ID" as in this embodiment, the correctness determination can always be made by referring to the content of the latest reference document during evaluation. This prevents data obsolescence and enables stable evaluation over the long term.

[0033] [Details of performance evaluation metrics and logic] Figure 4 shows the relationship between the evaluation indicators used by the performance evaluation unit 140 and the input data required for each evaluation. The performance evaluation unit 140 clearly identifies system problems by independently evaluating the document search performance and the response generation performance.

[0034] (1) Evaluation of document search performance The search performance evaluation unit 140A uses "search document (ID)" and "reference document ID" as input data. It compares these and calculates the following indicators.

[0035] • Recall: Recall is an indicator that shows how consistently the correct reference documents are included in the search results. It is expressed by the formula "(Number of correct documents found) / (Total number of correct documents)". In this system, since there is usually one reference document corresponding to one question, the recall is 1 if the document is found, and 0 otherwise. This allows for a direct evaluation of whether the user has reached the information they need.

[0036] • Precision: Precision is an indicator that shows what percentage of the documents included in the search results are the correct reference documents. It is expressed by the formula "(number of correct documents found) / (total number of documents found)". In a RAG system, if the search results contain many irrelevant documents (noise), the LLM that generates the answers will become confused and cause incorrect answers to be produced. Therefore, maintaining a high precision is important in preventing hallucination.

[0037] • F1 Score: The F1 score is the harmonic mean of recall and precision. The ideal state is one where there are no missed searches (high recall) and there is little noise (high precision), and it is used to evaluate the overall performance balance of a search engine. The F1 score is expressed by the following formula, F1 = 2 × (P × R) / (P + R), where R is recall and P is precision.

[0038] (2) Performance evaluation of response generation The response performance evaluation unit 140B uses at least a portion of the "question," "RAG response," and "reference document (text)" as input data. These are input into LLM2, and evaluation is performed from the following three perspectives. These evaluations are not simply keyword matching, but are performed by utilizing LLM2's semantic comprehension capabilities. Note that the combination of input data may be changed depending on the evaluation indicator. For example, in the evaluation of "relevance" described later, only the "question" and "RAG response" may be used without using the reference document.

[0039] • Faithfulness: Faithfulness is an indicator that evaluates whether all claims included in the RAG response are supported by the input reference document (correct information). Here, emphasis is placed on whether the response is "evidence-based" rather than on the correctness of the response. Even if the response is fluent, if it contains information not included in the reference document, it will be penalized as hallucination.

[0040] • Accuracy: Accuracy is an indicator that evaluates whether a RAG response correctly answers the user's question and does not contradict the content of the reference document. Here, not only the consistency of facts but also whether the intent of the question has been correctly interpreted is evaluated. For example, even if the facts are correct, if the answer is irrelevant to the question, it will be judged as having low accuracy.

[0041] • Relevancy: Relevance is an indicator that evaluates whether a RAG response is relevant to the topic, concise, and complete. It evaluates not only whether it contains the information the user needs, but also whether it contains unnecessary information. For example, if a response provides an excessively long explanation in response to a question, or adds too much supplementary information that wasn't asked, it will be evaluated as having low relevance because it detracts from the user's convenience. This helps maintain the quality of the responses.

[0042] Next, the prompt templates 156 used by the response performance evaluation unit 140B will be described in detail. These templates are designed to give LLM2 the role of a rigorous evaluator and to enable it to perform objective scoring for the fidelity, accuracy, and relevance indicators.

[0043] [Details of the fidelity assessment prompt] Figure 5 shows an example of a prompt template 156A for fidelity evaluation. The prompt template 156A includes, for example, a task definition area A1, an evaluation flow area A2, and a scoring criteria area A3.

[0044] Task definition area A1 instructs LLM2 to evaluate how accurately the RAG responses reflect the provided information sources (reference documents).

[0045] In evaluation flow area A2, LLM2 is asked to perform the following algorithm. The first step is to identify the main elements included in the RAG response. This involves verifying the individual elements that make up the content, rather than the entire text. The second step is to confirm whether each identified element adequately reflects the content of the information source. The third step is to consider the impact of elements that do not adequately reflect the content of the information source. That is, to determine whether the discrepancy is minor or significant enough to undermine the reliability of the response. The fourth step is an overall judgment. In this step, based on the evaluation of the individual elements, the LLM2 determines how faithful the RAG response is to the information source as a whole.

[0046] In scoring criterion area A3, scores are assigned according to the degree of fidelity. For example, a score of 5 (very fidelity) indicates that the entire RAG response matches the information source. A score of 4 (mostly fidelity) indicates that most of the response is consistent, with minor inconsistencies that do not impair understanding. A score of 3 (partially fidelity) indicates that the main parts are consistent, but there are significant inconsistencies that may impair understanding. A score of 2 (slightly fidelity) indicates that the main parts are inconsistent and the reliability is insufficient. A score of 1 (not fidelity) indicates that most of the response is inconsistent and lacks coherence. The output format of the fidelity assessment prompt template 156A is specified in JSON or similar, and may include not only the score but also the basis for verification and the reasons for inconsistencies. This allows developers to retrospectively verify the fidelity of RAG responses.

[0047] [Details of the accuracy assessment prompt] Figure 6 shows an example of a prompt template 156B for accuracy assessment. The prompt template 156B includes, for example, a critical instruction area A1, an assessment flow area A2, and a scoring criteria area A3.

[0048] In Key Instruction Area A1, six principles for evaluation are defined: (1) The evaluation is based solely on the content of the provided information source (reference document) and does not rely on external knowledge. (2) Additional content in the information source is not considered problematic; the focus is solely on whether the claims in the RAG response are substantiated by the information source. (3) All factual content included in the RAG response must be verifiable. (4) The content of the RAG response should be consistent not only with the information source but also with the user's question. (5) Claims that are poorly substantiated or contradictory will be treated as false. (6) Fact-based accuracy and logical consistency will be evaluated, rather than style or comprehensiveness.

[0049] In evaluation flow area A2, LLM2 is asked to perform the following algorithm: The first step is to extract the main elements contained in the RAG response. The second step is to check how well each element aligns with the information source and classify each element based on its degree of reflection. The third step is to evaluate whether the RAG response as a whole is logically appropriate to the question. In other words, accuracy evaluation assesses not only factual agreement but also logical appropriateness to the question.

[0050] In scoring criterion area A3, scores are assigned according to the degree of accuracy. "5 (Very accurate)" means the entire answer is consistent with the source and the question. "4 (Mostly accurate)" means the main content is correct with only minor inconsistencies. "3 (Partially accurate)" means there are some errors or inconsistencies that may affect the accuracy. "2 (Poor accuracy)" means the answer contains many errors or inconsistencies. "1 (Inaccurate)" means the answer is mostly inconsistent and inappropriate. The output format is specified as JSON, etc., and may include not only the score but also the content of the identified errors and the basis for the accuracy judgment. This allows developers to specifically track why an answer received a low score.

[0051] [Details of the relevance assessment prompt] Figure 7 shows an example of a prompt template 156C for relevance evaluation. The prompt template 156C includes, for example, a task definition area A1, an evaluation process area A2, and a scoring criteria area A3.

[0052] Task definition area A1 defines "evaluating how well RAG responses address the user's intent to the question." Here, relevance evaluation prioritizes whether "appropriate content is provided in response to the question" and "whether irrelevant information is included" rather than factual accuracy.

[0053] In evaluation process area A2, LLM2 is asked to perform the following algorithm: Step 1 is to extract the main elements that need to be answered from the question. Step 2 is to break down the RAG response into semantically coherent units (sentences, paragraphs, bullet points, etc.). Step 3 is to confirm the correspondence between the extracted main elements and each unit in the RAG response. In this process, as step 4, any content that does not connect to any main element is identified as "irrelevant information". Step 5 is to evaluate the extent to which the RAG response satisfies each main element (complete / partial / missing). Step 6 is to determine an overall score, taking into account factors that impair relevance, such as the amount of irrelevant information, insufficient answers, and excessive digressions.

[0054] In scoring criterion area A3, scores are assigned according to the degree of relevance. "5 (High relevance)" means all elements are met and there is no irrelevant information. "4 (Acceptable relevance)" means the elements are met and there is only a small amount of irrelevant information. "3 (Moderate relevance)" means some elements are missing or there is a significant amount of irrelevant information. "2 (Low relevance)" means many elements are missing and there is a noticeable lack of content or inappropriate information. "1 (No relevance)" means there is no match whatsoever with the intent of the question. The output format is defined as JSON, etc., and may include not only the score but also the degree of fulfillment of the main elements that formed the basis of the evaluation, and the specific content of the identified irrelevant information. This allows developers to analyze the relevance of RAG responses in detail.

[0055] It should be noted that the descriptions of prompt templates 156A to 156C in Figures 4 to 6, as explained above, are merely examples. The prompt templates only need to instruct the system to evaluate response performance such as fidelity, accuracy, and relevance. Furthermore, it is not necessary to evaluate all of the response performance metrics such as fidelity, accuracy, and relevance; evaluating only one of these metrics may suffice.

[0056] [Process flow] Figure 8 is a flowchart showing the overall flow of the evaluation process by the information processing system 100 according to this embodiment.

[0057] First, in S100, the question generation unit 120 reads the reference document 152 from the storage unit 150 and generates evaluation questions using LLM1. This step is performed when a new reference document is added or at the start of a periodic evaluation cycle. The generated questions are linked to the reference document ID and saved as an evaluation dataset 154.

[0058] Next, in S102, the acquisition unit 130 inputs the generated evaluation questions into the RAG system 200, which is the target of evaluation. The information processing system 100 calls the API of the RAG system 200 and sends the question text in the same way as an actual user. Next, in S104, the acquisition unit 130 acquires the output from the RAG system 200. The acquired data includes the answer text generated by the RAG system 200, as well as a list of IDs of search documents that the RAG system 200 used to generate the answers after searching them internally.

[0059] Next, in S106, the performance evaluation unit 140 performs a performance evaluation of the RAG system 200. This evaluation process consists of two stages: an evaluation of search performance and an evaluation of answer performance. In the evaluation of search performance, the search performance evaluation unit 140A compares the acquired search document ID with the correct answer reference document ID associated with the question generation. It then calculates the recall rate, etc., based on whether or not the correct answer document is included in the search results. In the evaluation of answer performance, the answer performance evaluation unit 140B calls LLM2. The answer performance evaluation unit 140B embeds the contents of the "question," "RAG answer," and the correct answer "reference document" into the detailed prompt template 156 mentioned above, and inputs it into LLM2.

[0060] LLM2 calculates scores (1-5) for fidelity, accuracy, and relevance after going through the instructed thought process (decomposition, verification, mapping, etc.). The inference results are returned not only as numerical values, but also as JSON data indicating the reasoning text and the parts that formed the basis of the judgment. Finally, in S108, the performance evaluation unit 140 aggregates the values ​​of each calculated evaluation index and outputs the evaluation results.

[0061] [summary] According to the information processing system 100 of this embodiment, the RAG system is evaluated using questions generated from a reference document and the reference document itself, eliminating the need for manual creation of correct answer data or generation of uncertain model answers. This significantly reduces the cost of constructing and maintaining evaluation datasets while enabling highly reliable evaluation.

[0062] Furthermore, by separating the evaluation of search performance and response performance, and by using multifaceted indicators such as fidelity, accuracy, and relevance in response generation, a detailed diagnosis of the RAG system becomes possible. For example, if search performance is high but fidelity is low, it becomes easier to formulate specific countermeasures such as needing improvement through prompt engineering, or if search performance itself is low, needing tuning of vector search. In particular, by using a rigorous evaluation logic based on prompt templates, objective and consistent evaluation results can be obtained.

[0063] Furthermore, by independently evaluating search performance (Recall) and answer performance (Faithfulness, etc.), it becomes possible to clearly distinguish whether the cause of an answer error lies in "search failure" or "generation failure (hallucination)," thereby streamlining the system improvement cycle.

[0064] [Differentiation] The present invention is not limited to the embodiments described above, and various modifications are possible. For example, in the above embodiments, LLM1 for question generation and LLM2 for performance evaluation were described separately, but these may be the same model. In another embodiment, the RAG LLM that generates answers within the RAG system 200 and the LLM2 that performs evaluation may be the same model.

[0065] Furthermore, although the RAG system 200 was treated as an external system in the above embodiment, the functions of the information processing system 100 may be incorporated as part of the RAG system 200 and operated as a self-diagnostic function. Also, the vector store (search documents) referenced by the RAG system 200 may be updated in real time in accordance with updates to the reference document 152, or it may be updated periodically by batch processing or the like. The information processing system 100 may be configured to regenerate the evaluation dataset 154 in accordance with the update timing of the vector store and to periodically perform performance monitoring of the RAG system.

[0066] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0067] 100 Information Processing Systems 110 Input / output section 120 Question generation part 130 Acquisition Department 140 Performance Evaluation Department 150 Storage section 152 Standard Documents 154 evaluation datasets 156 Prompt Templates 200 RAG System

Claims

1. An information processing system for evaluating a search extension generation (RAG) system, A question generation unit that generates evaluation questions based on a reference document, An acquisition unit that acquires at least one of the search documents retrieved by the RAG system and the answers generated by the RAG system as the output of the RAG system for the aforementioned question, The system includes a performance evaluation unit that evaluates the performance of the RAG system using the output of the RAG system acquired by the acquisition unit and the reference document, The question generation unit generates the questions linked to each page of the reference document, When the acquisition unit acquires the search document, the performance evaluation unit evaluates the performance of the document search by the RAG system by comparing the degree of agreement between the pages of the search document and the pages of the reference document linked to the query. Information processing system.

2. The performance evaluation unit calculates at least one of the following as the performance of the document search: recall, precision, and F1 value. The information processing system according to claim 1.

3. When the acquisition unit acquires the response, the performance evaluation unit evaluates the response based on the reference document to evaluate the performance of the response generation by the RAG system. The information processing system according to claim 1.

4. The performance evaluation unit calculates, as a measure of the performance of the response generation, a fidelity rating indicating the degree to which the response is supported by the reference document. The information processing system according to claim 3.

5. The performance evaluation unit calculates, as a measure of the performance of the response generation, an accuracy that indicates the degree to which the response correctly answers the question, based on the reference document. The information processing system according to claim 3.

6. The performance evaluation unit calculates the degree to which the answer is relevant to the question as a measure of the performance of answer generation. The information processing system according to claim 3.

7. The question generation unit generates the question using the first large-scale language model, The performance evaluation unit evaluates the performance of the RAG system using a second large-scale language model different from the first large-scale language model. The information processing system according to claim 1.

8. An information processing system for evaluating a search extension generation (RAG) system, A question generation unit that generates evaluation questions based on a reference document, An acquisition unit that acquires at least one of the search documents retrieved by the RAG system and the answers generated by the RAG system as the output of the RAG system for the aforementioned question, The system includes a performance evaluation unit that evaluates the performance of the RAG system using the output of the RAG system acquired by the acquisition unit and the reference document, The performance evaluation unit evaluates the performance of the RAG system, specifically the document search performance and the response generation performance, and based on the evaluation results of both, identifies whether the failure of the RAG system's response lies in the document search process or the response generation process. Information processing system.

9. An information processing method for evaluating a search extension generation (RAG) system, Computers Based on the reference document, generate evaluation questions, As output of the RAG system to the aforementioned question, at least one of the search documents retrieved by the RAG system and the answer generated by the RAG system is obtained. The performance of the RAG system is evaluated using the acquired output of the RAG system and the reference document. The generation of the aforementioned questions is performed by generating the questions in conjunction with each page of the aforementioned reference document. When the aforementioned search document is obtained, the performance evaluation is performed by comparing the degree of agreement between the pages of the search document and the pages of the reference document linked to the aforementioned query, thereby evaluating the performance of the document search by the RAG system. Information processing methods.

10. A program for evaluating a search extension generation (RAG) system, On the computer, Based on the reference document, generate evaluation questions. As output of the RAG system to the aforementioned question, at least one of the search documents retrieved by the RAG system and the answer generated by the RAG system is obtained. The performance of the RAG system is evaluated using the acquired output of the RAG system and the reference document. The generation of the aforementioned questions is performed by generating the questions in conjunction with each page of the aforementioned reference document. When the aforementioned search document is obtained, the performance evaluation is performed by comparing the degree of agreement between the pages of the search document and the pages of the reference document linked to the aforementioned query, thereby evaluating the performance of the document search by the RAG system. program.

11. An information processing method for evaluating a search extension generation (RAG) system, Computers Based on the reference document, generate evaluation questions, As output of the RAG system to the aforementioned question, at least one of the search documents retrieved by the RAG system and the answer generated by the RAG system is obtained. The performance of the RAG system is evaluated using the acquired output of the RAG system and the reference document. The performance evaluation described above assesses the performance of the RAG system, specifically the document search performance and the response generation performance, and based on the results of both evaluations, identifies whether the failure of the RAG system's response lies in the document search process or the response generation process. Information processing methods.

12. A program for evaluating a search extension generation (RAG) system, On the computer, Based on the reference document, generate evaluation questions. As output of the RAG system to the aforementioned question, at least one of the search documents retrieved by the RAG system and the answer generated by the RAG system is obtained. The performance of the RAG system is evaluated using the acquired output of the RAG system and the reference document. The performance evaluation described above assesses the performance of the RAG system, specifically the document search performance and the response generation performance, and based on the results of both evaluations, identifies whether the failure of the RAG system's response lies in the document search process or the response generation process. program.

Images