Information processing method, information processing program, and information processing system

Abstract

Evaluate the accuracy of the RAG system's answers. [Solution] The information processing method disclosed herein includes the steps of: (f) after all Q&A data corresponding to multiple target data satisfy a predetermined standard, a user terminal inputs a first question of a pair of a first question and a first answer to the first question contained in the Q&A data into a RAG system and has the RAG system provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

Description

[Technical field]

[0001] The present disclosure relates to an information processing method, an information processing program, and an information processing system. [Background technology]

[0002] Retrieval Augmented Generation (RAG) has been known for some time (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7527581 Summary of the Invention [Means for solving the problem]

[0004] An information processing method according to one embodiment of the present disclosure is an information processing method for evaluating answer accuracy of a RAG (Retrieval Augmented Generation) system via a user terminal, in which the user terminal (a) generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) causes a first large-scale language model to generate Q&A data, which is a pair of a question and an answer related to the target data, using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) outputs a first judgment result regarding the accuracy of the Q&A data for the target document; and (d) generates the Q&A data by regenerating the Q&A data when the first judgment result does not satisfy a predetermined standard. (e) executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputting a first question from a pair of a first question and a first answer to the first question included in the Q&A data into the RAG system and causing the RAG system to provide a second answer to the first question; and (g) outputting a second judgment result regarding accuracy of the second answer to the first answer.

[0005] An information processing program according to an embodiment of the present disclosure is a computer-executable information processing program, the information processing program including: (a) a step of generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) a step of generating Q&A data, which is a pair of a question and an answer related to the target data, in a first large-scale language model using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) a step of outputting a first judgment result regarding accuracy of the Q&A data for the target document; (d) a step of providing regenerated new Q&A data to step (c) when the first judgment result does not satisfy a predetermined criterion; (e) a step of executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; and (f) a step of extracting the first question from among a pair of a first question and a first answer to the first question included in the Q&A data after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, and (g) outputting a second determination result regarding the accuracy of the second answer to the first answer from the RAG system.

[0006] An information processing system according to one embodiment of the present disclosure is an information processing system for evaluating answer accuracy of a RAG (Retrieval Augmented Generation) system, and includes: a viewpoint generation unit that generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of a question and an answer related to the target data, using a first large-scale language model based on the generated viewpoint and target data among the plurality of target data that corresponds to the viewpoint; a first output unit that outputs a first determination result regarding accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first determination result satisfies a predetermined criterion; an answer acquisition unit that inputs the first question, of a pair of a first question and a first answer to the first question included in the Q&A data, into the RAG system after all of the Q&A data corresponding to the plurality of target data satisfies the predetermined criterion, and acquires a second answer to the first question from the RAG system; and a second output unit that outputs a second determination result regarding accuracy of the second answer to the first answer. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a functional configuration of an information processing system according to the present disclosure. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of a server according to the present disclosure. [Diagram 3] FIG. 2 is a block diagram showing a functional configuration of a user terminal according to the present disclosure. [Figure 4] 1 is a flowchart illustrating an example of a flow of information processing according to the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram of a chatbot according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. For ease of understanding, first, the background and problems of the present disclosure will be described, and then the details of the present disclosure will be described. <Search Extension Generation> Retrieval-Augmented Generation (RAG) is a method of searching external knowledge databases in real time and having generative AI generate answers based on that information. While regular generative models rely on learned knowledge, RAG excels in tasks where the latest information or accuracy is important. RAG augments answers based on search results, so it can provide updated content with higher accuracy. RAG has the following advantages over generative AI that uses fine tuning:

[0009] Reflects the latest information: Whereas fine-tuning uses historical data to update the model, RAG has access to new information in real time.

[0010] Reduced data update load: Fine tuning requires periodic model updates, but RAG does not require frequent retraining because it obtains the latest information through each search.

[0011] Flexibility: RAG can generate answers that are applicable across a wide range of domains, rather than being topic specific.

[0012] On the other hand, since RAG may contain erroneous information (hallucination) that is not included in internal documents, there is still room for improvement in the accuracy of the responses. <Outline of Information Processing System 1> The information processing system 1 in the present disclosure evaluates the answer accuracy of a RAG (Retrieval Augmented Generation) system. The information processing system 1 can also improve the answer accuracy of the RAG system based on recommendations according to the evaluation results. The information processing system 1 may be used for a chatbot that outputs answers corresponding to questions received online from users.

[0013] The information processing system 1 can be used in various fields, such as airplanes, railways, ships, hospitals, banks, and hotels.

[0014] More specifically, the information processing system 1 is an information processing system for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system, and includes: a viewpoint generation unit that generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of a question and an answer related to the target data, using a first large-scale language model by using the generated viewpoint and target data among the plurality of target data that corresponds to the viewpoint; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined criterion; an answer acquisition unit that inputs the first question, of a pair of a first question and a first answer to the first question included in the Q&A data, into the RAG system after all of the Q&A data corresponding to the plurality of target data satisfies the predetermined criterion, and acquires a second answer to the first question from the RAG system; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer.

[0015] According to the above configuration, the information processing system 1 can output the answer accuracy of the RAG system as the second judgment result. By having the user confirm the answer accuracy of the RAG system as the second judgment result, the information processing system 1 can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy.

[0016] Furthermore, according to the above configuration, unlike conventional Q&A reference chatbots, the information processing system 1 can automatically generate high-quality Q&As by simply storing internal company documents, etc., in a specified folder (such as a data store) by the user. Furthermore, the information processing system 1 can also perform accuracy evaluation side-by-side using multiple generation AIs, thereby reducing the possibility of hallucination and improving the answer accuracy of RAG.

[0017] The information processing system 1 will be described below with reference to the drawings.

[0018] FIG. 1 is a block diagram showing a functional configuration of an information processing system 1 according to the present disclosure.

[0019] FIG. 2 is a block diagram showing a functional configuration of the server 10 according to the present disclosure.

[0020] FIG. 3 is a block diagram showing a functional configuration of the user terminal 20 according to the present disclosure. <Configuration of Information Processing System 1> 1, an information processing system 1 includes a server 10, a user terminal 20, and an administrator terminal 30, which are connected via a network N.

[0021] The server 10 and the user terminal 20 may be realized by being integrated into a single information processing terminal device, but in the following description they will be described as being separate devices.

[0022] The network N may be a mobile communication system such as 3G, 4G, 5G, or 6G, LTE (Long Term Evolution), Wi-Fi (registered trademark), the Internet, or an in-house LAN.

[0023] <Basic configuration of Server 10> The server 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, and a communication IF 14. These are electrically connected to each other via a communication bus.

[0024] The processor 11 executes instructions written in a program and is composed of an arithmetic unit, a register, and / or a peripheral circuit.

[0025] The main storage device 12 temporarily stores programs and data to be processed by the programs and the like, and is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0026] The auxiliary storage device 13 stores data and programs, and is, for example, a flash memory, a hard disk drive (HDD), or a magneto-optical disk.

[0027] The communication IF 14 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

[0028] 2, the server 10 includes a storage unit 101 and a control unit 102. The storage unit 101 and the control unit 102 will be described below.

[0029] The storage unit 101 stores an application program 1011 , a large-scale language model 1012 , target documents 1013 , target data 1014 , and Q&A data 1015 .

[0030] The application programs 1011 are programs for causing the control unit 102 of the server 10 to function as each functional unit. The application programs 1011 include applications such as a web browser application.

[0031] The large-scale language model 1012 stores a plurality of large-scale language models used in the information processing system 1. The plurality of large-scale language models may be at least partially different from each other. The plurality of large-scale language models being different from each other means that they have different architectures or hyperparameters, etc., including differences in layer structure, number of neurons, training method, and / or optimization algorithm. Furthermore, the datasets and / or tuning methods used for learning may also be different from each other, which may cause differences in the quality and / or characteristics of the generated text. As a result, different responses can be expected for each model even for the same task.

[0032] The large-scale language model 1012 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of a user terminal 20 described below, or in another storage device.

[0033] Next, the target document 1013 will be described. In general, in search expansion generation, reference information is searched for from a predetermined data source based on instruction information that instructs the execution of a task, and a prompt including the instruction information and the search results of the reference information is input to a large-scale language model. The large-scale language model executes the task by referring to reference information that is not included in the training data. In this way, search expansion generation improves erroneous information (hallucination), which has been an issue in the answers of conventional generation AI.

[0034] The target document 1013 may be considered to correspond to the reference information described above. The target document 1013 may be internet information or the latest information, databases, and internal documents stored in an organization (such as a company), and the type is not limited. The target document 1013 may be stored in various formats, such as Word, PowerPoint, PDF, SQL format, CSV format, JSON format, XML format, and binary format. In the following, for convenience of explanation, the target document 1013 will be explained as a document (target document) in PDF format stored in a company.

[0035] The target document 1013 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of the user terminal 20 described below, or in another storage device.

[0036] Next, a description will be given of the target data 1014. The target data 1014 is a plurality of data pieces obtained by dividing the target document into predetermined units. The predetermined units may be determined as appropriate, such as, for example, one page, two pages, or a chapter in a document.

[0037] The Q&A data 1015 is a data set that associates expected questions for the target data 1014 with model answers. The Q&A data 1015 is created in order to determine how close an answer obtained by giving a question to the system to be evaluated is to the model answer.

[0038] The Q&A data 1015 typically includes "Questions" that a user may ask the system and "Answers" that are model answers to those questions. Furthermore, the Q&A data 1015 may include "Categories" that indicate the themes or topics to which the questions belong, and / or "Tags" that are keywords that aid in searching or filtering.

[0039] The Q&A data 1015 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of the user terminal 20 described below, or in another storage device.

[0040] 2, the control unit 102 includes a target document division unit 1020, a component classification unit 1021, a viewpoint generation unit 1022, a Q&A data generation unit 1023, a first output unit 1024, a Q&A data regeneration unit 1025, an answer acquisition unit 1026, and a second output unit 1027. The control unit 102 may include a reception unit 1028.

[0041] The target document division unit 1020 divides the target document into predetermined units. The predetermined units may be determined appropriately, such as, for example, one page, two pages, or a chapter in the document. In the following, the target document division unit 1020 will be described as dividing the target document stored in a predetermined folder into one page units.

[0042] The target document division unit 1020 may divide the target document into pages, for example, using an online tool (such as Smallpdf or ILovePDF), a function built into Adobe Acrobat, or a program such as Python. Specifically, the target document division unit 1020 divides the target document 1013 into pages, and stores a plurality of target data 1014 in the storage unit 101. The target document division unit 1020 may save the target data 1014 in a storage device different from the storage unit 101 of the server 10. When the target document 1013 is one page, the target document 1013 may be understood as having been divided into one by the target document division unit 1020.

[0043] The component classification unit 1021 classifies components included in the target data 1014 obtained by dividing the target document into pages by the target document division unit 1020, into, for example, "text only," "graph," "table," and "figure other than graph." The component classification unit 1021 may be realized by a large-scale language model such as GPT (Generative Pretrained Transformer), but there is no restriction on the scale or structure of the model.

[0044] The control unit 102 may not include the component classification unit 1021. However, by including the component classification unit 1021, the control unit 102 can cause the first output unit 1024 (described later) to output the first judgment result including the score for each component. This allows the control unit 102 to give a clue as to which classification item should be corrected to improve the accuracy of the Q&A data 1015. For example, when the first output unit 1024 outputs the first judgment result in the range of 1 to 10, the scores are "8.56" for the graph, "8.30" for the table, "8.83" for the figure other than the graph, and "6.75" for the plain text only. At this time, since the score for the plain text only is the lowest, a clue is given that the accuracy of the Q&A data 1015 can be improved by correcting only the plain text. Similarly, by including the component classification unit 1021, the control unit 102 can cause the second judgment result (described later) to include the judgment result for each component.

[0045] The component classification unit 1021 may be, for example, a recurrent neural network (RNN) such as a convolutional neural network (CNN), a long-short-term memory (LSTM), or a gated recurrent unit (GRU), or another language model that combines one or more models such as a Transformer model. The component classification unit 1021 may be realized by, for example, ChatGPT, Bard, Gemini, or Claude. This also applies to the viewpoint generation unit 1022, the Q&A data generation unit 1023, the first output unit 1024, the Q&A data regeneration unit 1025, the second output unit 1027, the recommendation output unit 1029, and the optimization unit 1030, which will be described later.

[0046] The viewpoint generating unit 1022 generates a viewpoint for each of a plurality of target data 1014 obtained by dividing the target document 1013 into predetermined units (one page). The viewpoint includes at least one of the title, summary, type of figure / table / text, conclusion, background, and purpose of the target data.

[0047] The viewpoint generation unit 1022 may be realized by a large-scale language model such as a Generative Pretrained Transformer (GPT), but there is no limitation on the scale or structure of the model.

[0048] The viewpoint generator 1022 inputs, for example, the following prompt into the large-scale language model: Example prompts "Based on what you can read from the data below, what can you say?" In response to this prompt, the viewpoint generator 1022 obtains, for example, the following output: Perspective: Performance by product Explanation: Trends in sales target achievement for each product, the most stable products and the most volatile products. The prompt may be set as a default, or the receiving unit 1028 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the viewpoint generating unit 1022 may input the received prompt to the large-scale language model.

[0049] The Q&A data generation unit 1023 uses the viewpoint generated by the viewpoint generation unit 1022 and the target data 1014 corresponding to the viewpoint among the multiple target data 1014 to generate Q&A data 1015 which is a pair of a question and an answer related to the target data 1014. The Q&A data generation unit 1023 may be realized by a large-scale language model. The Q&A data generation unit 1023 stores the generated Q&A data 1015 in the storage unit 101. The Q&A data generation unit 1023 may output the generated Q&A data 1015 to the first output unit 1024.

[0050] The first output unit 1024 outputs a first determination result regarding the accuracy of the Q&A data 1015 with respect to the target data 1014. The first output unit 1024 may be realized by a large-scale language model. The first determination result includes at least one of a determination value regarding the accuracy and a determination reason.

[0051] A good first determination result is likely to be output when the same large-scale language model is used by the Q&A data generation unit 1023 and the first output unit 1024. Therefore, the first output unit 1024 may use, but is not limited to, a second large-scale language model different from the Q&A data generation unit 1023 (first large-scale language model).

[0052] The first output unit 1024 receives, for example, the following prompt: Example prompts "Please refer to the data below to determine the accuracy of your answers to the questions. # Question: ${QUESTION} # Answer: ${ANSWER} Please rate your results on a scale of 1 to 10. If there are any discrepancies in the information on which your answer is based, please rate it 5 points or lower. If the information is accurate and there is simply a lack of a conclusion, please rate it 7 points or higher. Please indicate your results in the "score" and "reason." Then, the first output unit 1024 outputs, for example, the following first determination result. ·Example of first judgment result "Score: 3" "Reasoning: The first half of your answer is correct, but the second half contradicts the first half. Therefore, I gave it 3 points." As described above, the first output unit 1024 may output the first determination result including the score for each component. For example, the first output unit 1024 outputs the first determination result including the scores of "8.56" for the graph, "8.30" for the table, "8.83" for the figure other than the graph, and "6.75" for the plain text only. In this way, the first output unit 1024 may output the first determination result including the numbers after the decimal point.

[0053] The prompt may be set as a default, or the receiving unit 1028 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the first output unit 1024 may input the received prompt to the large-scale language model.

[0054] The first output unit 1024 may output the first judgment result to the Q&A data regenerator 1025, or may store the first judgment result in the storage unit 101 or an external storage device.

[0055] The Q&A data regeneration unit 1025 generates new Q&A data 1015 when the first judgment result output by the first output unit 1024 does not satisfy a predetermined standard. The Q&A data regeneration unit 1025 may be realized by a large-scale language model. The Q&A data regeneration unit 1025 generates Q&A data 1015, which is a pair of a question and an answer related to the target data 1014, by using the viewpoint generated by the viewpoint generation unit 1022 and the target data 1014 corresponding to the viewpoint among the multiple target data 1014. The Q&A data regeneration unit 1025 (third large-scale language model) may be the same as the first large-scale language model and / or the second large-scale language model described above, or may be a different large-scale language model.

[0056] In the above-mentioned prompt in which a score is given in the range of 1 to 10, the "predetermined standard" is, for example, "8", and if the score is 8 or more, it is treated as satisfying the predetermined standard. The predetermined standard may be determined arbitrarily. The "predetermined standard" shown here is an example, and the present disclosure is not limited thereto. For example, the "predetermined standard" may be appropriately set, such as "4" when a score is given in the range of 1 to 5. Alternatively, the "predetermined standard" may be set as an average score for all Q&A data 1015 being X points or more. Alternatively, the "predetermined standard" may be set as a combination of X points or more for a graph, Y points or more for a table, and Z points or more for text.

[0057] In this manner, in the present disclosure, the "predetermined standard" may be set in various ways. Also, in the present disclosure, for example, when the average score for all the Q&A data 1015 is X points or more, or when a standard such as X points or more in the case of a graph or Y points or more in the case of a table is satisfied, it can be considered that "all the Q&A data 1015 corresponding to the plurality of target data 1014 satisfy the predetermined standard."

[0058] The Q&A data regeneration unit 1025 outputs the regenerated new Q&A data 1015 to the first output unit 1024. The Q&A data regeneration unit 1025 may store the regenerated new Q&A data 1015 in the storage unit 101. The first output unit 1024 outputs a first determination result regarding the accuracy of the new Q&A data 1015 for the target document 1013 by the method described above.

[0059] The Q&A data regeneration unit 1025 may cause the user who has referred to the first judgment result to generate new Q&A data 1015. For example, the Q&A data regeneration unit 1025 outputs a screen or a voice instructing the user to generate new Q&A data 1015, and accepts the new Q&A data 1015 from the user. Then, the Q&A data regeneration unit 1025 outputs the new Q&A data 1015 accepted from the user to the first output unit 1024. The Q&A data regeneration unit 1025 can also be realized by such a configuration.

[0060] The above operations may be performed for all of the Q&A data 1015 corresponding to the multiple target data 1014.

[0061] After all of the Q&A data corresponding to the multiple target data satisfy a predetermined criterion, the answer acquiring unit 1026 operates as follows. Specifically, of a pair of a first question and a first answer to the first question included in the Q&A data, the answer acquiring unit 1026 inputs the first question to the RAG system, and acquires the second answer to the first question from the RAG system. The RAG system is a RAG system operated in a production environment, a test environment, a development environment, or the like.

[0062] The second output unit 1027 outputs a second determination result regarding the accuracy of the second answer to the first answer. The second output unit 1027 may be realized by a large-scale language model (fourth large-scale language model). The second determination result includes at least one of a determination value and a determination reason regarding the accuracy.

[0063] The second output unit 1027 receives, for example, the following prompt: Example prompts "You are the person in charge of assessing the quality of the Q&A system. Please evaluate the system's answer to the expected question (answer 2) by carefully comparing it with the model answer (answer 1). Pay close attention to the differences between the model answer and the system's answer when explaining numerical values ​​and units. # Expected question: ${QUESTION} # Model answer: ${{REF_ANSWER} # System answer: ${SYSTEM_ANSWER} Please rate your results on a scale of 1 to 10, with the model answer being 10 points. If there are even some differences between the system answer and the information on which the model answer is based, please rate it 5 points or less. If the information is accurate and there is simply a lack of a conclusion, please rate it 7 points or more. Then, the second output unit 1027 outputs, for example, the following second determination result. ·Second judgment result example 1 "Score: 5" "Reason: In addition to the two correct answers, there was one incorrect answer, so I gave it 5 points." ·Second judgment result example 2 "Score: 3" "Reason: The first half of your answer is correct, but the second half contradicts the first half. Therefore, I gave it 3 points." The second output unit 1027 may output the second judgment result including the score for each component. For example, the second output unit 1027 outputs the second judgment result including the scores of "8.68" for the graph, "7.75" for the table, "8.58" for the figure other than the graph, and "7.56" for the plain text only. In this way, the second output unit 1027 may output the second judgment result including the numbers after the decimal point.

[0064] The second output unit 1027 may store the second determination result in the storage unit 101 or an external storage device.

[0065] The prompt may be set as a default, or the receiving unit 1028 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the second output unit 1027 may input the received prompt to the large-scale language model.

[0066] According to the above configuration, the information processing system 1 can output the answer accuracy of the RAG system as the second judgment result. The information processing system 1 can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy by having the user confirm the answer accuracy of the RAG system as the second judgment result.

[0067] The fourth large-scale language model may be the same as the first, second or third large-scale language model, or another large-scale language model may be used. The fourth large-scale language model may also be composed of multiple large-scale language models. <Recommendation function> Referring back to FIG. 2, the control unit 102 may further include a recommendation output unit 1029 and an optimization unit 1030.

[0068] The recommendation output unit 1029 outputs a recommendation according to the second determination result. The recommendation output unit 1029 may be realized by a large-scale language model such as a Generative Pretrained Transformer (GPT), but there is no limit to the scale or structure of the model.

[0069] As an example, the recommendation output unit 1029 accepts the following prompt based on the second determination result output by the second output unit 1027. Example prompts Please explain the categories that received high and low ratings from the test results below, and provide your recommendations for the areas that received low ratings. # Test results Model used: gemini-pro-1.0 Layout Parser: OFF Plain text: 7.5 Image data: 7.2 Table data: 3.4" Then, the recommendation output unit 1029 outputs, for example, the following recommendation. Example of evaluation comment "Evaluation Results: The test results showed that answers regarding plain text and image data were appropriate and highly rated, while answers regarding table data were not rated as desirable. Recommendation: 1) Setting recommendation: The model currently being used is an old model, so we recommend using the latest stable model. 2) We recommend that you turn on the Layout Parser function. 3) Recommendations for document contents: We recommend that you do not use symbols in table data. We recommend that table data be structured as simply as possible. In this way, the recommendation output unit 1029 can analyze the tendency of the second judgment result with a low score, and can output a recommendation such as "The model most suitable for the current document is Gemini 1.5 pro". This can be obtained by instructing the recommendation output unit 1029 realized by the large-scale language model to output a recommendation of the analysis result. These are only examples, and the recommendation output unit 1029 can output a recommendation for the type of large-scale language model or the setting value of the RAG system itself, such as the parameters to be used. In addition, the recommendation output unit 1029 can also output a recommendation for the contents of the target document (or target data) that is the reference destination of the RAG system. In this way, the recommendation output unit 1029 can output a recommendation in a wide range of contents (range).

[0070] When the second determination result does not satisfy a predetermined standard, the optimization unit 1030 executes optimization based on the recommendation output by the recommendation output unit 1029. For example, as an example of optimization, the optimization unit 1030 executes processing such as changing the large-scale language model used by the RAG system to Gemini 1.5 pro (selecting the optimal LLM) and changing (tuning) the parameters of the large-scale language model used by the RAG system, thereby improving the second determination result. Alternatively, as another example, the optimization unit 1030 may execute processing such as changing the writing style of the target document referred to by the RAG system, for example, not using symbols in the table data used, thereby improving the second determination result.

[0071] When the second judgment result does not satisfy the predetermined standard, for example, the score of the second judgment result is set to "7 or more". In this case, "table data: 3.4" does not satisfy the predetermined standard, while "plain text: 7.5" and "image data: 7.2" satisfy the predetermined standard. The optimization unit 1030 determines that "table data: 3.4" does not satisfy the predetermined standard, and executes the above-mentioned optimization process. When all scores related to the second judgment result satisfy the predetermined standard, the optimization unit 1030 may end the process.

[0072] After performing the process of optimizing based on the recommendation, the optimization unit 1030 causes the answer acquisition unit 1026 to answer (acquire) a second answer. The subsequent process is as described above.

[0073] The information processing system 1, having the above configuration, can improve the response accuracy of the RAG system. <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service (e.g., a chatbot). The user terminal 20 operates the control unit 102 of the server 10 in response to, for example, an input operation of a user who uses a chatbot. The user terminal 20 may be, for example, a stationary PC (Personal Computer), a laptop PC, a mobile terminal such as a smartphone or a tablet, an HMD (Head Mount Display), or a wearable terminal such as a wristwatch terminal.

[0074] The user terminal 20 includes a storage unit 201 , a control unit 202 , an input unit 203 , and an output unit 204 . <Storage unit 201 of user terminal 20> The storage unit 201 of the user terminal 20 stores an application program 2011. The storage unit 201 may also store a user ID 2012.

[0075] The application program 2011 may be pre-stored in the storage unit 201. The application program 2011 may be downloaded from a web server operated by a service provider via a communication IF. The application program 2011 may include an application such as a web browser application.

[0076] The user ID 2012 is a user's account ID. The user transmits the user ID 2012 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2012 and provides a service to the user. If a service (e.g., a chatbot) does not require a user ID, the storage unit 201 does not need to store the user ID 2012. <Control unit 202 of user terminal 20> The control unit 202 of the user terminal 20 includes an input control unit 2021 and an output control unit 2022. The control unit 202 executes an application program 2011 stored in the storage unit 201 to realize each functional unit. <Input section 203 of user terminal 20> The input unit 203 of the user terminal 20 may include a camera 2031, a microphone 2032, a keyboard 2033, a touch panel 2064, and the like. <Output unit 204 of user terminal 20> The output unit 204 of the user terminal 20 may include a display 2041 and a speaker 2042 . <Configuration of administrator terminal 30> 1, the administrator terminal 30 is an information processing device operated by an administrator of a service (e.g., a chatbot). The administrator is also an operator of the information processing system 1. Through the administrator terminal 30, the administrator builds a large-scale language model to be used in the information processing system 1, or determines or modifies a prompt to be input to the large-scale language model.

[0077] The administrator terminal 30 may be, for example, a desktop PC (Personal Computer), a laptop PC, a mobile terminal such as a smartphone or a tablet, an HMD (Head Mount Display), or a wearable terminal such as a wristwatch terminal.

[0078] <Example of processing by information processing device> Next, the flow of processing of the information processing system 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the flow of information processing according to the present disclosure.

[0079] In S2, the target document division section 1020 divides the target document into predetermined units.

[0080] In S3, the component classification unit 1021 classifies the components contained in the target data 1014 obtained by dividing the target data 1014 into pages by the target document division unit 1020, into, for example, "text only," "graphs," "tables," and "figures other than graphs."

[0081] In S4, the viewpoint generating unit 1022 generates a viewpoint for each of a plurality of target data pieces obtained by dividing the target document into predetermined units (for example, one page).

[0082] In S6, the Q&A data generation unit 1023 generates Q&A data, which is a pair of questions and answers related to the target data, using the viewpoint generated by the viewpoint generation unit 1022 and target data corresponding to the viewpoint among the multiple target data.

[0083] In S8, the first output unit 1024 outputs a first determination result regarding the accuracy of the Q&A data for the target document.

[0084] If the first judgment result does not satisfy the predetermined standard (NO in S10), proceed to S12. In S12, the Q&A data regeneration unit 1025 regenerates new Q&A data. Then, in S8, the first output unit 1024 outputs the first judgment result regarding the accuracy of the new Q&A data for the target document. The first judgment result may include a score for each component.

[0085] In S10, if the first judgment result satisfies the predetermined criterion (YES in S10), the process proceeds to S14. After all of the Q&A data corresponding to the multiple target data satisfies the predetermined criterion, in S14, the answer acquisition unit 1026 inputs the first question, of the pair of the first question and the first answer to the first question included in the Q&A data, to the RAG system, and acquires the second answer to the first question from the RAG system.

[0086] In S16, the second output unit 1027 outputs a second determination result regarding the accuracy of the second answer relative to the first answer. The second determination result may include a score for each component element.

[0087] In S18, the recommendation output unit 1029 outputs a recommendation according to the second determination result.

[0088] In S20, if the second determination result satisfies the predetermined criterion (YES in S20), the process ends. In S20, if the second determination result does not satisfy the predetermined criterion (NO in S20), in S22, the optimization unit 1030 executes optimization based on the recommendation output by the recommendation output unit 1029. After executing the optimization process based on the recommendation, the optimization unit 1030 causes the second output unit 1027 to answer (obtain) a second answer again (proceed to S8).

[0089] <Chatbot> Next, a chatbot 50 according to the present disclosure will be described with reference to Fig. 5. Fig. 5 is a schematic diagram of a chatbot 50 according to the present disclosure.

[0090] The chatbot 50 is an example of a RAG system in which answer accuracy is improved by the information processing system 1 according to the present disclosure. The chatbot 50 includes a reception unit 51 that receives questions from users, and an answer presentation unit 52 that presents answers to the questions. This allows the chatbot 50 to provide the user with a RAG system with improved answer accuracy.

[0091] <Example of software implementation> The functions of the information processing system 1 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the control unit 102).

[0092] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0093] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0094] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0095] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be operated by the control device or may be operated by another device (for example, an edge computer or a cloud server). <Summary> The present disclosure describes at least the following aspects.

[0096] An information processing method according to a first aspect of the present disclosure is an information processing method for evaluating answer accuracy of a RAG (Retrieval Augmented Generation) system via a user terminal, in which the user terminal (a) generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) causes a first large-scale language model to generate Q&A data, which is a pair of questions and answers related to the target data, using the generated viewpoints and target data corresponding to the viewpoints among the plurality of target data; (c) outputs a first judgment result regarding the accuracy of the Q&A data for the target document; and (d) generates the Q&A data regenerated when the first judgment result does not satisfy a predetermined standard. (e) executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputting a first question from a pair of a first question and a first answer to the first question included in the Q&A data into the RAG system and causing the RAG system to provide a second answer to the first question; and (g) outputting a second judgment result regarding accuracy of the second answer to the first answer.

[0097] According to the above configuration, the information processing method according to aspect 1 of the present disclosure regenerates Q&A data until a first judgment result regarding the accuracy of the Q&A data for a target document satisfies a predetermined standard. Then, the information processing method according to aspect 1 of the present disclosure executes the steps (b) to (d) for all of the Q&A data corresponding to a plurality of target data. Then, the information processing method according to aspect 1 of the present disclosure positions the steps (a) to (e) as advance preparation, and following the advance preparation, outputs a second judgment result regarding the accuracy of the second answer to the first answer by the steps (f) and (g).

[0098] According to the above configuration, the information processing method according to the first aspect of the present disclosure can output the answer accuracy of the RAG system as the second judgment result. The information processing method according to the first aspect of the present disclosure can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy by having the user confirm the answer accuracy of the RAG system as the second judgment result.

[0099] In the information processing method according to aspect 2 of the present disclosure, in accordance with aspect 1, the step (c) is performed by a second large-scale language model different from the first large-scale language model.

[0100] According to the above configuration, the information processing method according to aspect 2 of the present disclosure can increase the possibility of outputting the first judgment result with higher accuracy by using different large-scale language models in steps (b) and (c).

[0101] In the information processing method according to aspect 3 of the present disclosure, in aspect 1 or 2, in the step (d), the new Q&A data is regenerated in the third large-scale language model by the step (b).

[0102] According to the above configuration, new Q&A data is also generated by a large-scale language model, so that the information processing method according to aspect 3 of the present disclosure can improve answer accuracy.

[0103] The third large-scale language model may be the same as or different from the first large-scale language model and / or the second large-scale language model.

[0104] In the information processing method according to aspect 4 of the present disclosure, in aspect 1 or 2, in the step (d), the user who has referred to the first determination result regenerates the new Q&A data.

[0105] According to the above configuration, the information processing method according to the fourth aspect of the present disclosure can regenerate Q&A data that meets the user's wishes, thereby increasing the user's satisfaction.

[0106] An information processing method according to a fifth aspect of the present disclosure is any one of the first to fourth aspects, wherein the first determination result and the second determination result include at least one of a determination value and a determination reason regarding the accuracy.

[0107] According to the above configuration, the first judgment result and the second judgment result include a quantitative criterion, that is, a judgment value, and / or a qualitative reason, that is, a judgment reason. As a result, the information processing method according to the fifth aspect of the present disclosure is expected to further improve the accuracy of answers.

[0108] An information processing method according to aspect 6 of the present disclosure is any one of aspects 1 to 5, wherein the viewpoints include at least one of the title, summary, type of figure / table / text, conclusion, background, and purpose of the target data.

[0109] According to the above-described configuration, by setting many viewpoint items, it is possible to generate Q&A data with higher accuracy.

[0110] In the information processing method according to any one of the first to sixth aspects of the present disclosure, the step (g) is performed by a fourth large-scale language model.

[0111] According to the above configuration, the accuracy of the second determination result can be further improved.

[0112] An information processing method according to aspect 8 of the present disclosure is any of aspects 1 to 7, in which, in step (a), each of the multiple target data is classified by component, and at least one of the first judgment result and the second judgment result includes a judgment result for each of the component.

[0113] According to the above-described configuration, a judgment result can be obtained for each component, making it easier to find clues as to which component should be modified to improve the response accuracy of the RAG system.

[0114] An information processing method according to a ninth aspect of the present disclosure is the same as in the eighth aspect, in which the components include at least one of text, a graph, a table, and a figure other than a graph.

[0115] The components include any category that the user has arbitrarily set in the prompt. As an example, the components may be text, graphs, tables, and figures other than graphs. By adopting the above configuration, it becomes easier to improve the accuracy of answers given by the RAG system.

[0116] The information processing method according to a tenth aspect of the present disclosure is any one of the first to ninth aspects, further including (h) outputting a recommendation according to the second determination result.

[0117] According to the above configuration, after performing accuracy verification of the RAG system using Q&A data, recommendations based on the second judgment result are output, thereby making it possible to understand how to further improve the answer accuracy of the RAG system using the second judgment result as material.

[0118] An information processing method according to aspect 11 of the present disclosure, in any of aspects 1 to 10, further includes: (i) a step of performing optimization of the RAG system based on the recommendation when the second judgment result does not satisfy a predetermined criterion.

[0119] According to the above configuration, after performing accuracy verification on the RAG system using Q&A data, a recommendation is output according to the second judgment result. Then, by executing optimization of the RAG system based on the recommendation, it is possible to further improve the answer accuracy of the RAG system using the second judgment result as a material.

[0120] An information processing method according to aspect 12 of the present disclosure is, in aspect 11, wherein the optimization relates to changing settings of the RAG system.

[0121] According to the above-described configuration, the answer accuracy of the RAG system can be further improved by changing the settings of the RAG system, such as changing the type of large-scale language model used in the RAG system and tuning parameters.

[0122] An information processing method according to aspect 13 of the present disclosure is, in aspect 11, wherein the optimization relates to modifying the target document.

[0123] According to the above-mentioned configuration, the accuracy of the answer of the RAG system can be further improved by modifying the target sentence, for example by reducing the number of symbols used in the target sentence.

[0124] An information processing program according to a fourteenth aspect of the present disclosure is a computer-executable information processing program, comprising: (a) a step of generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) a step of generating Q&A data, which is a pair of a question and an answer related to the target data, in a first large-scale language model using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) a step of outputting a first judgment result regarding accuracy of the Q&A data for the target document; (d) a step of providing regenerated new Q&A data to step (c) when the first judgment result does not satisfy a predetermined criterion; (e) a step of executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; and (f) a step of extracting the first question from among a pair of a first question and a first answer to the first question included in the Q&A data after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, and (g) outputting a second determination result regarding the accuracy of the second answer to the first answer from the RAG system.

[0125] According to the above configuration, the information processing program according to aspect 14 of the present disclosure can achieve the same effects as the information processing method according to aspect 1 of the present disclosure.

[0126] An information processing system according to aspect 15 of the present disclosure is an information processing system for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system, and includes: a perspective generation unit that generates a perspective for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of a question and an answer related to the target data, using a first large-scale language model using the generated perspective and target data among the plurality of target data that corresponds to the perspective; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined criterion; an answer acquisition unit that inputs the first question, of a pair of a first question and a first answer to the first question included in the Q&A data, into the RAG system and acquires a second answer to the first question from the RAG system after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer.

[0127] According to the above configuration, the information processing system according to aspect 15 of the present disclosure can achieve the same effects as the information processing method according to aspect 1 of the present disclosure.

[0128] An information processing system according to aspect 16 of the present disclosure, in accordance with aspect 15 above, is provided with a reception unit that receives at least one of a user's change to a predetermined prompt to be input to a large-scale language model used in the information processing system and the user's setting of a new prompt.

[0129] According to the above configuration, the information processing system according to the fifteenth aspect of the present disclosure can easily output the second determination result that reflects the user's intention.

[0130] (Additional Notes) The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0131] 1. Information Processing Systems 10 Server 20 User terminal 30 Administrator terminal 50 Chatbots 51 Reception 52 Answer presentation part 1011, 2011 Application Program 1012 Large-scale language model 1013 Target Documents 1014 Target Data 1015 Q&A Data 1020 Target document division unit 1021 Component classification part 1022 Viewpoint Generation Unit 1023 Q&A Data Generation Department 1024 1st output section 1025 Q&A Data Regeneration Department 1026 Answer acquisition part 1027 Second output section 1028 Reception 1029 Recommendation Output Unit 1030 Optimization Department

Claims

1. An information processing method for evaluating answer accuracy of a Retrieval Augmented Generation (RAG) system via a user terminal, comprising: The user terminal, (a) generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of a question and an answer related to the target data, in a first large-scale language model by using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) outputting a first determination result regarding the accuracy of the Q&A data for the target document to a second large-scale language model; (d) if the first judgment result does not satisfy a predetermined criterion, providing the regenerated new Q&A data to step (c); (e) executing the steps (b) to (d) on all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputting the first question, of a pair of a first question and a first answer to the first question included in the Q&A data, into the RAG system, and causing the RAG system to provide a second answer to the first question; (g) outputting a second determination result regarding the accuracy of the second response to the first response to a fourth large-scale language model.

2. The information processing method of claim 1, wherein the second large-scale language model is different from the first large-scale language model.

3. The information processing method according to claim 1 , wherein in the step (d), the new Q&A data is regenerated by the third large-scale language model in the step (b).

4. The information processing method according to claim 1 , wherein in the step (d), a user who has referred to the first determination result is prompted to regenerate the new Q&A data.

5. The information processing method according to claim 1 , wherein the first determination result and the second determination result include at least one of a determination value and a determination reason regarding the accuracy.

6. The information processing method according to claim 1 , wherein the viewpoints include at least one of a title, an abstract, a type of figure / table / text, a conclusion, a background, and a purpose of the target data.

7. In the step (a), each of the plurality of target data is classified into components; The information processing method according to claim 1 , wherein at least one of the first determination result and the second determination result includes a determination result for each of the components.

8. The information processing method according to claim 7 , wherein the components include at least one of text, a graph, a table, and a diagram other than a graph.

9. The information processing method according to claim 1 , further comprising: (h) outputting a recommendation according to the second determination result.

10. The information processing method according to claim 9 , further comprising: (i) a step of executing optimization of the RAG system based on the recommendation when the second determination result does not satisfy a predetermined criterion.

11. The information processing method according to claim 10 , wherein the optimization relates to changing settings of the RAG system.

12. The information processing method according to claim 10 , wherein the optimization relates to modification of the target document.

13. A computer-executable information processing program, (a) generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of a question and an answer related to the target data, in a first large-scale language model by using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) outputting a first determination result regarding the accuracy of the Q&A data for the target document by a second large-scale language model; (d) if the first judgment result does not satisfy a predetermined criterion, providing the regenerated new Q&A data to step (c); (e) executing the steps (b) to (d) on all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputting the first question, among a pair of a first question and a first answer to the first question included in the Q&A data, into a Retrieval Augmented Generation (RAG) system, and returning a second answer to the first question from the RAG system; and (g) outputting a second determination result regarding the accuracy of the second answer to the first answer using a fourth large-scale language model.

14. An information processing system for evaluating the accuracy of answers from a Retrieval Augmented Generation (RAG) system, comprising: a viewpoint generation unit for generating a viewpoint for each of a plurality of target data pieces obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of a question and an answer related to the target data, by using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data, using a first large-scale language model; a first output unit that outputs a first determination result regarding the accuracy of the Q&A data for the target document by a second large-scale language model; a Q&A data regeneration unit that regenerates new Q&A data until the first determination result satisfies a predetermined criterion; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputs the first question from a pair of a first question and a first answer to the first question included in the Q&A data into the RAG system and acquires a second answer to the first question from the RAG system; An information processing system comprising: a second output unit that outputs a second determination result regarding the accuracy of the second answer relative to the first answer using a fourth large-scale language model.

15. 15. The information processing system according to claim 14, further comprising a reception unit that receives at least one of a user's change to a predetermined prompt to be input to a large-scale language model used in the information processing system and a setting by the user of a new prompt.

Images