Information processing method, information processing program, and information processing system

Abstract

Evaluate the accuracy of the RAG system's answers. [Solution] The information processing method disclosed herein includes the steps of: (f) after all Q&A data corresponding to multiple target data meets a predetermined standard, a user terminal inputs a first question from a pair of a first question and a first answer to the first question contained in the Q&A data into a RAG system and causes the RAG system to provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing program, and an information processing system. [Background technology]

[0002] Retrieval Augmented Generation (RAG) has been known for some time (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7527581 Summary of the Invention [Means for solving the problem]

[0004] An information processing method according to one embodiment of the present disclosure is an information processing method for evaluating the accuracy of answers of a RAG (Retrieval Augmented Generation) system via a user terminal, in which the user terminal (a) generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generates Q&A data, which are pairs of questions and answers related to the target data, in a first large-scale language model using the generated viewpoints and target data corresponding to the viewpoints among the plurality of target data; (c) outputs a first determination result regarding the accuracy of the Q&A data for the target document; and (d) regenerates the Q&A data if the first determination result does not satisfy a predetermined standard. (e) executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data has satisfied the predetermined criteria, inputting the first question of a pair of a first question and a first answer to the first question contained in the Q&A data into the RAG system and causing the RAG system to provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

[0005] An information processing program according to an embodiment of the present disclosure is a computer-executable information processing program, the information processing program including: (a) generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of a question and an answer related to the target data, in a first large-scale language model using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) outputting a first determination result regarding the accuracy of the Q&A data for the target document; (d) providing regenerated new Q&A data to step (c) if the first determination result does not satisfy a predetermined standard; (e) executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; and (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined standard, extracting the first question from among pairs of a first question and a first answer to the first question included in the Q&A data using Retrieval Augmented Reality (RAG). and (g) outputting a second determination result regarding the accuracy of the second answer to the first answer.

[0006] An information processing system according to one embodiment of the present disclosure is an information processing system for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system, and includes: a viewpoint generation unit that generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model, using the generated viewpoints and target data among the plurality of target data that corresponds to the viewpoints; a first output unit that outputs a first determination result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first determination result satisfies a predetermined criterion; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputs the first question, of a pair of a first question and a first answer to the first question included in the Q&A data, into the RAG system and acquires a second answer to the first question from the RAG system; and a second output unit that outputs a second determination result regarding the accuracy of the second answer to the first answer. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of an information processing system according to the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of a server according to the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating a functional configuration of a user terminal according to the present disclosure. [Figure 4] 10 is a flowchart illustrating an example of a flow of information processing according to the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of a chatbot according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. For ease of understanding, the background and problems of the present disclosure will be described first, and then the details of the present disclosure will be described. <Search Expansion Generation> Retrieval-Augmented Generation (RAG) is a method of searching external knowledge databases in real time and having generative AI generate answers based on that information. While typical generative models rely on learned knowledge, RAG excels in tasks where up-to-date information or accuracy is important. RAG augments answers based on search results, providing more accurate and updated content. RAG has the following advantages over generative AI that uses fine tuning:

[0009] · Reflection of the latest information: Whereas fine-tuning uses historical data to update the model, RAG has access to new information in real time.

[0010] Reduced data update load: Fine tuning requires periodic model updates, but RAG retrieves the latest information each time it is searched, eliminating the need for frequent retraining.

[0011] · Flexibility: RAG can generate answers that are applicable across a wide range of topics, rather than being topic-specific.

[0012] On the other hand, RAG may contain false information (hallucination) that is not included in internal documents, so there is still room for improvement in the accuracy of the answers. <Outline of Information Processing System 1> The information processing system 1 in the present disclosure evaluates the answer accuracy of a Retrieval Augmented Generation (RAG) system. The information processing system 1 can also improve the answer accuracy of the RAG system based on recommendations according to the evaluation results. The information processing system 1 may be used in a chatbot that outputs answers corresponding to questions received online from users.

[0013] The information processing system 1 can be used in various fields, such as aircraft, railways, ships, hospitals, banks, and hotels.

[0014] More specifically, the information processing system 1 is an information processing system for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system, and includes: a viewpoint generation unit that generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model, using the generated viewpoints and target data among the plurality of target data that corresponds to the viewpoints; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined criterion; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfies the predetermined criterion, inputs the first question, of the pair of a first question and a first answer to the first question included in the Q&A data, into the RAG system and acquires a second answer to the first question from the RAG system; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer.

[0015] According to the above configuration, the information processing system 1 can output the answer accuracy of the RAG system as the second judgment result. By having the user confirm the answer accuracy of the RAG system as the second judgment result, the information processing system 1 can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy.

[0016] Furthermore, according to the above configuration, unlike conventional Q&A reference chatbots, the information processing system 1 can automatically generate high-quality Q&As simply by a user storing internal company documents, etc., in a predetermined folder (such as a data store.) Furthermore, the information processing system 1 can also perform accuracy evaluation side-by-side using multiple generation AIs, thereby reducing the possibility of hallucination and improving the accuracy of RAG responses.

[0017] The information processing system 1 will be described below with reference to the drawings.

[0018] FIG. 1 is a block diagram showing the functional configuration of an information processing system 1 according to the present disclosure.

[0019] FIG. 2 is a block diagram showing the functional configuration of the server 10 according to the present disclosure.

[0020] FIG. 3 is a block diagram showing the functional configuration of the user terminal 20 according to the present disclosure. <Configuration of Information Processing System 1> Referring to FIG. 1, an information processing system 1 includes a server 10, a user terminal 20, and an administrator terminal 30, which are connected via a network N.

[0021] The server 10 and the user terminal 20 may be realized as an integrated configuration as a single information processing terminal device, but in the following description they will be described as separate devices.

[0022] The network N may be a mobile communication system such as 3G, 4G, 5G, or 6G, LTE (Long Term Evolution), Wi-Fi (registered trademark), the Internet, or an in-house LAN.

[0023] <Basic configuration of Server 10> The server 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, and a communication IF 14. These are electrically connected to each other via a communication bus.

[0024] The processor 11 executes instructions written in a program and is configured with an arithmetic unit, a register, and / or peripheral circuits.

[0025] The main storage device 12 temporarily stores programs and data to be processed by the programs, and is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0026] The auxiliary storage device 13 stores data and programs, and is, for example, a flash memory, a hard disk drive (HDD), or a magneto-optical disk.

[0027] The communication IF 14 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

[0028] 2, the server 10 includes a storage unit 101 and a control unit 102. The storage unit 101 and the control unit 102 will be described below.

[0029] The storage unit 101 stores an application program 1011 , a large-scale language model 1012 , a target document 1013 , target data 1014 , and Q&A data 1015 .

[0030] The application program 1011 is a program for causing the control unit 102 of the server 10 to function as each functional unit. The application program 1011 includes an application such as a web browser application.

[0031] The large-scale language model 1012 stores multiple large-scale language models used in the information processing system 1. The multiple large-scale language models may be at least partially different from each other. The multiple large-scale language models being different from each other means that they have different architectures or hyperparameters, etc., including differences in layer structure, number of neurons, training method, and / or optimization algorithm. Furthermore, the datasets and / or tuning methods used for learning may also be different from each other, which may result in differences in the quality and / or characteristics of the generated text. This allows different responses to be expected for each model, even for the same task.

[0032] The large-scale language model 1012 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of a user terminal 20 (described later) or in another storage device.

[0033] Next, the target document 1013 will be described. Generally, in search expansion generation, reference information is searched for from a predetermined data source based on instruction information that instructs the execution of a task, and a prompt including the search results for the instruction information and the reference information is input to a large-scale language model. The large-scale language model executes the task by referring to reference information that is not included in the training data. In this way, search expansion generation improves the problem of misinformation (hallucination) that has been a problem in the answers of conventional generative AI.

[0034] The target document 1013 may be considered to correspond to the reference information described above. The target document 1013 may be internet information or the latest information, databases, internal documents, etc. stored within an organization (such as a company), and the type is not limited. The target document 1013 may be stored in various formats, such as Word, PowerPoint, PDF, SQL format, CSV format, JSON format, XML format, and binary format. For ease of explanation, the following description will be given assuming that the target document 1013 is a PDF-format document (target document) stored within a company.

[0035] The target document 1013 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of the user terminal 20 described below, or in another storage device.

[0036] Next, we will explain the target data 1014. The target data 1014 is a plurality of data pieces obtained by dividing the target document into predetermined units. The predetermined units may be determined as appropriate, such as one page, two pages, or a chapter within a document.

[0037] The Q&A data 1015 is a data set that associates expected questions with model answers for the target data 1014. The Q&A data 1015 is created to determine how close the answers obtained by giving questions to the system to be evaluated are to the model answers.

[0038] The Q&A data 1015 typically includes "Questions" that a user may ask the system and "Answers" that are model answers to those questions. Furthermore, the Q&A data 1015 may include "Categories" that indicate the themes or topics to which the questions belong, and / or "Tags" that are keywords that aid in searching or filtering.

[0039] The Q&A data 1015 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of the user terminal 20 described below, or in another storage device.

[0040] 2, the control unit 102 includes a target document division unit 1020, a component classification unit 1021, a viewpoint generation unit 1022, a Q&A data generation unit 1023, a first output unit 1024, a Q&A data regeneration unit 1025, an answer acquisition unit 1026, and a second output unit 1027. The control unit 102 may also include a reception unit 1028.

[0041] The target document division unit 1020 divides the target document into predetermined units. The predetermined units may be determined as appropriate, such as one page, two pages, or a chapter within the document. In the following description, the target document division unit 1020 will be described as dividing the target document stored in a predetermined folder into pages.

[0042] The target document divider 1020 may divide the target document into pages using, for example, an online tool (such as Smallpdf or ILovePDF), a function built into Adobe Acrobat, or a program such as Python. Specifically, the target document divider 1020 divides the target document 1013 into pages and stores multiple pieces of target data 1014 in the storage unit 101. The target document divider 1020 may save the target data 1014 in a storage device different from the storage unit 101 of the server 10. When the target document 1013 is one page, it may be understood that the target document 1013 has been divided into one piece by the target document divider 1020.

[0043] The component classification unit 1021 classifies the components included in the target data 1014 obtained by dividing the target document into pages by the target document division unit 1020 into categories such as "text only," "graph," "table," and "figure other than graph." The component classification unit 1021 may be realized by a large-scale language model such as a GPT (Generative Pretrained Transformer), but there are no restrictions on the scale or structure of the model.

[0044] The control unit 102 does not necessarily have to include the component classification unit 1021. However, by including the component classification unit 1021, the control unit 102 can cause the first output unit 1024 (described later) to output the first judgment result including the score for each component. This allows the control unit 102 to provide a clue as to which classification item should be corrected to improve the accuracy of the Q&A data 1015. For example, when the first output unit 1024 outputs the first judgment result in a range of 1 to 10, the scores may be "8.56" for graphs, "8.30" for tables, "8.83" for figures other than graphs, and "6.75" for plain text only. In this case, the score for plain text only is the lowest, providing a clue that correcting only the plain text will improve the accuracy of the Q&A data 1015. Similarly, by including the component classification unit 1021, the control unit 102 can also include the judgment result for each component in the second judgment result (described later).

[0045] The component classification unit 1021 may be, for example, a recurrent neural network (RNN) such as a convolutional neural network (CNN), a long-short-term memory (LSTM), or a gated recurrent unit (GRU), or another language model that combines one or more models such as a Transformer model. The component classification unit 1021 may be realized by, for example, ChatGPT, Bard, Gemini, or Claude. The same applies to the viewpoint generation unit 1022, the Q&A data generation unit 1023, the first output unit 1024, the Q&A data regeneration unit 1025, the second output unit 1027, the recommendation output unit 1029, and the optimization unit 1030, which will be described later.

[0046] The viewpoint generation unit 1022 generates a viewpoint for each of a plurality of target data 1014 obtained by dividing the target document 1013 into predetermined units (one page). The viewpoint includes at least one of the title, abstract, type of figure / table / text, conclusion, background, and purpose of the target data.

[0047] The viewpoint generation unit 1022 may be realized by a large-scale language model such as a Generative Pretrained Transformer (GPT), but there is no limitation on the scale or structure of the model.

[0048] The viewpoint generation unit 1022 inputs, for example, the following prompt into the large-scale language model: Example prompts "Based on what you can read from the data below, what can you say about it?" In response to this prompt, the viewpoint generator 1022 obtains, for example, the following output: Perspective: Performance by product Explanation: Trends in sales target achievement for each product, the most stable products and the most volatile products. The prompt may be set as a default, or the receiving unit 1028 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the viewpoint generating unit 1022 may input the received prompt into the large-scale language model.

[0049] The Q&A data generation unit 1023 uses the perspective generated by the perspective generation unit 1022 and target data 1014 corresponding to the perspective among the plurality of target data 1014 to generate Q&A data 1015 which is a pair of a question and an answer related to the target data 1014. The Q&A data generation unit 1023 may be realized by a large-scale language model. The Q&A data generation unit 1023 stores the generated Q&A data 1015 in the storage unit 101. The Q&A data generation unit 1023 may output the generated Q&A data 1015 to the first output unit 1024.

[0050] The first output unit 1024 outputs a first determination result regarding the accuracy of the Q&A data 1015 with respect to the target data 1014. The first output unit 1024 may be realized by a large-scale language model. The first determination result includes at least one of a determination value regarding the accuracy and a determination reason.

[0051] A good first determination result is more likely to be output when the same large-scale language model is used by the Q&A data generation unit 1023 and the first output unit 1024. Therefore, the first output unit 1024 may use, but is not limited to, a second large-scale language model that is different from the Q&A data generation unit 1023 (first large-scale language model).

[0052] The first output unit 1024 receives, for example, the following prompt: Example prompts "Please refer to the data below to determine the accuracy of your answers to the questions. # Question: ${QUESTION} # Answer: ${ANSWER} Please rate your assessment on a scale of 1 to 10. If there are any discrepancies in the information on which your answer is based, please rate it 5 points or lower. If the information is accurate but the conclusion is simply missing, please rate it 7 points or higher. Please indicate your assessment results with a "score" and "reason." Then, the first output unit 1024 outputs, for example, the following first determination result. ·Example of first judgment result "Score: 3" "Reason: The first half of your answer is correct, but the second half contradicts the first half. Therefore, I've given it 3 points." As described above, the first output unit 1024 may output a first determination result including a score for each component element. For example, the first output unit 1024 may output a first determination result including the scores "8.56" for graphs, "8.30" for tables, "8.83" for figures other than graphs, and "6.75" for plain text only. In this way, the first output unit 1024 may output a first determination result including digits after the decimal point.

[0053] The prompt may be set as a default, or the receiving unit 1028 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the first output unit 1024 may input the received prompt to the large-scale language model.

[0054] The first output unit 1024 may output the first determination result to the Q&A data regeneration unit 1025, or may store the first determination result in the storage unit 101 or an external storage device.

[0055] The Q&A data regeneration unit 1025 generates new Q&A data 1015 when the first determination result output by the first output unit 1024 does not satisfy a predetermined standard. The Q&A data regeneration unit 1025 may be realized by a large-scale language model. The Q&A data regeneration unit 1025 uses the viewpoint generated by the viewpoint generation unit 1022 and target data 1014 corresponding to the viewpoint among the plurality of target data 1014 to generate Q&A data 1015 which is a pair of a question and an answer related to the target data 1014. The Q&A data regeneration unit 1025 (third large-scale language model) may be the same as the first large-scale language model and / or the second large-scale language model described above, or may be a different large-scale language model.

[0056] In the aforementioned prompt in which a score is assigned in the range of 1 to 10, for example, the predetermined standard may be "8," and a score of 8 or higher is considered to satisfy the predetermined standard. The predetermined standard may be determined arbitrarily. The "predetermined standard" shown here is merely an example, and the present disclosure is not limited thereto. For example, the "predetermined standard" may be set appropriately, such as "4" when a score is assigned in the range of 1 to 5. Alternatively, the "predetermined standard" may be set as an average score of X points or higher for all Q&A data 1015. Alternatively, the "predetermined standard" may be set as a combination of X points or higher for graphs, Y points or higher for tables, and Z points or higher for text.

[0057] In this manner, in the present disclosure, the "predetermined standard" may be set in various ways. Also, in the present disclosure, for example, when the average score for all Q&A data 1015 is X points or more, or when a standard such as X points or more in the case of a graph or Y points or more in the case of a table is met, it can be considered that "all of the Q&A data 1015 corresponding to the plurality of target data 1014 meet the predetermined standard."

[0058] The Q&A data regeneration unit 1025 outputs the regenerated new Q&A data 1015 to the first output unit 1024. The Q&A data regeneration unit 1025 may store the regenerated new Q&A data 1015 in the storage unit 101. The first output unit 1024 outputs a first determination result regarding the accuracy of the new Q&A data 1015 for the target document 1013 by the method described above.

[0059] The Q&A data regeneration unit 1025 may cause the user, having referred to the first determination result, to generate new Q&A data 1015. For example, the Q&A data regeneration unit 1025 outputs a screen or a sound instructing the user to generate new Q&A data 1015, and accepts the new Q&A data 1015 from the user. Then, the Q&A data regeneration unit 1025 outputs the new Q&A data 1015 accepted from the user to the first output unit 1024. The Q&A data regeneration unit 1025 can also be realized by such a configuration.

[0060] The above operation may be performed for all of the Q&A data 1015 corresponding to the plurality of target data 1014 .

[0061] After all of the Q&A data corresponding to the multiple target data satisfy a predetermined standard, the answer acquisition unit 1026 operates as follows. Specifically, of a pair of a first question and a first answer to the first question included in the Q&A data, the answer acquisition unit 1026 inputs the first question into the RAG system and acquires the second answer to the first question from the RAG system. The RAG system is a RAG system that is operated in a production environment, a test environment, a development environment, or the like.

[0062] The second output unit 1027 outputs a second determination result regarding the accuracy of the second answer relative to the first answer. The second output unit 1027 may be realized by a large-scale language model (fourth large-scale language model). The second determination result includes at least one of a determination value regarding the accuracy and a determination reason.

[0063] The second output unit 1027 receives the following prompt, for example: Example prompts You are the quality judge for the Q&A system. Please evaluate the system's answer to the expected question (answer 2) by carefully comparing it with the model answer (answer 1). Pay close attention to the differences between the model answer and the system's answer when explaining numbers and units. # Expected question: ${QUESTION} # Model answer: ${{REF_ANSWER} # System answer: ${SYSTEM_ANSWER} Please rate your results on a scale of 1 to 10, with the model answer being 10 points. If there are even partial differences between the system answer and the information on which the model answer is based, please rate it 5 points or lower. If the information is accurate but the conclusion is simply missing, please rate it 7 points or higher. Then, the second output unit 1027 outputs, for example, the following second determination result. ·Second judgment result example 1 "Score: 5" "Reason: In addition to two correct answers, there is one incorrect answer, so I gave it 5 points." ·Second judgment result example 2 "Score: 3" "Reason: The first half of your answer is correct, but the second half contradicts the first half. Therefore, I gave it 3 points." The second output unit 1027 may output a second determination result including a score for each component element. For example, the second output unit 1027 outputs a second determination result including the scores "8.68" for graphs, "7.75" for tables, "8.58" for figures other than graphs, and "7.56" for plain text only. In this way, the second output unit 1027 may output a second determination result including digits after the decimal point.

[0064] The second output unit 1027 may store the second determination result in the storage unit 101 or an external storage device.

[0065] The prompt may be set as a default, or the receiving unit 1028 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the second output unit 1027 may input the received prompt to the large-scale language model.

[0066] According to the above configuration, the information processing system 1 can output the answer accuracy of the RAG system as the second judgment result. By having the user confirm the answer accuracy of the RAG system as the second judgment result, the information processing system 1 can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy.

[0067] The fourth large-scale language model may be the same as the first, second, or third large-scale language model, or another large-scale language model may be used. The fourth large-scale language model may also be composed of multiple large-scale language models. <Recommendation function> Referring back to FIG. 2, the control unit 102 may further include a recommendation output unit 1029 and an optimization unit 1030.

[0068] The recommendation output unit 1029 outputs a recommendation according to the second determination result. The recommendation output unit 1029 may be realized by a large-scale language model such as a Generative Pretrained Transformer (GPT), but there are no limitations on the scale or structure of the model.

[0069] As an example, the recommendation output unit 1029 receives the following prompt based on the second determination result output by the second output unit 1027. Example prompts Please explain the categories that received high and low ratings from the test results below, and make recommendations for the areas that received low ratings. # Test results Model used: gemini-pro-1.0 Layout Parser: OFF Plain text: 7.5 Image data: 7.2 Table data: 3.4" Then, the recommendation output unit 1029 outputs, for example, the following recommendation. Example of a rating comment "Evaluation Results: The test results show that answers regarding plain text and image data are appropriate and highly rated, while answers regarding table data are not rated as desirable. Recommendations: 1) Setting recommendation: The model currently being used is an old model, so we recommend using the latest stable model. 2) We recommend turning on the Layout Parser function. 3) Document content recommendations: We recommend not using symbols in table data. We recommend that table data be structured as simply as possible. In this way, the recommendation output unit 1029 can analyze the tendency of results with low scores in the second judgment result and output a recommendation such as, for example, "The model that is most suitable for the current document is Gemini 1.5 pro." This can be obtained by instructing the recommendation output unit 1029, which is realized by a large-scale language model, to output a recommendation of the analysis result. These are just examples, and the recommendation output unit 1029 can output recommendations for the setting values ​​of the RAG system itself, such as the type of large-scale language model or the parameters to be used. The recommendation output unit 1029 can also output recommendations for the content of the target document (or target data) that is referenced by the RAG system. In this way, the recommendation output unit 1029 can output recommendations over a wide range of content (scope).

[0070] If the second determination result does not satisfy a predetermined standard, the optimization unit 1030 performs optimization based on the recommendation output by the recommendation output unit 1029. For example, as an example of optimization, the optimization unit 1030 performs optimization based on the recommendation output by the recommendation output unit 1029. For example, the optimization unit 1030 performs optimization based on the recommendation output by the recommendation output unit 1029. pro (selection of the optimal LLM), or changing (tuning) the parameters of the large-scale language model used by the RAG system, in order to improve the second determination result. Alternatively, as another example, the optimization unit 1030 may improve the second determination result by performing processing such as changing the writing style of the target document referenced by the RAG system, for example, by avoiding the use of symbols in the table data used.

[0071] When the second judgment result does not satisfy the predetermined standard, for example, a score of "7 or more" in the second judgment result is defined as the predetermined standard. In this case, "Table data: 3.4" does not satisfy the predetermined standard, while "Plain text: 7.5" and "Image data: 7.2" do. The optimization unit 1030 determines that "Table data: 3.4" does not satisfy the predetermined standard and executes the optimization process described above. When all scores related to the second judgment result satisfy the predetermined standard, the optimization unit 1030 may terminate the process.

[0072] After performing the optimization process based on the recommendation, the optimization unit 1030 causes the answer acquisition unit 1026 to provide (acquire) a second answer. The subsequent process is as described above.

[0073] By having the above configuration, the information processing system 1 can improve the response accuracy of the RAG system. <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service (e.g., a chatbot). The user terminal 20 receives, for example, an input operation from a user who uses a chatbot, and operates the control unit 102 of the server 10. The user terminal 20 may be, for example, a desktop personal computer (PC), a laptop PC, a mobile terminal such as a smartphone or a tablet, an HMD (Head Mount Display), or a wearable terminal such as a wristwatch terminal.

[0074] The user terminal 20 includes a storage unit 201 , a control unit 202 , an input unit 203 , and an output unit 204 . <Storage unit 201 of user terminal 20> The storage unit 201 of the user terminal 20 stores an application program 2011. The storage unit 201 may also store a user ID 2012.

[0075] The application program 2011 may be pre-stored in the storage unit 201. The application program 2011 may be downloaded from a web server operated by a service provider via a communication IF. The application program 2011 may include an application such as a web browser application.

[0076] The user ID 2012 is the user's account ID. The user transmits the user ID 2012 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2012 and provides a service to the user. If a service (e.g., a chatbot) does not require a user ID, the storage unit 201 does not need to store the user ID 2012. <Control Unit 202 of User Terminal 20> The control unit 202 of the user terminal 20 includes an input control unit 2021 and an output control unit 2022. The control unit 202 executes an application program 2011 stored in the storage unit 201 to realize each functional unit. <Input unit 203 of user terminal 20> The input unit 203 of the user terminal 20 may include a camera 2031, a microphone 2032, a keyboard 2033, a touch panel 2064, and the like. <Output unit 204 of user terminal 20> The output unit 204 of the user terminal 20 may include a display 2041 and a speaker 2042 . <Configuration of administrator terminal 30> 1, the administrator terminal 30 is an information processing device operated by an administrator of a service (e.g., a chatbot). The administrator is also the operator of the information processing system 1. Through the administrator terminal 30, the administrator builds a large-scale language model to be used in the information processing system 1, or determines or modifies prompts to be input to the large-scale language model.

[0077] The administrator terminal 30 may be, for example, a desktop personal computer (PC), a laptop PC, a mobile terminal such as a smartphone or a tablet, an HMD (Head Mount Display), or a wearable terminal such as a wristwatch terminal.

[0078] <Example of processing by information processing device> Next, the flow of processing by the information processing system 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the flow of information processing according to the present disclosure.

[0079] In S2, the target document dividing unit 1020 divides the target document into predetermined units.

[0080] In S3, the component classification unit 1021 classifies the components contained in the target data 1014 obtained by dividing it into pages by the target document division unit 1020 into, for example, "text only," "graph," "table," and "figure other than graph," etc.

[0081] In S4, the viewpoint generating unit 1022 generates a viewpoint for each of a plurality of target data pieces obtained by dividing the target document into predetermined units (for example, one page).

[0082] In S6, the Q&A data generation unit 1023 generates Q&A data, which is a pair of questions and answers related to the target data, using the perspective generated by the perspective generation unit 1022 and target data corresponding to the perspective among the multiple target data.

[0083] In S8, the first output unit 1024 outputs a first determination result regarding the accuracy of the Q&A data for the target document.

[0084] If the first determination result does not satisfy the predetermined standard (NO in S10), the process proceeds to S12. In S12, the Q&A data regeneration unit 1025 regenerates new Q&A data. Then, in S8, the first output unit 1024 outputs the first determination result regarding the accuracy of the new Q&A data for the target document. The first determination result may include a score for each component.

[0085] In S10, if the first determination result satisfies the predetermined criterion (YES in S10), the process proceeds to S 14. After all of the Q&A data corresponding to the plurality of target data satisfies the predetermined criterion, in S14, the answer acquisition unit 1026 inputs the first question, of the pair of the first question and the first answer to the first question included in the Q&A data, to the RAG system, and acquires the second answer to the first question from the RAG system.

[0086] In S16, the second output unit 1027 outputs a second determination result regarding the accuracy of the second answer relative to the first answer. The second determination result may include a score for each component element.

[0087] In S18, the recommendation output unit 1029 outputs a recommendation according to the second determination result.

[0088] In S20, if the second determination result satisfies the predetermined criterion (YES in S20), the processing ends. In S20, if the second determination result does not satisfy the predetermined criterion (NO in S20), in S22, the optimization unit 1030 executes optimization based on the recommendation output by the recommendation output unit 1029. After executing the optimization process based on the recommendation, the optimization unit 1030 once again causes the second output unit 1027 to return (obtain) a second answer (proceed to S8).

[0089] <Chatbot> Next, a chatbot 50 according to the present disclosure will be described with reference to Figure 5. Figure 5 is a schematic diagram of a chatbot 50 according to the present disclosure.

[0090] The chatbot 50 is an example of a RAG system in which answer accuracy has been improved by the information processing system 1 according to the present disclosure. The chatbot 50 includes a reception unit 51 that receives questions from users and an answer presentation unit 52 that presents answers to the questions. This allows the chatbot 50 to provide the user with a RAG system with improved answer accuracy.

[0091] <Software implementation example> The functions of the information processing system 1 (hereinafter referred to as the "device") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in the control unit 102).

[0092] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0093] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0094] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0095] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server). <Summary> The present disclosure describes at least the following aspects.

[0096] An information processing method according to a first aspect of the present disclosure is an information processing method for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system via a user terminal, in which the user terminal (a) generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generates Q&A data, which is a pair of questions and answers related to the target data, in a first large-scale language model using the generated viewpoints and target data corresponding to the viewpoints among the plurality of target data; (c) outputs a first determination result regarding the accuracy of the Q&A data for the target document; and (d) regenerates the Q&A data if the first determination result does not satisfy a predetermined standard. (e) executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data has satisfied the predetermined criteria, inputting the first question of a pair of a first question and a first answer to the first question contained in the Q&A data into the RAG system and causing the RAG system to provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

[0097] According to the above configuration, the information processing method according to aspect 1 of the present disclosure regenerates Q&A data until a first determination result regarding the accuracy of the Q&A data for a target document satisfies a predetermined standard. The information processing method according to aspect 1 of the present disclosure then executes steps (b) to (d) for all of the Q&A data corresponding to multiple target documents. The information processing method according to aspect 1 of the present disclosure then positions steps (a) to (e) as advance preparation, and, following this advance preparation, outputs a second determination result regarding the accuracy of the second answer to the first answer through steps (f) and (g).

[0098] According to the above configuration, the information processing method according to aspect 1 of the present disclosure can output the answer accuracy of the RAG system as the second judgment result. By having the user confirm the answer accuracy of the RAG system as the second judgment result, the information processing method according to aspect 1 of the present disclosure can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy.

[0099] In the information processing method according to aspect 2 of the present disclosure, in accordance with aspect 1, the step (c) is performed by a second large-scale language model different from the first large-scale language model.

[0100] According to the above configuration, the information processing method according to aspect 2 of the present disclosure can increase the possibility of outputting a first determination result with higher accuracy by using different large-scale language models in steps (b) and (c).

[0101] In the information processing method according to aspect 3 of the present disclosure, in aspect 1 or 2, in step (d), the third large-scale language model is caused to regenerate the new Q&A data by step (b).

[0102] According to the above configuration, new Q&A data is also generated using a large-scale language model, and therefore the information processing method according to aspect 3 of the present disclosure can improve answer accuracy.

[0103] The third large-scale language model may be the same as or different from the first large-scale language model and / or the second large-scale language model.

[0104] In the information processing method according to aspect 4 of the present disclosure, in aspect 1 or 2, in step (d), the user who has referred to the first determination result regenerates the new Q&A data.

[0105] According to the above configuration, the information processing method according to the fourth aspect of the present disclosure can regenerate Q&A data that meets the user's wishes, thereby increasing the user's satisfaction.

[0106] In the information processing method according to Aspect 5 of the present disclosure, in any one of Aspects 1 to 4, the first determination result and the second determination result include at least one of a determination value and a determination reason regarding the accuracy.

[0107] According to the above configuration, the first determination result and the second determination result include a quantitative criterion, i.e., a determination value, and / or a qualitative reason, i.e., a determination reason. As a result, the information processing method according to aspect 5 of the present disclosure is expected to further improve the accuracy of answers.

[0108] An information processing method according to aspect 6 of the present disclosure is any one of aspects 1 to 5, wherein the aspects include at least one of the title, abstract, type of figure / table / text, conclusion, background, and purpose of the target data.

[0109] According to the above configuration, by setting many viewpoint items, it is possible to generate Q&A data with higher accuracy.

[0110] An information processing method according to a seventh aspect of the present disclosure is any one of the first to sixth aspects, wherein step (g) is performed by a fourth large-scale language model.

[0111] According to the above configuration, the accuracy of the second determination result can be further improved.

[0112] An information processing method according to aspect 8 of the present disclosure is any of aspects 1 to 7, wherein in step (a), each of the plurality of target data is classified by component, and at least one of the first judgment result and the second judgment result includes a judgment result for each of the component.

[0113] According to the above configuration, it is possible to obtain a judgment result for each component, making it easier to find clues as to which component should be corrected to improve the accuracy of the RAG system's responses.

[0114] An information processing method according to a ninth aspect of the present disclosure is the information processing method of the eighth aspect, wherein the components include at least one of text, graphs, tables, and figures other than graphs.

[0115] The components include any category that the user selects in the prompt. For example, the components may be text, graphs, tables, and non-graph figures. This configuration helps improve the accuracy of the RAG system's answers.

[0116] The information processing method according to Aspect 10 of the present disclosure is the information processing method according to any one of Aspects 1 to 9, further including (h) outputting a recommendation according to the second determination result.

[0117] According to the above configuration, after verifying the accuracy of the RAG system using Q&A data, recommendations based on the second judgment result are output, thereby making it possible to understand how to further improve the response accuracy of the RAG system using the second judgment result as material.

[0118] The information processing method of aspect 11 of the present disclosure, in any of aspects 1 to 10, further includes the step of (i) performing optimization of the RAG system based on the recommendation when the second judgment result does not satisfy a predetermined criterion.

[0119] According to the above configuration, after verifying the accuracy of the RAG system using Q&A data, a recommendation is output according to the second judgment result. Then, by optimizing the RAG system based on the recommendation, the answer accuracy of the RAG system can be further improved using the second judgment result as material.

[0120] An information processing method according to aspect 12 of the present disclosure is, in aspect 11, wherein the optimization relates to changing the settings of the RAG system.

[0121] According to the above configuration, the accuracy of answers given by the RAG system can be further improved by changing the settings of the RAG system, such as changing the type of large-scale language model used in the RAG system or tuning parameters.

[0122] An information processing method according to aspect 13 of the present disclosure is the same as aspect 11, in which the optimization relates to modifying the target document.

[0123] According to the above configuration, the accuracy of the answers given by the RAG system can be further improved by correcting the target sentence, for example by reducing the number of symbols used in the target sentence.

[0124] An information processing program according to a fourteenth aspect of the present disclosure is a computer-executable information processing program, the information processing program including: (a) generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of a question and an answer related to the target data, in a first large-scale language model using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; (c) outputting a first determination result regarding the accuracy of the Q&A data for the target document; (d) providing regenerated new Q&A data to step (c) if the first determination result does not satisfy a predetermined standard; (e) executing steps (b) to (d) for all of the Q&A data corresponding to the plurality of target data; and (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined standard, extracting the first question from among pairs of a first question and a first answer to the first question included in the Q&A data using Retrieval Augmented Reality (RAG). and (g) outputting a second determination result regarding the accuracy of the second answer to the first answer.

[0125] According to the above configuration, the information processing program according to the fourteenth aspect of the present disclosure can achieve the same effects as the information processing method according to the first aspect of the present disclosure.

[0126] An information processing system according to aspect 15 of the present disclosure is an information processing system for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system, and includes: a viewpoint generation unit that generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that uses the generated viewpoints and target data among the plurality of target data corresponding to the viewpoints to generate Q&A data, which are pairs of questions and answers related to the target data, using a first large-scale language model; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined criterion; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criterion, inputs the first question, of pairs of a first question and a first answer to the first question included in the Q&A data, into the RAG system and acquires a second answer to the first question from the RAG system; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer.

[0127] According to the above configuration, the information processing system according to aspect 15 of the present disclosure can achieve the same effects as the information processing method according to aspect 1 of the present disclosure.

[0128] An information processing system according to aspect 16 of the present disclosure is, in aspect 15, provided with a reception unit that receives at least one of a user's change to a predetermined prompt to be input into a large-scale language model used in the information processing system and the user's setting of a new prompt.

[0129] According to the above configuration, the information processing system according to the fifteenth aspect of the present disclosure can easily output the second determination result that reflects the user's intention.

[0130] (Additional notes) The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0131] 1. Information Processing Systems 10 Servers 20 User terminal 30 Administrator terminal 50 Chatbots 51 Reception 52 Answer presentation part 1011, 2011 Application Program 1012 Large-scale language models 1013 Target Documents 1014 Target Data 1015 Q&A data 1020 Target document segmentation unit 1021 Component classification part 1022 Viewpoint Generation Unit 1023 Q&A Data Generation Department 1024 First output section 1025 Q&A Data Regeneration Department 1026 Answer acquisition part 1027 Second Output Unit 1028 Reception 1029 Recommendation Output Unit 1030 Optimization Department

Claims

1. A method for generating Q&A data for evaluating answer accuracy of a Retrieval Augmented Generation (RAG) system, comprising: (a) generating a plurality of target data by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model using the target data; generating a viewpoint for each of the plurality of target data; In the step (b), the Q&A data is generated by the first large-scale language model using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; outputting a first determination result regarding the accuracy of the Q&A data for the target document using a second large-scale language model; further comprising a step of classifying the plurality of target data into components; A method for generating Q&A data, wherein the first judgment result includes a judgment result for each of the components.

2. A method for generating Q&A data for evaluating the accuracy of answers in a Retrieval Augmented Generation (RAG) system, comprising: (a) generating a plurality of target data by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model using the target data; generating a viewpoint for each of the plurality of target data; In the step (b), the Q&A data is generated by the first large-scale language model using the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data; outputting a first determination result regarding the accuracy of the Q&A data for the target document using a second large-scale language model; a third large-scale language model for generating new Q&A data when the first determination result does not satisfy a predetermined standard;

3. After all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criteria, a step of inputting the first question from a pair of a first question and a first answer to the first question included in the Q&A data into the RAG system and returning a second answer to the first question from the RAG system; The method for generating Q&A data according to claim 2 , further comprising: outputting a second determination result regarding the accuracy of the second answer to the first answer using a fourth large-scale language model.

4. A method for generating Q&A data for evaluating the accuracy of answers in a Retrieval Augmented Generation (RAG) system, comprising: (a) generating a plurality of target data by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model using the target data; generating a viewpoint in natural language for each of the plurality of target data; A method for generating Q&A data, in which in the step (b), the generated viewpoint and target data corresponding to the viewpoint among the plurality of target data are input, and the Q&A data is generated using the first large-scale language model.

5. A method for generating Q&A data as described in claim 4, further comprising a step of outputting a first judgment result regarding the accuracy of the Q&A data for the target document using a second large-scale language model.

6. A computer-executable information processing program, (a) generating a plurality of target data by dividing a target document into predetermined units; (b) generating Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model using the target data; generating a viewpoint in natural language for each of the plurality of target data; an information processing program that, in the step (b), inputs the generated viewpoint and target data corresponding to the viewpoint from among the plurality of target data, and generates the Q&A data using the first large-scale language model.

7. An information processing system for evaluating the accuracy of answers in a RAG (Retrieval Augmented Generation) system, comprising: a target document division unit that divides the target document into predetermined units to generate a plurality of target data; a Q&A data generation unit that uses the target data to generate Q&A data, which is a pair of questions and answers related to the target data, using a first large-scale language model; a viewpoint generation unit that generates a viewpoint in natural language for each of the plurality of target data, The Q&A data generation unit inputs the generated viewpoint and target data corresponding to the viewpoint from among the plurality of target data, and generates the Q&A data using the first large-scale language model.

Images