Information processing method, information processing program, and information processing system

Abstract

Evaluate the accuracy of the RAG system's responses. [Solution] The information processing method of the present disclosure includes the steps of: (f) after all Q&A data corresponding to multiple target data have met predetermined criteria, inputting the first question from a pair of first questions and first answers to the first question included in the Q&A data into the RAG system and having the RAG system provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

Description

Technical Field

[0001] This disclosure relates to an information processing method, an information processing program, and an information processing system.

Background Art

[0002] Conventionally, Retrieval Augmented Generation (RAG) has been known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] An information processing method according to one form of the present disclosure is an information processing method for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system via a user terminal, wherein the user terminal (a) generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generates Q&A data, which is a pair of questions and answers related to the target data, in a first large-scale language model using the generated viewpoints and target data from the plurality of target data corresponding to the viewpoints; (c) outputs a first judgment result regarding the accuracy of the Q&A data for the target document; and (d) if the first judgment result does not meet a predetermined standard, regenerate The process includes: (c) providing newly generated Q&A data to process (c); (e) executing process (d) from process (b) for all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criteria, inputting the first question from the pair of first question and first answer to the first question included in the Q&A data into the RAG system and having the RAG system provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

[0005] An information processing program according to one form of the present disclosure is a computer-executable information processing program comprising: (a) a step of generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) a step of generating Q&A data, which is a pair of questions and answers related to the target data, in a first large-scale language model using the generated viewpoints and the target data from the plurality of target data corresponding to the viewpoints; (c) a step of outputting a first judgment result regarding the accuracy of the Q&A data for the target document; (d) a step of providing newly generated Q&A data to step (c) if the first judgment result does not meet a predetermined standard; (e) a step of executing steps (b) through (d) for all of the Q&A data corresponding to the plurality of target data; and (f) after all of the Q&A data corresponding to the plurality of target data meet the predetermined standard, the first question is selected from the pairs of first questions and first answers included in the Q&A data, and the first question is selected as RAG (Retrieval Augmented (g) The computer is instructed to perform the following steps: (a) input into the Generation system and have the RAG system provide a second answer to the first question; and (b) output a second judgment result regarding the accuracy of the second answer to the first answer.

[0006] An information processing system according to one embodiment of the present disclosure is an information processing system for evaluating the accuracy of answers from a RAG (Retrieval Augmented Generation) system, comprising: a viewpoint generation unit that generates viewpoints for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of questions and answers related to the target data, using the generated viewpoints and target data from the plurality of target data corresponding to the viewpoints, using a first large-scale language model; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined standard; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined standard, inputs the first question from the pair of a first question and a first answer included in the Q&A data into the RAG system and obtains a second answer to the first question from the RAG system; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer. [Brief explanation of the drawing]

[0007] [Figure 1] This block diagram shows the functional configuration of the information processing system related to this disclosure. [Figure 2] This is a block diagram showing the functional configuration of the server related to this disclosure. [Figure 3] This block diagram shows the functional configuration of the user terminal related to this disclosure. [Figure 4] This flowchart shows an example of the information processing flow related to this disclosure. [Figure 5] This is a schematic diagram of the chatbot related to this disclosure. [Modes for carrying out the invention]

[0008] Hereinafter, one embodiment of this disclosure will be described in detail with reference to the drawings. For ease of understanding, the background and challenges of this disclosure will be described first, followed by a detailed description of the disclosure. <Search extension generation> Retrieval-Augmented Generation (RAG) is a method that searches external knowledge databases in real time and uses that information to generate answers using a generative AI. While typical generative models rely on pre-trained knowledge, RAG excels in tasks where up-to-date information or accuracy is crucial. Because RAG reinforces its answers based on search results, it can provide more accurate and updated content. RAG offers the following advantages compared to generative AI using fine-tuning:

[0009] • Reflection of the latest information: While fine-tuning updates the model using historical data, RAG allows access to new information in real time.

[0010] • Reduced data update burden: Fine-tuning requires periodic model updates, but RAG obtains the latest information through searches each time, eliminating the need for frequent retraining.

[0011] • Flexibility: RAG can generate answers that are not specialized for a particular topic, but can be adapted to a wide range of areas.

[0012] On the other hand, RAG may contain misinformation (hallucination) not included in internal documents, so there is still room for improvement in the accuracy of its responses. <Overview of Information Processing System 1> The information processing system 1 in this disclosure evaluates the response accuracy of a RAG (Retrieval Augmented Generation) system. Furthermore, the information processing system 1 can improve the response accuracy of the RAG system based on recommendations corresponding to the evaluation results. The information processing system 1 may also be used in a chatbot that outputs answers to questions received online from users.

[0013] Information processing system 1 can be used in various fields, such as aircraft, railways, ships, hospitals, banks, and hotels.

[0014] More specifically, the information processing system 1 is an information processing system for evaluating the accuracy of answers from a RAG (Retrieval Augmented Generation) system, and comprises: a viewpoint generation unit that generates viewpoints for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that uses the generated viewpoints and target data from the plurality of target data corresponding to the viewpoints to generate Q&A data, which are pairs of questions and answers related to the target data, using a first large-scale language model; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined standard; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined standard, inputs the first question from the pair of a first question and a first answer included in the Q&A data into the RAG system and obtains a second answer to the first question from the RAG system; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer.

[0015] According to the above configuration, the information processing system 1 can output the response accuracy of the RAG system as a second judgment result. By allowing the user to confirm the response accuracy of the RAG system as a second judgment result, the information processing system 1 can promote the construction of a RAG system with fewer falsehoods (hallucination) and improved response accuracy.

[0016] Further, according to the above configuration, unlike the conventional Q&A reference type chatbot, the information processing system 1 can automatically generate high-quality Q&A simply by the user storing in-house documents and the like in a predetermined folder (such as a data store). In addition, since the information processing system 1 can also perform horizontal accuracy evaluation using a plurality of generation AIs, the possibility of hallucination can be reduced, and the answer accuracy of RAG can be improved.

[0017] Hereinafter, the information processing system 1 will be described while referring to the drawings.

[0018] FIG. 1 is a block diagram showing the functional configuration of the information processing system 1 according to the present disclosure.

[0019] FIG. 2 is a block diagram showing the functional configuration of the server 10 according to the present disclosure.

[0020] FIG. 3 is a block diagram showing the functional configuration of the user terminal 20 according to the present disclosure. <Configuration of Information Processing System 1> Referring to FIG. 1, the information processing system 1 includes a server 10, a user terminal 20, and an administrator terminal 30 connected via a network N.

[0021] The server 10 and the user terminal 20 may be realized by a configuration integrated as one information processing terminal device, but hereinafter, they will be described as separate devices.

[0022] The network N may be a mobile communication system such as 3G, 4G, 5G, 6G, LTE (Long Term Evolution), Wi-Fi (registered trademark), the Internet, or an in-house LAN.

[0023] <Basic Configuration of Server 10> The server 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, and a communication IF 14. These are electrically connected to each other via a communication bus.

[0024] The processor 11 executes instructions written in the program. The processor 11 consists of an arithmetic unit, registers, and / or peripheral circuits.

[0025] The main memory 12 temporarily stores programs and data processed by programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0026] The auxiliary storage device 13 stores data and programs, and is, for example, a flash memory, an HDD (Hard Disk Drive), or a magneto-optical disk.

[0027] Communication IF14 is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards.

[0028] Next, referring to Figure 2, the server 10 includes a storage unit 101 and a control unit 102. The storage unit 101 and the control unit 102 will be described below.

[0029] The memory unit 101 stores the application program 1011, the large-scale language model 1012, the target document 1013, the target data 1014, and the Q&A data 1015.

[0030] Application program 1011 is a program that causes the control unit 102 of server 10 to function as various functional units. Application program 1011 includes applications such as a web browser application.

[0031] The large-scale language model 1012 stores multiple large-scale language models used in the information processing system 1. These multiple large-scale language models may differ from each other in at least some respects. Differentiating multiple large-scale language models means that each has a different architecture or hyperparameters, including differences in layer structure, number of neurons, training methods, and / or optimization algorithms. Furthermore, the datasets used for training and / or tuning methods may also differ, potentially leading to differences in the quality and / or characteristics of the generated text. This allows for different responses from each model to the same task.

[0032] The large-scale language model 1012 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in the storage unit of the user terminal 20 (not shown) as described later, or in other storage devices.

[0033] Next, we will describe the target document 1013. Generally, in search-enhanced generation, reference information is retrieved from a predetermined data source based on instruction information that directs the execution of a task, and a prompt containing the search results for the instruction information and reference information is input to a large-scale language model. The large-scale language model executes the task by referring to reference information that is not included in the training data. As a result, search-enhanced generation improves upon hallucination, which was a problem in the responses of conventional generative AI.

[0034] The target document 1013 can be considered to fall under the aforementioned reference information. The target document 1013 may be the latest information, databases, and internal documents stored on the internet or within an organization (such as a company), and its type is not limited. The target document 1013 may be stored in various formats such as Word, PowerPoint, PDF, SQL format, CSV format, JSON format, XML format, and binary format. For the sake of explanation below, the target document 1013 will be described as a PDF document (target document) stored within a company.

[0035] The target document 1013 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in the storage unit of the user terminal 20 (not shown) as described later, or in other storage devices.

[0036] Next, we will describe the target data 1014. The target data 1014 consists of multiple data sets obtained by dividing the target document into predetermined units. These predetermined units can be determined as appropriate, for example, by page, by two pages, or by chapter within the document.

[0037] Q&A data 1015 refers to a dataset that associates anticipated questions with their model answers for the target data 1014. Q&A data 1015 is created to determine how close the answers obtained by posing questions to the system being evaluated are to the model answers.

[0038] Q&A data 1015 typically includes a "Question" that a user might ask the system, and an "Answer" that is a model answer to that question. Furthermore, Q&A data 1015 may also include a "Category" indicating the theme or topic to which the question belongs, and / or "Tags" that serve as keywords to aid in searching or filtering.

[0039] The Q&A data 1015 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in the storage unit of the user terminal 20 (not shown) as described later, or in other storage devices.

[0040] Next, referring to Figure 2, the control unit 102 includes a target document division unit 1020, a component classification unit 1021, a viewpoint generation unit 1022, a Q&A data generation unit 1023, a first output unit 1024, a Q&A data regeneration unit 1025, an answer acquisition unit 1026, and a second output unit 1027. The control unit 102 may also include a reception unit 1028.

[0041] The target document splitting unit 1020 divides the target document into predetermined units. These predetermined units can be determined as appropriate, for example, by page, by two pages, or by chapter within the document. In the following description, the target document splitting unit 1020 will be described as splitting a target document stored in a predetermined folder into individual pages.

[0042] The target document splitting unit 1020 may split the target document into individual pages using, for example, an online tool (such as Smallpdf or iLovePDF), a function built into Adobe Acrobat, or a program such as Python. Specifically, the target document splitting unit 1020 splits the target document 1013 into individual pages and stores multiple target data 1014 in the storage unit 101. The target document splitting unit 1020 may also store the target data 1014 in a storage device different from the storage unit 101 of the server 10. When the target document 1013 is one page, the target document 1013 may be understood as having been split into one by the target document splitting unit 1020.

[0043] The component classification unit 1021 classifies the components contained in the target data 1014, which has been divided into pages by the target document division unit 1020, into categories such as "text only," "graphs," "tables," and "figures other than graphs." The component classification unit 1021 may be implemented using a large-scale language model such as GPT (Generative Pretrained Transformer), but there are no restrictions on its size or the structure of the model.

[0044] The control unit 102 does not necessarily have to include a component classification unit 1021. However, by including the component classification unit 1021, the control unit 102 can output a first judgment result, including a score for each component, to the first output unit 1024 (described later). This allows the control unit 102 to provide clues as to which classification items should be modified to improve the accuracy of the Q&A data 1015. For example, suppose the first output unit 1024 outputs a first judgment result in the range of 1 to 10, with scores of "8.56" for graphs, "8.30" for tables, "8.83" for figures other than graphs, and "6.75" for plain text only. In this case, since the score for plain text only is the lowest, it provides a clue that the accuracy of the Q&A data 1015 can be improved by modifying only the plain text. Similarly, by including the component classification unit 1021, the control unit 102 can also include judgment results for each component in the second judgment result (described later).

[0045] The component classification unit 1021 may be, for example, a recurrent neural network (RNN) such as a convolutional neural network (CNN), LSTM (Longshort-term memory), GRU (Gated recurrent unit), or another language model that combines one or more models such as a Transformer model. The component classification unit 1021 may be implemented by, for example, ChatGPT, Bard, Gemini, or Claude. The same applies to the viewpoint generation unit 1022, Q&A data generation unit 1023, first output unit 1024, Q&A data regeneration unit 1025, second output unit 1027, recommendation output unit 1029, and optimization unit 1030, which will be described later.

[0046] The perspective generation unit 1022 generates perspectives for each of the multiple target data 1014 obtained by dividing the target document 1013 into predetermined units (1 page). Each perspective includes at least one of the following from the target data: title, summary, figure / table / text type, conclusion, background, and purpose.

[0047] The perspective generation unit 1022 may be implemented using a large-scale language model such as GPT (Generative Pretrained Transformer), but there are no restrictions on its size or the structure of the model.

[0048] The perspective generation unit 1022 inputs, for example, the following prompt to the large-scale language model. • Example prompts "Based on the information gleaned from the following data, what conclusions can be drawn?" In response to this prompt, the viewpoint generation unit 1022 obtains, for example, the following output. "Perspective: Performance by Product" Explanation: Trends in sales target achievement for each product, showing the most stable product and the most volatile product. The prompt may be set by default. Alternatively, the reception unit 1028 may receive at least one of the following: a user change of a predetermined (default) prompt, or a user setting of a new prompt, and the perspective generation unit 1022 may input the received prompt into the large-scale language model.

[0049] The Q&A data generation unit 1023 uses the viewpoints generated by the viewpoint generation unit 1022 and the target data 1014 corresponding to the viewpoints from among the multiple target data 1014 to generate Q&A data 1015, which are pairs of questions and answers related to the target data 1014. The Q&A data generation unit 1023 may be implemented by a large-scale language model. The Q&A data generation unit 1023 stores the generated Q&A data 1015 in the storage unit 101. The Q&A data generation unit 1023 may output the generated Q&A data 1015 to the first output unit 1024.

[0050] The first output unit 1024 outputs a first judgment result regarding the accuracy of the Q&A data 1015 with respect to the target data 1014. The first output unit 1024 may be implemented by a large-scale language model. The first judgment result includes at least one of a judgment value and a reason for the judgment regarding accuracy.

[0051] When the same large-scale language model is used in both the Q&A data generation unit 1023 and the first output unit 1024, a favorable first judgment result is more likely to be output. Therefore, the first output unit 1024 may use a second large-scale language model different from that of the Q&A data generation unit 1023 (first large-scale language model), although this is not limited to the first output unit 1024.

[0052] The first output unit 1024 accepts, for example, the following prompt: • Example prompts Please refer to the following data to determine the accuracy of your answers to the questions. # Question: ${QUESTION} # Answer: ${ANSWER} Please rate the result on a scale of 1 to 10. If there are any discrepancies in the information used to support your answer, please rate it at 5 points or lower. If the information is accurate and only a conclusion is lacking, please rate it at 7 points or higher. Please indicate your rating with both the "score" and the "reason." The first output unit 1024 then outputs, for example, the following first determination result. ·Example of first judgment result "Score: 3" "Reason: The first part of the answer is correct, but the second part contradicts the first part. Therefore, I gave it 3 points." As mentioned above, the first output unit 1024 may output a first judgment result that includes a score for each component. For example, the first output unit 1024 may output a first judgment result that includes scores of "8.56" for the graph, "8.30" for the table, "8.83" for figures other than graphs, and "6.75" for plain text only. In this way, the first output unit 1024 may output a first judgment result that includes decimal numbers.

[0053] The prompt may be set by default. Alternatively, the reception unit 1028 may receive at least one of the following: a user change of a predetermined (default) prompt, or a user setting of a new prompt, and the first output unit 1024 may input the received prompt to the large-scale language model.

[0054] The first output unit 1024 may output the first determination result to the Q&A data regeneration unit 1025, or it may store the first determination result in the storage unit 101 or an external storage device.

[0055] The Q&A data regeneration unit 1025 generates new Q&A data 1015 if the first determination result output by the first output unit 1024 does not meet a predetermined criterion. The Q&A data regeneration unit 1025 may be implemented by a large-scale language model. The Q&A data regeneration unit 1025 generates Q&A data 1015, which is a pair of questions and answers related to the target data 1014, using the viewpoint generated by the viewpoint generation unit 1022 and the target data 1014 corresponding to the viewpoint from among the multiple target data 1014. The Q&A data regeneration unit 1025 (third large-scale language model) may be the same as the first large-scale language model and / or the second large-scale language model described above, or it may be a different large-scale language model.

[0056] In the aforementioned prompts where scores are assigned in the range of 1 to 10, the "predetermined standard" is, for example, set to "8," and any score of 8 or higher is treated as meeting the standard. The "predetermined standard" can be determined arbitrarily. The "predetermined standard" shown herein is merely an example, and this disclosure is not limited thereto. For example, the "predetermined standard" may be set to "4" when scores are assigned in the range of 1 to 5, or it may be set as appropriate. Alternatively, the "predetermined standard" may be set as the average score of all Q&A data 1015 being X or higher. Alternatively, the "predetermined standard" may be set as a combination such as X or higher for graphs, Y or higher for tables, and Z or higher for text.

[0057] Thus, in this disclosure, "predetermined criteria" may be set in various ways. Furthermore, in this disclosure, for example, if the average score for all Q&A data 1015 is X points or higher, or if the criteria such as X points or higher in the case of a graph, or Y points or higher in the case of a table are met, then it can be considered that "all Q&A data 1015 corresponding to multiple target data 1014 have met the prescribed criteria."

[0058] The Q&A data regeneration unit 1025 outputs the newly regenerated Q&A data 1015 to the first output unit 1024. The Q&A data regeneration unit 1025 may also store the newly regenerated Q&A data 1015 in the storage unit 101. The first output unit 1024 outputs a first judgment result regarding the accuracy of the new Q&A data 1015 for the target document 1013 using the method described above.

[0059] The Q&A data regeneration unit 1025 may cause the user, who has referred to the first determination result, to generate new Q&A data 1015. For example, the Q&A data regeneration unit 1025 outputs a screen or audio prompting the user to generate new Q&A data 1015, and receives the new Q&A data 1015 from the user. The Q&A data regeneration unit 1025 then outputs the new Q&A data 1015 received from the user to the first output unit 1024. The Q&A data regeneration unit 1025 can also be realized with this configuration.

[0060] The above operations may be performed for all Q&A data 1015 corresponding to multiple target data 1014.

[0061] The answer acquisition unit 1026 operates as follows after all Q&A data corresponding to multiple target data meet predetermined criteria. Specifically, the answer acquisition unit 1026 inputs the first question from the pair of the first question and the first answer to the first question included in the Q&A data into the RAG system and acquires the second answer to the first question from the RAG system. The RAG system is the RAG system that is operating in the production environment, test environment, or development environment.

[0062] The second output unit 1027 outputs a second judgment result regarding the accuracy of the second response to the first response. The second output unit 1027 may be implemented by a large-scale language model (fourth large-scale language model). The second judgment result includes at least one of a judgment value and a reason for the judgment regarding accuracy.

[0063] The second output unit 1027 accepts, for example, the following prompt: • Example prompts "You are the quality assessor for the Q&A system. Evaluate the system's response (second answer) to anticipated questions by comparing it closely with the model answer (first answer). Pay close attention to any differences between the model answer and the system's response regarding numerical values ​​and units." # Anticipated Question: ${QUESTION} # Model answer: ${{REF_ANSWER} # System Answer: ${SYSTEM_ANSWER} Please rate your answer on a scale of 1 to 10, with the model answer being scored at 10 points. If there are any discrepancies between the system response and the information used to support the model answer, please rate it at 5 points or lower. If the information is accurate but the conclusion is simply lacking, please rate it at 7 points or higher. The second output unit 1027 then outputs, for example, the following second determination result. ·Second judgment result example 1 "Score: 5" "Reason: I gave it 5 points because it included one incorrect answer in addition to two correct answers." ·Second judgment result example 2 "Score: 3" "Reason: The first part of the answer is correct, but the second part contradicts the first part. Therefore, I gave it 3 points." The second output unit 1027 may output a second judgment result that includes a score for each component. For example, the second output unit 1027 may output a second judgment result that includes scores such as "8.68" for graphs, "7.75" for tables, "8.58" for figures other than graphs, and "7.56" for plain text only. In this way, the second output unit 1027 may output a second judgment result that includes decimal numbers.

[0064] The second output unit 1027 may store the second determination result in the storage unit 101 or an external storage device.

[0065] The prompt may be set by default. Alternatively, the reception unit 1028 may receive at least one of the following: a user change of a predetermined (default) prompt, or a user setting of a new prompt, and the second output unit 1027 may input the received prompt to the large-scale language model.

[0066] With the above configuration, the information processing system 1 can output the response accuracy of the RAG system as a second judgment result. By allowing the user to confirm the response accuracy of the RAG system as a second judgment result, the information processing system 1 can promote the construction of a RAG system with less misinformation (hallucination) and improved response accuracy.

[0067] The fourth large-scale language model may be the same as the first, second, or third large-scale language model described above, or other large-scale language models may be used. Furthermore, the fourth large-scale language model may be composed of multiple large-scale language models. <Recommendation function> Referring again to Figure 2, the control unit 102 may further include a recommendation output unit 1029 and an optimization unit 1030.

[0068] The recommendation output unit 1029 outputs recommendations according to the second judgment result. The recommendation output unit 1029 may be implemented using a large-scale language model such as GPT (Generative Pretrained Transformer), but there are no restrictions on its size or the structure of the model.

[0069] As an example, the recommendation output unit 1029 accepts the following prompt based on the second determination result output by the second output unit 1027. • Example prompts "Please explain the categories that received high and low ratings based on the following test results, and provide recommendations for the areas that received low ratings." # Test results Model used: gemini-pro-1.0 Layout Parser: OFF Plain text: 7.5 Image data: 7.2 Table data: 3.4 The recommendation output unit 1029 then outputs, for example, the following recommendations. Examples of review comments "Evaluation Results: While the test results showed that responses to plain text and image data were appropriate and received high marks, the evaluation of responses to tabular data was unsatisfactory." Recommendation: 1) Recommendation for settings: The model you are currently using is an older model, so we recommend using the latest stable version. 2) We recommend turning on the Layout Parser function. 3) Recommendations for document content: It is recommended to avoid using symbols in table data. It is also recommended to keep table data as simple as possible. In this way, the recommendation output unit 1029 analyzes the trend of results with low scores in the second judgment result and can output recommendations such as, "The model best suited to the current document is Gemini 1.5 pro." This is achieved by instructing the recommendation output unit 1029, which is implemented by the large-scale language model, to output recommendations based on the analysis results. These are just examples, and the recommendation output unit 1029 can also output recommendations for settings of the RAG system itself, such as the type of large-scale language model or the parameters used. Furthermore, the recommendation output unit 1029 can also output recommendations for the content of the target document (or target data) that is referenced by the RAG system. In this way, the recommendation output unit 1029 can output recommendations across a wide range of content (scope).

[0070] If the second judgment result does not meet predetermined criteria, the optimization unit 1030 performs optimization based on the recommendation output unit 1029. For example, as an example of optimization, the optimization unit 1030 may perform processes such as changing the large-scale language model used by the RAG system to Gemini 1.5 pro (selection of the optimal LLM), or changing the parameters of the large-scale language model used by the RAG system (tuning), in order to improve the second judgment result. Alternatively, as another example, the optimization unit 1030 may perform processes such as changing the writing style of the target document referenced by the RAG system, for example, by not using symbols in the table data used, in order to improve the second judgment result.

[0071] If the second judgment result does not meet the predetermined criteria, for example, the predetermined criterion is defined as a score of "7 or higher" in the second judgment result. In this case, "Table data: 3.4" does not meet the predetermined criterion, while "Plain text: 7.5" and "Image data: 7.2" do. The optimization unit 1030 determines that "Table data: 3.4" does not meet the predetermined criterion and executes the optimization process described above. When all scores related to the second judgment result meet the predetermined criterion, the optimization unit 1030 may terminate the process.

[0072] The optimization unit 1030 performs optimization based on the recommendation, and then prompts the response acquisition unit 1026 to provide (acquire) a second response. The subsequent processing is as described above.

[0073] By having the above configuration, information processing system 1 can improve the response accuracy of the RAG system. <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service (e.g., a chatbot). The user terminal 20 receives input from a user using a chatbot, for example, and operates the control unit 102 of the server 10. The user terminal 20 may be, for example, a stationary PC (Personal Computer), a laptop PC, a mobile device such as a smartphone or tablet, an HMD (Head-Mount Display), or a wearable device such as a smartwatch.

[0074] The user terminal 20 includes a storage unit 201, a control unit 202, an input unit 203, and an output unit 204. <Storage unit 201 of user terminal 20> The storage unit 201 of the user terminal 20 stores the application program 2011. The storage unit 201 may also store the user ID 2012.

[0075] The application program 2011 may be pre-stored in the memory unit 201. The application program 2011 may also be downloaded from a web server operated by a service provider via a communication interface. The application program 2011 may include applications such as a web browser application.

[0076] User ID 2012 is the user's account ID. The user sends User ID 2012 from the user terminal 20 to the server 10. The server 10 identifies the user based on User ID 2012 and provides services to the user. If the service (e.g., a chatbot) does not require a User ID, the storage unit 201 does not need to store User ID 2012. <Control unit 202 of user terminal 20> The control unit 202 of the user terminal 20 comprises an input control unit 2021 and an output control unit 2022. The control unit 202 realizes each functional unit by executing the application program 2011 stored in the storage unit 201. <Input section 203 of user terminal 20> The input section 203 of the user terminal 20 may include a camera 2031, a microphone 2032, a keyboard 2033, and a touch panel 2064, etc. <Output section 204 of user terminal 20> The output unit 204 of the user terminal 20 may include a display 2041 and a speaker 2042. <Configuration of Administrator Terminal 30> Referring to Figure 1, the administrator terminal 30 is an information processing device operated by the administrator of a service (e.g., a chatbot). The administrator is also the operator of the information processing system 1. The administrator uses the administrator terminal 30 to build large-scale language models used in the information processing system 1, or to determine and modify prompts to be input into the large-scale language model.

[0077] The administrator terminal 30 may be, for example, a stationary PC (Personal Computer), a laptop PC, a mobile device such as a smartphone or tablet, an HMD (Head-Mount Display), or a wearable device such as a smartwatch.

[0078] <Example of processing by an information processing device> Next, the processing flow of Information Processing System 1 will be explained with reference to Figure 4. Figure 4 is a flowchart showing an example of the information processing flow related to this disclosure.

[0079] In S2, the target document division unit 1020 divides the target document into predetermined units.

[0080] In S3, the component classification unit 1021 classifies the components contained in the target data 1014, which has been divided into pages by the target document division unit 1020, into categories such as "text only," "graphs," "tables," and "figures other than graphs."

[0081] In S4, the viewpoint generation unit 1022 generates viewpoints for each of the multiple target data obtained by dividing the target document into predetermined units (for example, each page).

[0082] In S6, the Q&A data generation unit 1023 uses the perspective generated by the perspective generation unit 1022 and the target data from among the multiple target data that corresponds to the perspective to generate Q&A data, which consists of pairs of questions and answers related to the target data.

[0083] In S8, the first output unit 1024 outputs a first judgment result regarding the accuracy of the Q&A data for the target document.

[0084] If the first judgment result does not meet the predetermined criteria (NO in S10), the process proceeds to S12. In S12, the Q&A data regeneration unit 1025 regenerates new Q&A data. Then, in S8, the first output unit 1024 outputs a first judgment result regarding the accuracy of the new Q&A data for the target document. The first judgment result may include a score for each component.

[0085] In S10, if the first judgment result meets the predetermined criteria (YES in S10), the process proceeds to S14. After all Q&A data corresponding to multiple target data meet the predetermined criteria, in S14, the answer acquisition unit 1026 inputs the first question from the pair of first question and first answer included in the Q&A data into the RAG system and obtains the second answer to the first question from the RAG system.

[0086] In S16, the second output unit 1027 outputs a second judgment result regarding the accuracy of the second response to the first response. The second judgment result may include a score for each component.

[0087] In S18, the recommendation output unit 1029 outputs a recommendation according to the second judgment result.

[0088] In S20, if the second judgment result meets the predetermined criteria (YES in S20), the process ends. In S20, if the second judgment result does not meet the predetermined criteria (NO in S20), in S22, the optimization unit 1030 performs optimization based on the recommendation output by the recommendation output unit 1029. After performing the optimization process based on the recommendation, the optimization unit 1030 again prompts the second output unit 1027 to provide (obtain) a second response (proceeds to S8).

[0089] <Chatbot> Next, the chatbot 50 related to this disclosure will be described with reference to Figure 5. Figure 5 is a schematic diagram of the chatbot 50 related to this disclosure.

[0090] Chatbot 50 is an example of a RAG system whose response accuracy has been improved by the information processing system 1 related to this disclosure. Chatbot 50 comprises a reception unit 51 that receives questions from users and a response presentation unit 52 that presents answers to those questions. As a result, Chatbot 50 can provide users with a RAG system with improved response accuracy.

[0091] <Examples of implementation using software> The functions of the information processing system 1 (hereinafter referred to as the "device") can be realized by a program that causes the device to function as a computer, and by a program that causes each control block of the device (particularly each part included in the control unit 102) to function as a computer.

[0092] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0093] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0094] Furthermore, some or all of the functions of each of the above control blocks can also be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

[0095] Furthermore, each process described in the above embodiments may be performed by AI (Artificial Intelligence). In this case, the AI ​​may operate on the control device described above, or it may operate on other devices (for example, an edge computer or a cloud server). <Summary> This disclosure includes at least the following aspects:

[0096] An information processing method according to aspect 1 of this disclosure is an information processing method for evaluating the answer accuracy of a RAG (Retrieval Augmented Generation) system via a user terminal, wherein the user terminal (a) generates a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) generates Q&A data, which is a pair of questions and answers related to the target data, in a first large-scale language model using the generated viewpoints and target data from the plurality of target data corresponding to the viewpoints; (c) outputs a first judgment result regarding the accuracy of the Q&A data for the target document; and (d) if the first judgment result does not meet a predetermined standard, regenerate The process includes: (c) providing newly generated Q&A data to process (c); (e) executing process (d) from process (b) for all of the Q&A data corresponding to the plurality of target data; (f) after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criteria, inputting the first question from the pair of first question and first answer to the first question included in the Q&A data into the RAG system and having the RAG system provide a second answer to the first question; and (g) outputting a second judgment result regarding the accuracy of the second answer to the first answer.

[0097] According to the above configuration, the information processing method relating to aspect 1 of the present disclosure regenerates the Q&A data until the first judgment result regarding the accuracy of the Q&A data for the target document meets a predetermined standard. The information processing method relating to aspect 1 of the present disclosure then executes steps (b) to (d) for all of the Q&A data corresponding to multiple target data. The information processing method relating to aspect 1 of the present disclosure then positions steps (a) to (e) as preliminary preparation, and following this preliminary preparation, outputs a second judgment result regarding the accuracy of the second answer to the first answer through steps (f) and (g).

[0098] With the above configuration, the information processing method according to Embodiment 1 of this disclosure can output the response accuracy of the RAG system as a second judgment result. By allowing the user to confirm the response accuracy of the RAG system as a second judgment result, the information processing method according to Embodiment 1 of this disclosure can promote the construction of a RAG system with fewer false reports (hallucination) and improved response accuracy.

[0099] The information processing method according to aspect 2 of this disclosure is characterized in that, in aspect 1, step (c) is performed by a second large-scale language model different from the first large-scale language model.

[0100] According to the above configuration, the information processing method according to aspect 2 of this disclosure can increase the possibility of outputting a first determination result with higher accuracy by using different large-scale language models in process (b) and process (c).

[0101] The information processing method according to aspect 3 of this disclosure, in aspect 1 or 2, in step (d), causes the third large-scale language model to regenerate the new Q&A data by step (b).

[0102] According to the above configuration, new Q&A data is also generated by a large-scale language model, so the information processing method according to aspect 3 of this disclosure can improve the accuracy of responses.

[0103] The third large-scale language model may be the same as or different from the first large-scale language model and / or the second large-scale language model.

[0104] The information processing method according to aspect 4 of this disclosure, in aspect 1 or 2, causes the user who has referred to the first determination result to regenerate the new Q&A data in step (d).

[0105] According to the above configuration, the information processing method relating to aspect 4 of this disclosure can regenerate Q&A data that conforms to the user's intentions and can increase user satisfaction.

[0106] The information processing method according to aspect 5 of the present disclosure is such that, in any of aspects 1 to 4, the first determination result and the second determination result include at least one of the determination value and the determination reason for accuracy.

[0107] According to the above configuration, the first judgment result and the second judgment result include a quantitative criterion in the form of a judgment value, and / or a qualitative reason in the form of a judgment reason. As a result, the information processing method according to aspect 5 of this disclosure can be expected to further improve the accuracy of the response.

[0108] The information processing method relating to aspect 6 of this disclosure is such that, in any of aspects 1 to 5, the viewpoint includes at least one of the following of the target data: title, summary, type of figure / table / text, conclusion, background, and purpose.

[0109] According to the above configuration, by setting a large number of perspective items, it is possible to generate Q&A data with higher accuracy.

[0110] In the information processing method according to aspect 7 of the present disclosure, in any of aspects 1 to 6, step (g) is performed by a fourth large-scale language model.

[0111] According to the above configuration, the accuracy of the second judgment result can be further improved.

[0112] The information processing method according to aspect 8 of the present disclosure is characterized in that, in any of aspects 1 to 7, in step (a), each of the plurality of target data is classified according to its constituent elements, and at least one of the first determination result and the second determination result includes the determination result for each constituent element.

[0113] According to the above configuration, it is possible to obtain judgment results for each component, making it easier to find clues as to which components should be modified to improve the accuracy of the RAG system's responses.

[0114] The information processing method according to aspect 9 of this disclosure, in aspect 8, wherein the components include at least one of text, graphs, tables, and figures other than graphs.

[0115] The aforementioned components include any category arbitrarily set by the user in the prompt. For example, the components may be text, graphs, tables, and figures other than graphs. This configuration makes it easier to improve the accuracy of the RAG system's responses.

[0116] The information processing method according to aspect 10 of the present disclosure further includes, in any of aspects 1 to 9, a step of (h) outputting a recommendation corresponding to the second determination result.

[0117] According to the above configuration, after verifying the accuracy of the RAG system using Q&A data, recommendations are output according to the second judgment result, allowing for an understanding of how to further improve the answer accuracy of the RAG system using the second judgment result as a basis.

[0118] The information processing method according to aspect 11 of the present disclosure further includes, in any of aspects 1 to 10, a step of performing optimization of the RAG system based on the recommendation if the second determination result does not meet a predetermined standard.

[0119] According to the above configuration, after verifying the accuracy of the RAG system using Q&A data, recommendations are output according to the second judgment result. Then, by performing optimization of the RAG system based on the recommendations, the accuracy of the RAG system's responses can be further improved using the second judgment result as material.

[0120] The information processing method according to aspect 12 of this disclosure relates to the optimization in aspect 11, wherein the optimization relates to the setting change of the RAG system.

[0121] According to the above configuration, the accuracy of the RAG system's responses can be further improved by changing the settings of the RAG system, such as changing the type of large-scale language model used in the RAG system or tuning the parameters.

[0122] The information processing method relating to aspect 13 of this disclosure is characterized in that, in aspect 11, the optimization relates to the modification of the target document.

[0123] According to the above configuration, the accuracy of the RAG system's responses can be further improved by modifying the target text, such as by reducing the number of symbols used in the target text.

[0124] An information processing program according to aspect 14 of this disclosure is a computer-executable information processing program comprising: (a) a step of generating a viewpoint for each of a plurality of target data obtained by dividing a target document into predetermined units; (b) a step of generating Q&A data, which is a pair of questions and answers related to the target data, in a first large-scale language model using the generated viewpoints and the target data from the plurality of target data corresponding to the viewpoints; (c) a step of outputting a first judgment result regarding the accuracy of the Q&A data for the target document; (d) a step of providing newly generated Q&A data to step (c) if the first judgment result does not meet a predetermined standard; (e) a step of executing steps (b) through (d) for all of the Q&A data corresponding to the plurality of target data; and (f) after all of the Q&A data corresponding to the plurality of target data meet the predetermined standard, the first question is selected from the pairs of first questions and first answers included in the Q&A data, and the first question is selected as RAG (Retrieval Augmented (g) The computer is instructed to perform the following steps: (a) input into the Generation system and have the RAG system provide a second answer to the first question; and (b) output a second judgment result regarding the accuracy of the second answer to the first answer.

[0125] According to the above configuration, the information processing program according to aspect 14 of this disclosure can achieve the same effects as the information processing method according to aspect 1 of this disclosure.

[0126] An information processing system according to aspect 15 of the present disclosure is an information processing system for evaluating the accuracy of answers from a RAG (Retrieval Augmented Generation) system, comprising: a viewpoint generation unit that generates viewpoints for each of a plurality of target data obtained by dividing a target document into predetermined units; a Q&A data generation unit that generates Q&A data, which is a pair of questions and answers related to the target data, using the generated viewpoints and target data from the plurality of target data corresponding to the viewpoints, using a first large-scale language model; a first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document; a Q&A data regeneration unit that regenerates new Q&A data until the first judgment result satisfies a predetermined standard; an answer acquisition unit that, after all of the Q&A data corresponding to the plurality of target data satisfy the predetermined standard, inputs the first question from the pair of a first question and a first answer included in the Q&A data into the RAG system and obtains a second answer to the first question from the RAG system; and a second output unit that outputs a second judgment result regarding the accuracy of the second answer to the first answer.

[0127] According to the above configuration, the information processing system according to aspect 15 of the present disclosure can achieve the same effects as the information processing method according to aspect 1 of the present disclosure.

[0128] The information processing system according to aspect 16 of this disclosure includes, in aspect 15 above, a receiving unit that accepts at least one of the following: a user changing a predetermined prompt to be input into a large-scale language model used in the information processing system, and the user setting a new prompt.

[0129] According to the above configuration, the information processing system according to aspect 15 of this disclosure can easily output a second judgment result that reflects the user's intentions.

[0130] (Additional notes) This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure. [Explanation of Symbols]

[0131] 1. Information Processing System 10 servers 20 User Terminals 30 Administrator terminals 50 Chatbots 51 Reception Department 52 Answer presentation part 1011, 2011 Application Program 1012 Large-scale language models 1013 Target Documents 1014 Target Data 1015 Q&A Data 1020 Target Document Division Section 1021 Component classification part 1022 Perspective Generation Unit 1023 Q&A Data Generation Department 1024 First Output Section 1025 Q&A Data Regeneration Unit 1026 Answer acquisition part 1027 Second Output Section 1028 Reception Department 1029 Recommendation Output Section 1030 Optimization Unit

Claims

1. An information processing method for evaluating the response accuracy of a RAG (Retrieval Augmented Generation) system via a user terminal, The aforementioned user terminal (a) A step of generating a viewpoint for each of the multiple target data obtained by dividing the target document into predetermined units, (b) A step of causing a first large-scale language model to generate Q&A data, which is a pair of questions and answers related to the target data, using the generated perspective and the target data from the plurality of target data that corresponds to the perspective, (c) A step of outputting a first judgment result regarding the accuracy of the Q&A data for the target document, (d) If the first determination result does not meet the predetermined criteria, the process of providing the newly generated Q&A data to process (c), (e) A step of performing steps (b) through (d) for all of the Q&A data corresponding to the plurality of target data, (f) After all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criteria, input the first question from the pair of first question and first answer included in the Q&A data into the RAG system and have the RAG system provide a second answer to the first question, (g) An information processing method comprising the step of outputting a second judgment result regarding the accuracy of the second response to the first response.

2. The information processing method according to claim 1, wherein step (c) is performed by a second large-scale language model different from the first large-scale language model.

3. The information processing method according to claim 1 or 2, wherein in step (d), the new Q&A data is regenerated in the third large-scale language model by step (b).

4. The information processing method according to claim 1 or 2, wherein in step (d), the user who referred to the first determination result is instructed to regenerate the new Q&A data.

5. The information processing method according to claim 1 or 2, wherein the first determination result and the second determination result include at least one of the determination value and the determination reason relating to accuracy.

6. The information processing method according to claim 1 or 2, wherein the aforementioned viewpoint includes at least one of the title, summary, type of figure / table / text, conclusion, background, and purpose of the target data.

7. The information processing method according to claim 1 or 2, wherein step (g) is performed by a fourth large-scale language model.

8. In step (a) above, each of the multiple target data is classified according to its constituent elements, The information processing method according to claim 1 or 2, wherein at least one of the first determination result and the second determination result includes a determination result for each of the components.

9. The information processing method according to claim 8, wherein the constituent elements include at least one of text, graphs, tables, and figures other than graphs.

10. (h) The information processing method according to claim 1 or 2, further comprising the step of outputting a recommendation corresponding to the second determination result.

11. (i) The information processing method according to claim 10, further comprising the step of performing optimization of the RAG system based on the recommendation if the second determination result does not meet a predetermined standard.

12. The optimization is an information processing method according to claim 11 relating to changing the settings of the RAG system.

13. The optimization is an information processing method according to claim 11 relating to the modification of the target document.

14. A computer-executable information processing program, (a) A step of generating a viewpoint for each of the multiple target data obtained by dividing the target document into predetermined units, (b) A step of generating Q&A data, which is a pair of questions and answers related to the target data, in a first large-scale language model using the generated perspective and the target data from the plurality of target data that corresponds to the perspective, (c) A step of outputting a first judgment result regarding the accuracy of the Q&A data for the target document, (d) If the first determination result does not meet the predetermined criteria, the process of providing the newly generated Q&A data to process (c), (e) A step of performing steps (b) through (d) for all of the Q&A data corresponding to the plurality of target data, (f) After all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criteria, input the first question from the pair of the first question and the first answer to the first question included in the Q&A data into the RAG (Retrieval Augmented Generation) system, and the RAG system provides a second answer to the first question. (g) An information processing program that causes a computer to perform the steps of outputting a second judgment result regarding the accuracy of the second response to the first response.

15. An information processing system for evaluating the response accuracy of a RAG (Retrieval Augmented Generation) system, A viewpoint generation unit that generates viewpoints for each of the multiple target data obtained by dividing the target document into predetermined units, A Q&A data generation unit generates Q&A data, which is a pair of questions and answers related to the target data, using the generated perspective and the target data corresponding to the perspective from among the plurality of target data, using a first large-scale language model. A first output unit that outputs a first judgment result regarding the accuracy of the Q&A data for the target document, A Q&A data regeneration unit regenerates new Q&A data until the first determination result meets a predetermined standard, After all of the Q&A data corresponding to the plurality of target data satisfy the predetermined criteria, the answer acquisition unit inputs the first question from the pair of first question and first answer included in the Q&A data into the RAG system and obtains a second answer to the first question from the RAG system. An information processing system comprising: a second output unit that outputs a second judgment result regarding the accuracy of the second response to the first response.

16. The information processing system according to claim 15, further comprising a reception unit that accepts at least one of the following: a user changing a predetermined prompt to be input to a large-scale language model used in the information processing system, and the user setting a new prompt.

Images