Q&A data generation method, Q&A data generation program, and Q&A data generation system

Abstract

Evaluate the accuracy of answers from a question-answering system. [Solution] The method includes: (a) a step of dividing, by content, one or more contents contained in each of a plurality of first target data obtained by dividing a target document into predetermined units using a first large-scale language model; (b) a step of converting, by a second large-scale language model, each of the divided contents into a conversion format corresponding to the divided contents to generate conversion information; (c) a step of generating second target data based on the conversion information; and (d) a step of generating Q&A data, which is a pair of questions and answers related to the second target data, based on the second target data using a third large-scale language model.

Description

[Technical Field]

[0001] The present disclosure relates to a method for generating Q&A data, a program for generating Q&A data, and a system for generating Q&A data. [Background technology]

[0002] BACKGROUND ART Techniques for controlling the output of answers to questions written in natural language are known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-036945 Summary of the Invention [Means for solving the problem]

[0004] A method for generating Q&A data according to one embodiment of the present disclosure is a method for generating Q&A data for evaluating the answer accuracy of a question-answering system, and includes the steps of: (a) dividing, by content, one or more contents contained in each of a plurality of first target data obtained by dividing a target document into predetermined units using a first large-scale language model; (b) converting, by a second large-scale language model, the divided contents into a conversion format corresponding to each of the divided contents to generate conversion information; (c) generating second target data based on the conversion information; and (d) generating Q&A data, which are pairs of questions and answers related to the second target data, based on the second target data using a third large-scale language model.

[0005] A Q&A data generation program according to one embodiment of the present disclosure is a computer-executable Q&A data generation program, which includes: (a) a step of dividing, by content, one or more contents included in each of a plurality of first target data obtained by dividing a target document into predetermined units using a first large-scale language model; and (b) a step of converting, by a second large-scale language model, each of the divided contents into a conversion format corresponding to the divided contents to generate conversion information. (c) generating second target data based on the conversion information; and (d) generating Q&A data, which is a pair of questions and answers related to the second target data, based on the second target data using a third large-scale language model.

[0006] A Q&A data generation system according to one embodiment of the present disclosure is a Q&A data generation system for evaluating the answer accuracy of a question and answer system, and includes a division unit that divides one or more contents contained in each of a plurality of first target data obtained by dividing a target document into predetermined units into pieces of content, a conversion information generation unit that converts each of the divided contents into a conversion format corresponding to the divided contents to generate conversion information, a second target data generation unit that generates second target data based on the conversion information, and a Q&A data generation unit that generates Q&A data that is a pair of questions and answers related to the second target data based on the second target data. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of an information processing system according to the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of a server according to the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating a functional configuration of a user terminal according to the present disclosure. [Figure 4] 10 is a flowchart illustrating an example of a flow of information processing according to the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of a chatbot according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. For ease of understanding, the outline, background, and problems of the present disclosure will be described first, and then the details of the present disclosure will be described. <Outline of Information Processing System 1> The information processing system 1 (Q&A data generation system 1020) in the present disclosure is a Q&A data generation system for evaluating the answer accuracy of a question and answer system, and includes a division unit that divides one or more contents contained in each of a plurality of first target data obtained by dividing a target document into predetermined units into pieces of content, a conversion information generation unit that converts each of the divided contents into a conversion format corresponding to the divided contents to generate conversion information, a second target data generation unit that generates second target data based on the conversion information, and a Q&A data generation unit that generates Q&A data, which are pairs of questions and answers related to the second target data, based on the second target data.

[0009] According to the above configuration, the information processing system 1 (Q&A data generation system 1020) can generate Q&A data with high accuracy for evaluating the answer accuracy of a question and answer system. In this case, unlike conventional Q&A reference chatbots, the information processing system 1 (Q&A data generation system 1020) can automatically generate high-quality Q&A data simply by a user storing internal company documents, etc., in a predetermined folder (such as a data store).

[0010] Furthermore, as will be described in detail later, the information processing system 1 evaluates the answer accuracy of the question-answering system. At this time, the information processing system 1 can also perform accuracy evaluation side-by-side using multiple generation AIs, thereby reducing the possibility of hallucination and improving the answer accuracy of the question-answering system. Furthermore, the information processing system 1 can improve the answer accuracy of the question-answering system based on recommendations according to the evaluation results.

[0011] The information processing system 1 may also be used in a chatbot that outputs answers to questions received online from users. The information processing system 1 may be used in various fields, such as aircraft, railways, ships, hospitals, banks, and hotels.

[0012] The question answering system can be applied to any question answering system that answers a given question, such as a RAG (Retrieval Augmented Generation) system or an AI agent. <Search Extension Generation> Retrieval-Augmented Generation (RAG) is a method of searching external knowledge databases in real time and having generative AI generate answers based on that information. While regular generative models rely on learned knowledge, RAG excels in tasks where up-to-date information or accuracy is important. RAG augments answers based on search results, providing updated content with higher accuracy. RAG has the following advantages over generative AI that uses fine tuning:

[0013] · Reflection of the latest information: Whereas fine-tuning uses historical data to update the model, RAG has access to new information in real time.

[0014] Reduced data update load: Fine tuning requires periodic model updates, but RAG retrieves the latest information each time it is searched, eliminating the need for frequent retraining.

[0015] · Flexibility: RAG can generate answers that are applicable across a wide range of topics, rather than being topic-specific.

[0016] On the other hand, RAG may contain false information (hallucination) that is not included in internal documents, so there is still room for improvement in the accuracy of the answers. <AIエージェント> An AI agent is an information processing device that processes input information and executes appropriate responses or actions to autonomously achieve a predetermined goal. AI agents receive input from the environment or user, analyze the content, and then respond or act in accordance with the goal. Input analysis can use natural language processing, rule-based reasoning, or machine learning models.

[0017] AI agents may be capable of perception, reasoning, and action, and when used as chatbots in particular, they are responsible for accepting user questions and searching for or generating appropriate information to answer. In this form, the AI ​​agent responds in natural language while maintaining the context of the dialogue. If necessary, they can also perform database queries, external API calls, or computational processing to generate responses.

[0018] In this way, AI agents can perform intelligent processing according to their purpose while interacting with humans informationally through a dialogue interface, and therefore can be usefully configured as response systems such as chatbots.

[0019] On the other hand, like RAGs, AI agents may contain false information (hallucination), so there is still room for improvement in the accuracy of their answers.

[0020] An information processing system 1 (Q&A data generation system 1020) according to the present disclosure is applied to a question and answering system. The question and answering system may be, but is not limited to, a RAG or an AI agent. The information processing system 1 will be described below with reference to the drawings. As an example, the information processing system 1 will be described below as being applied to a RAG system.

[0021] FIG. 1 is a block diagram showing the functional configuration of an information processing system 1 according to the present disclosure.

[0022] FIG. 2 is a block diagram showing the functional configuration of the server 10 according to the present disclosure.

[0023] FIG. 3 is a block diagram showing the functional configuration of the user terminal 20 according to the present disclosure. <Configuration of Information Processing System 1> Referring to FIG. 1, an information processing system 1 includes a server 10, a user terminal 20, and an administrator terminal 30, which are connected via a network N.

[0024] The server 10 and the user terminal 20 may be realized as an integrated configuration as a single information processing terminal device, but in the following description they will be described as separate devices.

[0025] The network N may be a mobile communication system such as 3G, 4G, 5G, or 6G, LTE (Long Term Evolution), Wi-Fi (registered trademark), the Internet, or an in-house LAN.

[0026] <Basic configuration of Server 10> The server 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, and a communication IF 14. These are electrically connected to each other via a communication bus.

[0027] The processor 11 executes instructions written in a program and is configured with an arithmetic unit, a register, and / or peripheral circuits.

[0028] The main storage device 12 temporarily stores programs and data to be processed by the programs, and is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0029] The auxiliary storage device 13 stores data and programs, and is, for example, a flash memory, a hard disk drive (HDD), or a magneto-optical disk.

[0030] The communication IF 14 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

[0031] 2, the server 10 includes a storage unit 101 and a control unit 102. The storage unit 101 and the control unit 102 will be described below.

[0032] The storage unit 101 stores an application program 1011 , a large-scale language model 1012 , a target document 1013 , target data 1014 , conversion information 1015 , and Q&A data 1016 .

[0033] The application program 1011 is a program for causing the control unit 102 of the server 10 to function as each functional unit. The application program 1011 includes an application such as a web browser application.

[0034] The large-scale language model 1012 stores multiple large-scale language models used in the information processing system 1. The multiple large-scale language models may be at least partially different from each other. The multiple large-scale language models being different from each other means that they have different architectures or hyperparameters, etc., including differences in layer structure, number of neurons, training method, and / or optimization algorithm. Furthermore, the datasets and / or tuning methods used for learning may also be different from each other, which may result in differences in the quality and / or characteristics of the generated text. This allows different responses to be expected for each model, even for the same task.

[0035] The large-scale language model 1012 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of a user terminal 20 (described later) or in another storage device.

[0036] Next, the target document 1013 will be described. Generally, in search expansion generation, reference information is searched for from a predetermined data source based on instruction information that instructs the execution of a task, and a prompt including the search results for the instruction information and the reference information is input to a large-scale language model. The large-scale language model executes the task by referring to reference information that is not included in the training data. In this way, search expansion generation improves the problem of misinformation (hallucination) that has been a problem in the answers of conventional generative AI.

[0037] The target document 1013 may be considered to correspond to the reference information described above. The target document 1013 may be internet information or the latest information, databases, internal documents, etc. stored within an organization (such as a company), and the type is not limited. The target document 1013 may be stored in various formats, such as Word, PowerPoint, PDF, SQL format, CSV format, JSON format, XML format, and binary format. For ease of explanation, the following description will be given assuming that the target document 1013 is a PDF-format document (target document) stored within a company.

[0038] The target document 1013 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of the user terminal 20 described below, or in another storage device.

[0039] Next, the target data 1014 will be described. The target data 1014 includes first target data and second target data. The first target data is a plurality of data pieces obtained by dividing the target document into predetermined units. The predetermined units may be determined as appropriate, such as one page, two pages, or a chapter within a document. The second target data, the details of which will be described later, is data generated based on the conversion information.

[0040] The Q&A data 1016 is a data set that associates questions and answers related to the second target data based on the second target data. The Q&A data 1016 is generated to determine how close the answers obtained by giving questions to the system to be evaluated are to model answers.

[0041] The Q&A data 1016 typically includes "Questions" that a user may ask the system and "Answers" that are model answers to those questions. Furthermore, the Q&A data 1016 may include "Categories" that indicate the themes or topics to which the questions belong, and / or "Tags" that are keywords that aid in searching or filtering.

[0042] The Q&A data 1016 does not necessarily have to be stored in the storage unit 101 of the server 10, but may be stored in a storage unit (not shown) of the user terminal 20 described below, or in another storage device.

[0043] 2 , the control unit 102 includes a Q&A data generation system 1020, a viewpoint generation unit 1021, a first output unit 1022, a Q&A data regeneration unit 1023, an answer acquisition unit 1024, and a second output unit 1025. The control unit 102 may also include a reception unit 1026, a recommendation output unit 1027, and an optimization unit 1028.

[0044] The Q&A data generation system 1020 includes a target document division unit 1020A, a content division unit 1020B, a conversion information generation unit 1020C, a second target data generation unit 1020D, and a Q&A data generation unit 1020E.

[0045] The target document division unit 1020A divides the target document into predetermined units. The predetermined units may be determined as appropriate, such as one page, two pages, or a chapter within the document. In the following description, the target document division unit 1020A will be described as dividing the target document stored in a predetermined folder into pages.

[0046] The target document divider 1020A may divide the target document into pages using, for example, an online tool (such as Smallpdf or ILovePDF), a function built into Adobe Acrobat, or a program such as Python. Specifically, the target document divider 1020A divides the target document 1013 into pages and stores multiple pieces of target data 1014 in the storage unit 101. The target document divider 1020A may save the target data 1014 in a storage device different from the storage unit 101 of the server 10. When the target document 1013 is one page, it may be understood that the target document 1013 has been divided into one piece by the target document divider 1020A.

[0047] The content division unit 1020B divides one or more contents included in the first target data obtained by dividing the target document into pages by the target document division unit 1020A into each content. Examples of the contents include text, tables, graphs, figures, and images. As an example, suppose the first target data contains text, graphs, and images. In this case, the content division unit 1020B divides the data into text, graphs, and images. In this case, suppose the first target data contains only text. In this case, it can be understood that the content division unit 1020B divides the data into texts.

[0048] Tables may include any table, such as a data table, crosstab, matrix table, comparison table, checklist table, etc. Graphs may include any graph, such as a pie chart, bar chart, line graph, doughnut chart, or bubble chart. Diagrams may include any diagram, such as a Venn diagram, configuration diagram, SWOT analysis diagram, sequence, or sample diagram. Images may include any image, such as a photographic image, illustration, icon, screenshot, UI image, etc. Images may be in any format, such as JPEG, PNG, GIF, BMP, or TIFF format.

[0049] By including the content dividing unit 1020B, the Q&A data generation system 1020 can output the first determination result including the score for each piece of content to the first output unit 1022 (described later). This allows the control unit 102 to provide a clue as to which piece of content should be modified to improve the accuracy of the Q&A data 1016. For example, when the first output unit 1022 outputs the first determination result in the range of 1 to 10, the scores are assumed to be "8.56" for the graph, "8.30" for the table, "8.83" for the figure other than the graph, and "6.75" for the plain text only. In this case, the score for the plain text only is the lowest, providing a clue that the accuracy of the Q&A data 1016 can be improved by modifying only the plain text.

[0050] Similarly, by including the content dividing unit 1020B, the Q&A data generation system 1020 can also include the determination result for each piece of content in the second determination result (described later).

[0051] Similarly, the Q&A data generation system 1020 includes a viewpoint generation unit 1021 (described later), so that the first and second judgment results (described later) can also include judgment results for each viewpoint.

[0052] The content division unit 1020B may be realized by a large-scale language model (first large-scale language model) such as a Generative Pretrained Transformer (GPT), but there are no limitations on the scale or model structure. The content division unit 1020B may be, for example, a recurrent neural network (RNN) such as a convolutional neural network (CNN), a long-short-term memory (LSTM), or a gated recurrent unit (GRU), or another language model that combines one or more models such as a Transformer model. The content division unit 1020B may be realized by, for example, ChatGPT, Bard (registered trademark), Gemini (registered trademark), or Claude (registered trademark). The same applies to other large-scale language generation models, such as the conversion information generation unit 1020C, the Q&A data generation unit 1020E, the viewpoint generation unit 1021, the first output unit 1022, the Q&A data regeneration unit 1023, the second output unit 1025, the recommendation output unit 1027, and the optimization unit 1028, which will be described later.

[0053] The conversion information generation unit 1020C generates conversion information by converting the divided content into a conversion format corresponding to each divided content. The conversion information generation unit 1020C may be realized by a large-scale language model (second large-scale language model) such as a GPT (Generative Pretrained Transformer), but there are no restrictions on the scale or model structure.

[0054] Conversion formats may include, for example, Markdown, Vega-Lite, PlantUML, Mermaid, Tabular Data Package, DOT, JSON, HTML, XML, YAML, TOML, RDF, SGML, BibTeX, LaTeX, GML, GraphML, Asciidoc, R Markdown, UML, BPMN, TikZ, GeoJSON, Ditaa, Chart.js, and Plotly. Conversion formats may include any data description format.

[0055] The conversion information generation unit 1020C may use Markdown, Markdown, Vega-Lite, PlantUML, and Markdown as conversion formats corresponding to the content of text, tables, graphs, figures, and images, in that order. Markdown, Markdown, Vega-Lite, PlantUML, and Markdown may be said to be conversion formats suitable for text, tables, graphs, figures, and images, in that order. The conversion information generation unit 1020C generates conversion information based on a suitable conversion format depending on the content, thereby structuring the divided content into a format that is more readable for a large-scale language model.

[0056] Specifically, the conversion information generation unit 1020C operates as follows. First, a correspondence table between content and conversion formats is prepared. After identifying the content included in the first target data, the conversion information generation unit 1020C selects a conversion format corresponding to the identified content from the correspondence table. The content may be text, a table, a graph, a diagram, an image, or the like. For example, the conversion information generation unit 1020C selects Markdown for a table. Then, the conversion information generation unit 1020C specifies the conversion format in a prompt and generates conversion information using the conversion format corresponding to the content.

[0057] The conversion information generation unit 1020C may determine the conversion format depending on the content. Specifically, instead of preparing a correspondence table between content and conversion formats in advance, the conversion information generation unit 1020C selects the conversion format itself based on the following prompt, for example.

[0058] "Please provide a detailed and accurate output of the content below in a searchable, question-and-answer format. If visual information is included, please present it in the most appropriate format (e.g., Markdown, Mermaid)." In this way, the conversion information generation unit 1020C generates conversion information by converting the divided content into a conversion format corresponding to the content using various methods. The conversion information generation unit 1020C stores the generated conversion information in the storage unit 101. The conversion information generation unit 1020C may output the generated conversion information to the second target data generation unit 1020D.

[0059] The second target data generation unit 1020D generates second target data based on the conversion information generated by the conversion information generation unit 1020C.

[0060] Specifically, the second target data generation unit 1020D generates second target data by adding conversion information to first target data that includes the content among the plurality of first target data. The second target data generation unit 1020D may generate second target data based only on the conversion information, or may use the conversion information itself as the second target data.

[0061] For example, if the first target data includes only one graph, the second target data generation unit 1020D may generate the second target data based only on the conversion information generated from that graph. The second target data generation unit 1020D may also use the conversion information generated from that graph as the second target data.

[0062] Alternatively, if the first target data includes multiple contents such as text, graphs, and images, the second target data generating unit 1020D may generate the second target data by adding conversion information generated from each of the multiple contents to the first target data.

[0063] In this way, the second target data generation unit 1020D generates the second target data based on the conversion information generated by the conversion information generation unit 1020C.

[0064] The second target data generation unit 1020D stores the generated second target data in the storage unit 101. The second target data generation unit 1020D may output the generated second target data to the Q&A data generation unit 1020E and / or the viewpoint generation unit 1021.

[0065] The Q&A data generation unit 1020E generates Q&A data, which is a pair of questions and answers related to the second target data, based on the second target data. The Q&A data generation unit 1020E may be realized by a large-scale language model (third large-scale language model). The Q&A data generation unit 1020E stores the generated Q&A data 1016 in the storage unit 101. The Q&A data generation unit 1020E may output the generated Q&A data 1016 to the first output unit 1022.

[0066] The Q&A data generating unit 1020E may generate the Q&A data 1016 taking into consideration the viewpoints generated by the viewpoint generating unit 1021. The configuration will be described below.

[0067] The viewpoint generating unit 1021 generates a viewpoint for the second target data. The viewpoint includes at least one of the title, summary, type of figure / table / text, conclusion, background, and purpose of the target data.

[0068] The viewpoint generation unit 1021 may be realized by a large-scale language model (fourth large-scale language model) such as a Generative Pretrained Transformer (GPT), but there is no limitation on the scale or model structure.

[0069] The viewpoint generation unit 1021 inputs, for example, the following prompt into the large-scale language model: Example prompts "Based on what you can read from the data below, what can you say about it?" In response to this prompt, the viewpoint generation unit 1021 obtains, for example, the following output: Perspective: Performance by product Explanation: Trends in sales target achievement for each product, the most stable products and the most volatile products. The prompt may be set as a default, or the receiving unit 1026 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the viewpoint generating unit 1021 may input the received prompt into the large-scale language model.

[0070] The viewpoint generation unit 1021 outputs the generated viewpoint to the Q&A data generation unit 1020E. As a result, the Q&A data generation unit 1020E acquires the second target data generated by the second target data generation unit 1020D and the viewpoint generated by the viewpoint generation unit 1021. Then, the Q&A data generation unit 1020E generates Q&A data based on the second target data and the viewpoint.

[0071] Next, the first output unit 1022 will be described. The first output unit 1022 outputs a first determination result regarding the accuracy of the Q&A data for the second target data. The first output unit 1022 may output the first determination result regarding the accuracy of the target document and / or the Q&A data for the first target data. The first output unit 1022 may be realized by a large-scale language model. The first determination result includes at least one of a determination value regarding the accuracy and a determination reason.

[0072] A good first determination result is more likely to be output when the same large-scale language model is used by the Q&A data generation unit 1020E and the first output unit 1022. Therefore, the first output unit 1022 may use, but is not limited to, a large-scale language model (fifth large-scale language model) different from the Q&A data generation unit 1020E (third large-scale language model).

[0073] The first output unit 1022 receives, for example, the following prompt: Example prompts "Please refer to the data below to determine the accuracy of your answers to the questions. # Question: ${QUESTION} # Answer: ${ANSWER} Please rate your assessment on a scale of 1 to 10. If there are any discrepancies in the information on which you based your answer, please rate it as 5 points or lower. If the information is accurate but the conclusion is simply missing, please rate it as 7 points or higher. Please indicate your assessment results with a "score" and "reason." Then, the first output unit 1022 outputs, for example, the following first determination result. ·Example of first judgment result "Score: 3" "Reason: The first half of your answer is correct, but the second half contradicts the first half. Therefore, I've given it 3 points." As described above, the first output unit 1022 may output a first determination result including a score for each piece of content included in the first target data. For example, the first output unit 1022 outputs a first determination result including scores of "8.56" for a graph, "8.30" for a table, "8.83" for a figure other than a graph, and "6.75" for plain text only. In this way, the first output unit 1022 may output a first determination result including digits after the decimal point. Furthermore, the first output unit 1022 may output a first determination result including a score for each aspect included in the second target data.

[0074] The prompt may be set as a default, or the receiving unit 1026 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the first output unit 1022 may input the received prompt to the large-scale language model.

[0075] The first output unit 1022 may output the first determination result to the Q&A data regeneration unit 1023, or may store the first determination result in the storage unit 101 or an external storage device.

[0076] If the first determination result output by the first output unit 1022 does not satisfy a predetermined criterion, the Q&A data regeneration unit 1023 generates new Q&A data 1016. The Q&A data regeneration unit 1023 may be realized by a large-scale language model.

[0077] The Q&A data regeneration unit 1023 regenerates Q&A data, which is a pair of a question and an answer related to the second target data, based on the second target data. The Q&A data regeneration unit 1023 may regenerate Q&A data, which is a pair of a question and an answer related to the second target data, based on the viewpoint generated by the viewpoint generation unit 1021 and the second target data. The Q&A data regeneration unit 1023 may use a large-scale language model that is the same as any of the above-mentioned first to fifth large-scale language models, or may use a large-scale language model different from the first to fifth large-scale language models.

[0078] In the aforementioned prompt in which a score is assigned in the range of 1 to 10, for example, the predetermined standard may be "8," and a score of 8 or higher is considered to satisfy the predetermined standard. The predetermined standard may be determined arbitrarily. The "predetermined standard" shown here is merely an example, and the present disclosure is not limited thereto. For example, the "predetermined standard" may be set appropriately, such as "4" when a score is assigned in the range of 1 to 5. Alternatively, the "predetermined standard" may be set as an average score of X points or higher for all Q&A data 1016. Alternatively, the "predetermined standard" may be set as a combination of X points or higher for graphs, Y points or higher for tables, and Z points or higher for text.

[0079] In this manner, in the present disclosure, the "predetermined standard" may be set in various ways. Also, in the present disclosure, for example, when the average score for all of the Q&A data 1016 is X points or more, or when a standard such as X points or more in the case of a graph or Y points or more in the case of a table is met, it can be considered that "all of the Q&A data 1016 corresponding to the plurality of target data 1014 meet the predetermined standard."

[0080] The Q&A data regeneration unit 1023 outputs the regenerated new Q&A data 1016 to the first output unit 1022. The Q&A data regeneration unit 1023 may store the regenerated new Q&A data 1016 in the storage unit 101. The first output unit 1022 outputs the first determination result regarding the accuracy of the new Q&A data 1016 with respect to the second target data by the method described above.

[0081] The Q&A data regeneration unit 1023 may cause the user, having referred to the first determination result, to generate new Q&A data 1016. For example, the Q&A data regeneration unit 1023 outputs a screen or a sound instructing the user to generate new Q&A data 1016, and accepts the new Q&A data 1016 from the user. Then, the Q&A data regeneration unit 1023 outputs the new Q&A data 1016 accepted from the user to the first output unit 1022. The Q&A data regeneration unit 1023 can also be realized by such a configuration.

[0082] The above operation may be performed for all of the Q&A data 1016 corresponding to the plurality of target data 1014. Alternatively, the information processing system 1 may not include the Q&A data regeneration unit 1023. In this case, the information processing system 1 may omit the process related to the regeneration of the Q&A data described above.

[0083] After all of the Q&A data corresponding to the multiple first target data satisfy a predetermined standard, the answer acquisition unit 1024 operates as follows. Specifically, of a pair of a first question and a first answer to the first question included in the Q&A data, the answer acquisition unit 1024 inputs the first question into the RAG system and acquires the second answer to the first question from the RAG system. The RAG system is a RAG system that is operated in a production environment, a test environment, a development environment, or the like.

[0084] The second output unit 1025 outputs a second determination result regarding the accuracy of the second answer relative to the first answer. The second output unit 1025 may be realized by a large-scale language model (sixth large-scale language model). The second determination result includes at least one of a determination value regarding the accuracy and a determination reason.

[0085] The second output unit 1025 receives, for example, the following prompt: Example prompts You are the quality judge for the Q&A system. Please evaluate the system's answer to the expected question (answer 2) by carefully comparing it with the model answer (answer 1). Pay close attention to the differences between the model answer and the system's answer when explaining numbers and units. # Expected question: ${QUESTION} # Model answer: ${{REF_ANSWER} # System answer: ${SYSTEM_ANSWER} Please rate your results on a scale of 1 to 10, with the model answer being 10 points. If there are even partial differences between the system answer and the information on which the model answer is based, please rate it 5 points or lower. If the information is accurate but the conclusion is simply missing, please rate it 7 points or higher. Then, the second output unit 1025 outputs, for example, the following second determination result. ·Second judgment result example 1 "Score: 5" "Reason: In addition to two correct answers, there is one incorrect answer, so I gave it 5 points." ·Second judgment result example 2 "Score: 3" "Reason: The first half of your answer is correct, but the second half contradicts the first half. Therefore, I gave it 3 points." The second output unit 1025 may output a second determination result including a score for each piece of content. For example, the second output unit 1025 outputs a second determination result including scores of "8.68" for a graph, "7.75" for a table, "8.58" for a figure other than a graph, and "7.56" for plain text only. In this way, the second output unit 1025 may output a second determination result including digits after the decimal point. Similarly, the second output unit 1025 may output a second determination result including a score for each perspective included in the second target data.

[0086] The second output unit 1025 may store the second determination result in the storage unit 101 or an external storage device.

[0087] The prompt may be set as a default, or the receiving unit 1026 may receive at least one of a user's change to a predetermined (default) prompt and a user's setting of a new prompt, and the second output unit 1025 may input the received prompt to the large-scale language model.

[0088] According to the above configuration, the information processing system 1 can output the answer accuracy of the RAG system as the second judgment result. By having the user confirm the answer accuracy of the RAG system as the second judgment result, the information processing system 1 can promote the construction of a RAG system with less erroneous information (hallucination) and improved answer accuracy.

[0089] The sixth large-scale language model may be the same as the first to fifth large-scale language models described above, or another large-scale language model may be used. The fifth large-scale language model may also be composed of multiple large-scale language models. <Recommendation function> Referring back to FIG. 2, the control unit 102 may include a recommendation output unit 1027 and an optimization unit 1028.

[0090] The recommendation output unit 1027 outputs a recommendation according to the second determination result. The recommendation output unit 1027 may be realized by a large-scale language model such as a Generative Pretrained Transformer (GPT), but there are no limitations on the scale or structure of the model.

[0091] As an example, the recommendation output unit 1027 receives the following prompt based on the second determination result output by the second output unit 1025. Example prompts Please explain the categories that received high and low ratings from the test results below, and make recommendations for the areas that received low ratings. # Test results Model used: gemini-pro-1.0 Layout Parser: OFF Plain text: 7.5 Image data: 7.2 Table data: 3.4" Then, the recommendation output unit 1027 outputs, for example, the following recommendation. Example of a rating comment "Evaluation Results: The test results showed that answers regarding plain text and image data were appropriate and highly rated, but answers regarding table data were not rated as desirable. Recommendations: 1) Setting recommendation: The model currently being used is an old model, so we recommend using the latest stable model. 2) We recommend turning on the Layout Parser function. 3) Document content recommendations: We recommend not using symbols in table data. We recommend that table data be structured as simply as possible. In this way, the recommendation output unit 1027 analyzes the tendency of results with low scores in the second judgment result and can output a recommendation such as, for example, "The model that is most suitable for the current document is Gemini 1.5 pro." This can be obtained by instructing the recommendation output unit 1027, which is realized by a large-scale language model, to output a recommendation of the analysis result. These are just examples, and the recommendation output unit 1027 can output recommendations for the setting values ​​of the RAG system itself, such as the type of large-scale language model or the parameters to be used. The recommendation output unit 1027 can also output recommendations for the content of the target document (or target data) that is referenced by the RAG system. In this way, the recommendation output unit 1027 can output recommendations over a wide range of content (scope).

[0092] If the second determination result does not satisfy a predetermined standard, the optimization unit 1028 performs optimization based on the recommendation output by the recommendation output unit 1027. For example, as an example of optimization, the optimization unit 1028 performs processing such as changing the large-scale language model used by the RAG system to Gemini 1.5 pro (selecting the optimal LLM) or changing (tuning) the parameters of the large-scale language model used by the RAG system, thereby improving the second determination result. Alternatively, as another example, the optimization unit 1028 may perform processing such as changing the writing style of the target document referenced by the RAG system, for example, by avoiding the use of symbols in the table data used, thereby improving the second determination result.

[0093] When the second judgment result does not satisfy the predetermined standard, for example, a score of "7 or more" in the second judgment result is defined as the predetermined standard. In this case, "Table data: 3.4" does not satisfy the predetermined standard, while "Plain text: 7.5" and "Image data: 7.2" do. The optimization unit 1028 determines that "Table data: 3.4" does not satisfy the predetermined standard and executes the above-mentioned optimization process. When all scores related to the second judgment result satisfy the predetermined standard, the optimization unit 1028 may terminate the process.

[0094] After performing the optimization process based on the recommendation, the optimization unit 1028 causes the answer acquisition unit 1024 to provide (acquire) a second answer. The subsequent process is as described above.

[0095] The information processing system 1, having the above configuration, can improve the accuracy of answers provided by the RAG system. Unlike conventional Q&A reference chatbots, the information processing system 1 can be configured to automatically generate high-quality Q&A data simply by a user storing internal company documents in a predetermined folder (such as a data store). <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service (e.g., a chatbot). The user terminal 20 receives, for example, an input operation from a user who uses a chatbot, and operates the control unit 102 of the server 10. The user terminal 20 may be, for example, a desktop personal computer (PC), a laptop PC, a mobile terminal such as a smartphone or a tablet, an HMD (Head Mount Display), or a wearable terminal such as a wristwatch terminal.

[0096] The user terminal 20 includes a storage unit 201 , a control unit 202 , an input unit 203 , and an output unit 204 . <Storage unit 201 of user terminal 20> The storage unit 201 of the user terminal 20 stores an application program 2011. The storage unit 201 may also store a user ID 2012.

[0097] The application program 2011 may be pre-stored in the storage unit 201. The application program 2011 may be downloaded from a web server operated by a service provider via a communication IF. The application program 2011 may include an application such as a web browser application.

[0098] The user ID 2012 is the user's account ID. The user transmits the user ID 2012 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2012 and provides a service to the user. If a service (e.g., a chatbot) does not require a user ID, the storage unit 201 does not need to store the user ID 2012. <Control Unit 202 of User Terminal 20> The control unit 202 of the user terminal 20 includes an input control unit 2021 and an output control unit 2022. The control unit 202 executes an application program 2011 stored in the storage unit 201 to realize each functional unit. <Input unit 203 of user terminal 20> The input unit 203 of the user terminal 20 may include a camera 2031, a microphone 2032, a keyboard 2033, a touch panel 2064, and the like. <Output unit 204 of user terminal 20> The output unit 204 of the user terminal 20 may include a display 2041 and a speaker 2042 . <Configuration of administrator terminal 30> 1, the administrator terminal 30 is an information processing device operated by an administrator of a service (e.g., a chatbot). The administrator is also the operator of the information processing system 1. Through the administrator terminal 30, the administrator builds a large-scale language model to be used in the information processing system 1, or determines or modifies prompts to be input to the large-scale language model.

[0099] The administrator terminal 30 may be, for example, a desktop personal computer (PC), a laptop PC, a mobile terminal such as a smartphone or a tablet, an HMD (Head Mount Display), or a wearable terminal such as a wristwatch terminal.

[0100] <Example of processing by information processing system 1> Next, the flow of processing by the information processing system 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the flow of information processing according to the present disclosure.

[0101] In S1, the target document division unit 1020A divides the target document into predetermined units.

[0102] In S2, the content division unit 1020B divides one or more pieces of content included in the first target data obtained by dividing the first target data into pages by the target document division unit 1020A. The pieces of content are, for example, text, tables, graphs, figures, and images.

[0103] In S3, the conversion information generation unit 1020C converts the divided content into a conversion format corresponding to each divided content and generates conversion information. In other words, the conversion information generation unit 1020C processes the divided content for each content in a conversion format suitable for RAG.

[0104] In S4, the second target data generation unit 1020D generates second target data based on the conversion information generated by the conversion information generation unit 1020C.

[0105] In S5, the Q&A data generating unit 1020E generates Q&A data, which is a pair of questions and answers related to the second target data, based on the second target data.

[0106] After S4, the process may proceed to S6. In S6, the viewpoint generation unit 1021 generates a viewpoint for the second target data. The viewpoint generation unit 1021 outputs the generated viewpoint to the Q&A data generation unit 1020E. Then, the Q&A data generation unit 1020E may generate the above-mentioned Q&A data based on the second target data and the viewpoint.

[0107] In S8, the first output unit 1022 outputs the first determination result regarding the accuracy of the Q&A data for the second target data.

[0108] If the first determination result does not satisfy the predetermined standard (NO in S10), the process proceeds to S12. In S12, the Q&A data regeneration unit 1023 regenerates new Q&A data. Then, in S8, the first output unit 1022 outputs the first determination result regarding the accuracy of the new Q&A data for the second target data. The first determination result may include a score for each content.

[0109] In S10, if the first determination result satisfies the predetermined criterion (YES in S10), the process proceeds to S 14. After all of the Q&A data corresponding to the plurality of first target data satisfy the predetermined criterion, in S14, the answer acquisition unit 1024 inputs the first question, of the pair of the first question and the first answer to the first question included in the Q&A data, to the RAG system, and acquires the second answer to the first question from the RAG system.

[0110] In S16, the second output unit 1025 outputs a second determination result regarding the accuracy of the second answer relative to the first answer. The second determination result may include a score for each piece of content.

[0111] In S18, the recommendation output unit 1027 outputs a recommendation according to the second determination result.

[0112] In S20, if the second determination result satisfies the predetermined criterion (YES in S20), the processing ends. In S20, if the second determination result does not satisfy the predetermined criterion (NO in S20), in S22, the optimization unit 1028 executes optimization based on the recommendation output by the recommendation output unit 1027. After executing the optimization process based on the recommendation, the optimization unit 1028 once again causes the second output unit 1025 to return (obtain) a second answer (proceed to S14).

[0113] <Chatbot> Next, a chatbot 50 according to the present disclosure will be described with reference to Figure 5. Figure 5 is a schematic diagram of a chatbot 50 according to the present disclosure.

[0114] The chatbot 50 is an example of a RAG system in which answer accuracy has been improved by the information processing system 1 according to the present disclosure. The chatbot 50 includes a reception unit 51 that receives questions from users and an answer presentation unit 52 that presents answers to the questions. This allows the chatbot 50 to provide the user with a RAG system with improved answer accuracy.

[0115] <Software implementation example> The functions of the information processing system 1 (hereinafter referred to as the "device") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in the control unit 102).

[0116] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0117] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0118] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0119] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server). <Summary> The present disclosure describes at least the following aspects.

[0120] A method for generating Q&A data according to aspect 1 of the present disclosure is a method for generating Q&A data for evaluating the accuracy of answers provided by a question-answering system, and includes the steps of: (a) dividing, by content, one or more contents contained in each of a plurality of first target data pieces obtained by dividing a target document into predetermined units using a first large-scale language model; (b) converting, by a second large-scale language model, the divided contents into a conversion format corresponding to each of the divided contents to generate conversion information; (c) generating second target data based on the conversion information; and (d) generating Q&A data, which are pairs of questions and answers related to the second target data, based on the second target data using a third large-scale language model.

[0121] According to the above configuration, the Q&A data generation method according to the first aspect of the present disclosure generates conversion information corresponding to each piece of content included in each piece of first target data. The conversion information structures the content classified by type and formats it in a format that is easy for a large-scale language model to read. Then, the Q&A data generation method according to the first aspect of the present disclosure generates Q&A data, which are pairs of questions and answers related to the second target data, based on the second target data generated based on the conversion information.

[0122] According to the above configuration, the Q&A data generation method according to the first aspect of the present disclosure can generate Q&A data with high accuracy for evaluating the answer accuracy of a question and answer system.

[0123] In the method for generating Q&A data according to aspect 2 of the present disclosure, in the step (c) of aspect 1, the second target data is generated by adding the conversion information to a first target data among the plurality of first target data that contained the content, or is generated based on the conversion information alone.

[0124] Various cases are conceivable, such as when the first target data includes multiple pieces of content, when it includes only a single piece of content, etc. According to the above configuration, the Q&A data generation method according to the second aspect of the present disclosure can flexibly change the method for generating second target data according to various cases.

[0125] Therefore, the Q&A data generation method according to the second aspect of the present disclosure can generate Q&A data with even higher accuracy for evaluating the answer accuracy of a question and answering system.

[0126] A method for generating Q&A data according to aspect 3 of the present disclosure, in accordance with aspect 1 or 2, includes a step of generating a viewpoint for the second target data using a fourth large-scale language model, and in step (d), the third large-scale language model generates the Q&A data based on the second target data and the viewpoint.

[0127] According to the above configuration, the Q&A data generating method according to the third aspect of the present disclosure generates Q&A data based on the viewpoint generated for the second target data and the second target data.

[0128] Therefore, the Q&A data generation method according to the third aspect of the present disclosure can generate Q&A data with even higher accuracy for evaluating the answer accuracy of a question and answering system.

[0129] A method for generating Q&A data according to aspect 4 of the present disclosure is any one of aspects 1 to 3, wherein the conversion format includes at least one of Markdown, Vega-Lite, PlantUML, Mermaid, Tabular Data Package, DOT, JSON, HTML, XML, YAML, TOML, RDF, SGML, BibTeX, LaTeX, GML, GraphML, Asciidoc, R Markdown, UML, BPMN, TikZ, GeoJSON, Ditaa, Chart.js, and Plotly.

[0130] According to the above configuration, the method for generating Q&A data according to the fourth aspect of the present disclosure is not limited to a specific conversion format, and various conversion formats can be adopted.

[0131] In the method for generating Q&A data according to aspect 5 of the present disclosure, in aspect 4, Markdown, Markdown, Vega-Lite, PlantUML, and Markdown are used, in that order, as the conversion formats corresponding to the content of text, tables, graphs, figures, and images.

[0132] Markdown, Markdown, Vega-Lite, PlantUML, and Markdown are suitable conversion formats for text, tables, graphs, diagrams, and images, respectively. The Q&A data generation method according to aspect 5 of the present disclosure generates desirable conversion information according to the content, thereby structuring the segmented content into a format that is more readable for a large-scale language model.

[0133] Therefore, the Q&A data generation method according to the fifth aspect of the present disclosure can generate Q&A data with even higher accuracy for evaluating the answer accuracy of a question and answering system.

[0134] A sixth aspect of the present disclosure provides a method for generating Q&A data in any one of the first to fifth aspects, wherein the second large-scale language model determines the conversion format depending on the content.

[0135] Specifically, the second large-scale language model may determine the conversion format depending on the content, without associating the content with the conversion format in advance.

[0136] According to the above configuration, the Q&A data generation method according to the sixth aspect of the present disclosure can more flexibly generate Q&A data for evaluating the answer accuracy of a question and answer system.

[0137] A method for generating Q&A data according to a seventh aspect of the present disclosure is any one of the first to sixth aspects, and further includes (e) outputting a first determination result regarding the accuracy of the Q&A data for the second target data.

[0138] According to the above configuration, the method for generating Q&A data according to the seventh aspect of the present disclosure can objectively evaluate the accuracy of the Q&A data for the second target data based on the first determination result.

[0139] A method for generating Q&A data according to aspect 8 of the present disclosure includes, in aspect 7, (f) a step of providing regenerated new Q&A data to step (e) if the first judgment result does not satisfy a predetermined standard.

[0140] According to the above configuration, the Q&A data generation method according to the eighth aspect of the present disclosure regenerates the Q&A data until the first determination result regarding the accuracy of the Q&A data satisfies a predetermined standard.

[0141] Therefore, the Q&A data generation method according to the eighth aspect of the present disclosure can generate Q&A data with even higher accuracy for evaluating the answer accuracy of a question and answer system.

[0142] A method for generating Q&A data according to aspect 9 of the present disclosure includes, in aspect 8, (g) after all of the Q&A data corresponding to the plurality of first target data satisfy the predetermined criteria, a step of inputting a first question from a pair of a first question and a first answer to the first question contained in the Q&A data into the question answering system and providing a second answer to the first question from the question answering system, and (h) a step of outputting a second judgment result regarding the accuracy of the second answer to the first answer.

[0143] According to the above configuration, the Q&A data generation method according to the ninth aspect of the present disclosure can output the answer accuracy of the question and answer system as the second determination result.

[0144] Therefore, the Q&A data generation method according to aspect 9 of the present disclosure can promote the construction of a question-answering system with less false information (hallucination) and improved answer accuracy by allowing the user to confirm the answer accuracy of the question-answering system as a second judgment result.

[0145] A tenth aspect of the present disclosure provides a method for generating Q&A data in the ninth aspect, wherein the first determination result and the second determination result include at least one of a determination value and a determination reason regarding the accuracy.

[0146] According to the above configuration, the first determination result and the second determination result include a quantitative criterion, i.e., a determination value, and / or a qualitative reason, i.e., a determination reason. As a result, the Q&A data generation method according to aspect 10 of the present disclosure is expected to further improve answer accuracy.

[0147] The Q&A data generation method according to aspect 11 of the present disclosure is in accordance with aspect 10, further including: (i) outputting a recommendation according to the second determination result.

[0148] According to the above configuration, after verifying the accuracy of the question answering system using Q&A data, recommendations are output based on the second judgment result, thereby making it possible to understand how to further improve the answer accuracy of the question answer based on the second judgment result.

[0149] The method for generating Q&A data according to aspect 12 of the present disclosure further includes, in aspect 11, (j) a step of optimizing the question and answering system based on the recommendation if the second judgment result does not satisfy a predetermined standard.

[0150] According to the above configuration, after verifying the accuracy of the question and answering system using Q&A data, a recommendation is output according to the second determination result. Then, by optimizing the question and answering system based on the recommendation, the answer accuracy of the question and answering system can be further improved based on the second determination result.

[0151] In the method for generating Q&A data according to aspect 13 of the present disclosure, in aspect 3, the aspects include at least one of the title, summary, type of figure / table / text, conclusion, background, and purpose of the second target data.

[0152] According to the above configuration, the method for generating Q&A data according to aspect 13 of the present disclosure can generate Q&A data with higher accuracy by setting many viewpoint items.

[0153] The Q&A data generation program according to aspect 14 of the present disclosure is a computer-executable Q&A data generation program that causes a computer to execute the following steps: (a) using a first large-scale language model, divide one or more contents contained in each of a plurality of first target data obtained by dividing a target document into predetermined units, by content; (b) using a second large-scale language model, converting each of the divided contents into a conversion format corresponding to each of the divided contents to generate conversion information; (c) generating second target data based on the conversion information; and (d) using a third large-scale language model, generating Q&A data that is a pair of questions and answers related to the second target data based on the second target data.

[0154] According to the above configuration, the Q&A data generation program according to the fourteenth aspect of the present disclosure achieves the same effects as the Q&A data generation method according to the first aspect.

[0155] A Q&A data generation system according to aspect 15 of the present disclosure is a Q&A data generation system for evaluating the answer accuracy of a question answering system, and includes a division unit that divides one or more contents contained in each of a plurality of first target data obtained by dividing a target document into predetermined units into pieces of content, a conversion information generation unit that converts each of the divided contents into a conversion format corresponding to the divided contents to generate conversion information, a second target data generation unit that generates second target data based on the conversion information, and a Q&A data generation unit that generates Q&A data that is a pair of questions and answers related to the second target data based on the second target data.

[0156] According to the above configuration, the Q&A data generation system according to the fourteenth aspect of the present disclosure achieves the same effects as the Q&A data generation method according to the first aspect.

[0157] The Q&A data generation system according to aspect 15 of the present disclosure, in aspect 16, includes a reception unit that receives at least one of a user's change to a predetermined prompt to be input into a large-scale language model used in the Q&A data generation system and a user's setting of a new prompt.

[0158] According to the above configuration, the Q&A data generation system according to the sixteenth aspect of the present disclosure can easily reflect the user's intentions.

[0159] (Additional notes) The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0160] 1. Information Processing Systems 10 Servers 20 User terminal 30 Administrator terminal 50 Chatbots 1026 Reception 101, 201 Storage section 102, 202 control section 203 Input section 204 Output section 1011, 2011 Application Program 1012 Large-scale language models 1013 Target Documents 1014 Target Data 1015 Conversion Information 1020 Q&A Data Generation System 1020A Target document division unit 1020B Content division section 1020C Conversion information generation unit 1020D Second target data generation unit 1020E Q&A Data Generation Department 1021 Viewpoint Generation Unit 1022 First output section 1023 A data regeneration section 1024 Answer acquisition part 1025 Second Output Unit 1027 Recommendation Output Unit 1028 Optimization Department

Claims

1. A method for generating Q&A data for evaluating answer accuracy of a question-answering system, comprising: (a) dividing, by a first large-scale language model, one or more pieces of content included in each of a plurality of first target data pieces obtained by dividing a target document into predetermined units; (b) generating conversion information by converting the divided content into conversion formats corresponding to each of the divided content using a second large-scale language model; (c) generating second target data based on the conversion information; (d) generating Q&A data, which are pairs of questions and answers related to the second target data, based on the second target data using a third large-scale language model; In the step (c), the second target data is generated by adding the conversion information to a first target data item that includes the content among the plurality of first target data items, or is generated based on the conversion information alone; A method for generating Q&A data, wherein the second large-scale language model receives a prompt instructing it to determine a conversion format suitable for the content, and determines the conversion format according to the content.

2. generating a viewpoint for the second target data using a fourth large-scale language model; The Q&A data generating method according to claim 1 , wherein in step (d), the third large-scale language model generates the Q&A data based on the second target data and the viewpoint.

3. 3. The Q&A data generation method according to claim 1 or 2, wherein the conversion format includes at least one of Markdown, Vega-Lite, PlantUML, Mermaid, Tabular Data Package, DOT, JSON, HTML, XML, YAML, TOML, RDF, SGML, BibTeX, LaTeX, GML, GraphML, Asciidoc, R Markdown, UML, BPMN, TikZ, GeoJSON, Ditaa, Chart.js, and Plotly.

4. The Q&A data generation method according to claim 3, wherein Markdown, Markdown, Vega-Lite, PlantUML, and Markdown are used as the conversion formats corresponding to the content of text, tables, graphs, figures, and images, in that order.

5. 3. The method for generating Q&A data according to claim 1, further comprising the step of: (e) inputting the second target data and the Q&A data into a large-scale language model; and outputting a first judgment result regarding the accuracy of the Q&A data relative to the second target data.

6. 6. The method for generating Q&A data according to claim 5, further comprising: (f) a step of, if the first judgment result does not satisfy a predetermined standard, providing regenerated new Q&A data to said step (e).

7. (g) after all of the Q&A data corresponding to the plurality of first target data satisfy the predetermined criterion, inputting the first question, of a pair of a first question and a first answer to the first question included in the Q&A data, into the question and answering system, and providing a second answer to the first question from the question and answering system; (h) inputting the first answer and the second answer into a large-scale language model and outputting a second determination result including at least one of a determination value and a determination reason regarding the accuracy of the second answer relative to the first answer.

8. (i) inputting the second determination result into a large-scale language model and outputting recommendations according to the second determination result, the recommendations including at least one of recommendations for the content of the target document and recommendations for setting values ​​to be used in the question-answering system.

9. (j) if the second judgment result does not satisfy a predetermined standard, executing optimization of the question answering system based on the recommendation.

10. The Q&A data generating method according to claim 2 , wherein the viewpoints include at least one of a title, an abstract, a type of figure / table / text, a conclusion, a background, and a purpose of the second target data.

11. A computer-executable Q&A data generation program, (a) dividing, by a first large-scale language model, one or more pieces of content included in each of a plurality of first target data pieces obtained by dividing a target document into predetermined units; (b) generating conversion information by converting the divided content into conversion formats corresponding to each of the divided content using a second large-scale language model; (c) generating second target data based on the conversion information; (d) generating Q&A data, which is a pair of questions and answers related to the second target data, based on the second target data by using a third large-scale language model, In the step (c), the second target data is generated by adding the conversion information to a first target data item that includes the content among the plurality of first target data items, or is generated based on the conversion information alone; A Q&A data generation program in which a prompt is input to instruct the second large-scale language model to determine a conversion format suitable for the content, and the second large-scale language model determines the conversion format according to the content.

12. A system for generating Q&A data for evaluating answer accuracy of a question-answering system, comprising: a division unit that divides the target document into a plurality of first target data pieces each obtained by dividing the target document into predetermined units, into one or more pieces of content included in each of the plurality of first target data pieces; a conversion information generating unit that converts the divided content into conversion formats corresponding to the divided content and generates conversion information; a second target data generation unit that generates second target data based on the conversion information; a Q&A data generation unit that generates Q&A data, which is a pair of questions and answers related to the second target data, based on the second target data; the second target data is generated by adding the conversion information to a first target data item that includes the content among the plurality of first target data items, or is generated based only on the conversion information; A Q&A data generation system in which the conversion information generation unit is composed of a large-scale language model, and a prompt is input to instruct the unit to determine a conversion format suitable for the content, and the conversion format is determined according to the content.

13. 13. The Q&A data generation system according to claim 12, further comprising a reception unit that receives at least one of a user's change to a predetermined prompt to be input to a large-scale language model used in the Q&A data generation system and a user's setting of a new prompt.

Images