Generation device, generation method, and generation program

Abstract

To clarify a boundary between correct answers.SOLUTION: The generation device includes an extraction unit that extracts a first query and a second query from the query group, an input unit that receives an input of a correct answer to the first query and associates the correct answer with the first query, and an output unit that outputs the first query and the second query to the language model. A generation unit configured to generate an inquiry for requesting the language model to present a specific second query and a correct answer candidate of the specific second query, the specific second query allowing the language model to estimate a correct answer of another second query from the second query by referring to a combination of the first query and the correct answer and knowing the correct answer; SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a generating device, a generating method, and a generating program for generating data. [Background technology]

[0002] Non-Patent Document 1 discloses Active-Prompt, a new method for adapting large language models (LLMs) to various tasks using task-specific example prompts annotated with human-designed thought-chain reasoning.

[0003] Non-Patent Document 2 discloses vote-k, an unsupervised graph-based selective annotation method for selecting diverse and representative examples to annotate, based on a framework of pre-selecting a pool of examples to annotate from unlabeled data and obtaining task examples from the annotated pool at test time.

[0004] Non-Patent Document 3 discloses Evoke, an automatic prompt refinement framework. Evoke has two instances of the same LLM: one acts as a reviewer (LLM-Reviewer) and scores the current prompt, and the other acts as an author (LLM-Author) and edits the prompt taking into account the edit history and reviewer feedback. This feedback loop between the author and reviewer refines the prompt with each iteration. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Diao, Shizhe, et al. "Active prompting with chain-of-thought for large language models." arXiv preprint arXiv:2302.12246 (2024). [Non-patent document 2] Su, Hongjin, et al. "Selective annotation makes language models better few-shot learners." arXiv preprint arXiv:2209.01975 (2022). [Non-patent document 3] Hu, Xinyu, et al. "Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing." arXiv preprint arXiv:2310.13855 (2024). Summary of the Invention [Problem to be solved by the invention]

[0006] The above-mentioned Non-Patent Documents 1 and 2 do not take into consideration the extraction of queries in consideration of answer information.

[0007] The above-mentioned Non-Patent Document 3 assumes that correct answers are assigned in advance, and does not take into consideration the construction of a representative correct answer group by assigning answers.

[0008] The present invention aims to clarify the boundaries between correct answers. [Means for solving the problem]

[0009] A generation device according to one aspect of the invention disclosed in the present application is capable of accessing a database that stores a group of queries, and a language model that, when a natural sentence, which is text data in a natural language, is input, interprets the natural sentence and probabilistically predicts an answer sentence to the natural sentence, and is characterized by having: an extraction unit that extracts a first query and a second query from the group of queries; an input unit that accepts an input of a correct answer to the first query extracted by the extraction unit and associates it with the first query; a generation unit that generates a query requesting the language model to present a specific second query and a correct answer candidate for the specific second query, which allows the language model to infer a correct answer to another second query from the second query by referring to a combination of the first query and the correct answer to the first query and knowing the correct answer; and an acquisition unit that acquires an answer including the specific second query and the correct answer candidate for the specific second query from the language model as a result of the query generated by the generation unit being output to the language model. [Effects of the Invention]

[0010] According to the exemplary embodiment of the present invention, it is possible to clarify the boundary between correct answers. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram showing an example of an instruction to a language model. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. [Figure 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the generating device. [Figure 4] FIG. 4 is an explanatory diagram illustrating an example of the dataset DB. [Figure 5] FIG. 5 is an explanatory diagram showing an example of generating a prompt. [Figure 6] FIG. 6 is a flowchart illustrating a detailed example of a processing procedure of the data set generation example 1. [Figure 7] FIG. 7 is a flowchart showing a detailed example of a processing procedure of the data set generation example 2 (first half). [Figure 8] FIG. 8 is a flowchart showing a detailed example of the processing procedure of the second data set generation example (second half). [Figure 9] FIG. 9 is an explanatory diagram showing prompts and example answers in the data set generation example 2. DETAILED DESCRIPTION OF THE INVENTION

[0012] To obtain generalizability from a set of queries, the set should contain many boundary cases where it is difficult to uniquely determine the correct answer. A query refers to a question or problem statement for which an answer is sought. Methods for extracting representative queries that can predict the correct answers of other queries from a set of queries for which no correct answer has been given cannot, in principle, take into account the tendency of correct answers specific to each use case. For this reason, it is not possible to extract queries that conform to the developer's required specifications, such as boundary cases.

[0013] For example, in document classification, when classifying textbooks into beginner, intermediate, and advanced readers, the correct answer will differ (be biased) depending on the intended reader, so the classification boundaries for beginner, intermediate, and advanced readers depend on the requirements specifications.

[0014] Furthermore, in sentiment analysis, the string "difficult to predict" is often judged as negative, but if a query containing this string is a comment about a movie, it should be evaluated as positive.

[0015] This embodiment semi-automatically determines the developer's required specifications, and identifies a representative query from a group of queries for which no correct answer has been given, from which the correct answer for the other queries can be inferred. This improves the degree to which the developer's required specifications are fulfilled. The following describes this embodiment with reference to the accompanying drawings.

[0016] <Figure 1 Example of instructions to a language model> FIG. 1 is an explanatory diagram showing an example of instructions to a language model. The language model 100 is, for example, an LLM, and also has a dataset for learning. When a natural sentence, which is text data in a natural language, is input, the language model 100 interprets the natural sentence and probabilistically predicts a response sentence to the natural sentence. In other words, the language model 100 is a type of probabilistic model used in natural language processing, and is a model for probabilistically predicting how likely a given word or sentence is to occur in natural language.

[0017] Specifically, in the field of natural language processing, the language model 100 is a mathematical model for learning language patterns, grammatical rules, etc., and generating and understanding natural language. For example, the language model 100 calculates the occurrence probability of a given word string or sentence, or compares the occurrence probabilities of multiple word strings or sentences, thereby automatically generating the most likely word or sentence based on the context when predicting the next word or sentence.

[0018] In this way, the language model 100, which has been trained on a huge amount of data set, receives a query called a prompt 101 and generates an answer 102 to that query.

[0019] Prompt 101 has, as templates, a first fixed phrase 111 and a second fixed phrase 112. First fixed phrase 111 is an instruction statement that specifies a query and correct answer pair 113 (in FIG. 1, a query Q3 and its correct answer A3).

[0020] The second template 112 is a directive that instructs extraction of "a query whose correct answer makes it possible to guess the correct answers of other queries by knowing the correct answer" (hereinafter referred to as a representative query). "The correct answer" is the correct answer of the representative query. In this example, as shown in the answer 102, the representative query 121 is the query Q2, and "the correct answer" is the correct answer 123 (A2). Note that the query Q3 specified in the first template 111 is also a representative query.

[0021] The "next query" in the second template 112 is a representative query candidate 114 that includes the representative query 121. In the prompt 101, queries Q1, Q2, Q4, and Q5 are exemplified as the representative query candidates 114.

[0022] The "other queries" in the second template 112 are queries other than the representative query 121 among the representative query candidates 114. In this example, the "other queries" in the second template 112 are the queries Q1, Q4, and Q5 among the representative query candidates 114, excluding query Q2, which will be the representative query 121. However, when the prompt 101 is generated, it is not known which queries fall under the representative query 121 (Q2) and the "other queries" (Q1, Q2, Q4, Q5).

[0023] The answer 102 has a selected representative query 121. The selected representative query 121 is a query selected by the language model 100 from among the representative query candidates 114 by interpreting the prompt 101. In FIG. 1, the query Q2 is selected as the selected representative query 121 as an example.

[0024] 1, the two adjacent classes are intermediate and advanced. That is, by knowing “A2,” which is the correct answer 123 of the selected representative query 121, the language model 100 can predict the correct answers A1, A4, and A5 of the other queries Q1, Q4, and Q5.

[0025] Terminal 130 is a computer operated by developer D, and developer D operates it to display prompt 101, output prompt 101 to language model 100, obtain answer 102 from language model 100, and display answer 102. Developer D operates terminal 130 to refer to the displayed answer 102 and assign a correct answer 123 appropriate for selected representative query 121. In the example of FIG. 1 , developer D assigns correct answer A2, which indicates either intermediate or advanced level, to query Q2, which is the selected representative query 121. By assigning correct answer 123, language model 100 becomes able to infer correct answers A1, A4, and A5 for other queries Q1, Q4, and Q5.

[0026] After this, query Q2 and its correct answer A2 become available as pair 113. By repeating this process, developer D's requirements are semi-automatically determined, making it possible to identify queries where the boundaries between classes are ambiguous or difficult to distinguish. Therefore, it becomes possible to automatically generate prompts with higher performance with the same number of correct answers as with existing technologies. Note that Figure 1 explains an example of use by developer D, and the following explanation also uses developer D as an example, but the user may be someone other than developer D.

[0027] <Figure 2 Example of computer hardware configuration> FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. The computer 200 includes a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, the storage device 202, the input device 203, the output device 204, and the communication IF 205 are connected via a bus 206. The processor 201 controls the computer 200. The storage device 202 serves as a working area for the processor 201. The storage device 202 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 202 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 203 inputs data. Examples of the input device 203 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 204 outputs data. The output device 204 may be, for example, a display, a printer, or a speaker. The communication IF 205 connects to a network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network) to send and receive data.

[0028] The computer 200 executes the processing shown in Fig. 1. The computer 200 may also have the language model 100, and may be capable of communicating with another computer 200 that implements the language model 100. The computer 200 may also be the terminal 130, and may be capable of communicating with the terminal 130.

[0029] <Figure 3: Example of functional configuration of generation device> 3 is a block diagram showing an example of the functional configuration of a generating device. The generating device 300 is configured by a computer 200. The generating device 300 has an extracting unit 301, an inputting unit 302, a generating unit 303, an acquiring unit 304, a calculating unit 305, and a selecting unit 306. Specifically, the extracting unit 301, the inputting unit 302, the generating unit 303, the acquiring unit 304, the calculating unit 305, and the selecting unit 306 are realized, for example, by causing the processor 201 to execute a program stored in the storage device 202 shown in FIG. 2.

[0030] The generating device 300 also includes a dataset DB 310. The dataset DB 310 is realized by the storage device 202 shown in FIG.

[0031] [Figure 4 Dataset DB310] FIG. 4 is an explanatory diagram showing an example of the dataset DB 310. The dataset DB 310 has a query 401 and a correct answer 402 as fields. In FIG. 4, queries Q1 to Q5, ... are stored as the queries 401, and correct answers A1 to A5, ... are stored as the correct answers 402. In FIG. 4, the query 401 indicates a math problem, and the correct answer 402 indicates the difficulty level of the query 401. Note that in FIG. 4, the correct answer 402 is "null." The correct answer 402 can be updated.

[0032] Returning to FIG. 3 , the extraction unit 301 extracts, as representative queries, queries 401 that do not have a correct answer 402 from the dataset DB 310. The extraction unit 301 also extracts, as representative query candidates, queries 401 that do not have a correct answer 402 other than the representative query from the dataset DB 310. A query 401 that does not have a correct answer 402 is a query 401 whose correct answer 402 is "null." The extraction unit 301 extracts representative queries and representative query candidates 114 from the dataset DB 310 randomly or by operation of developer D. Note that the number of representative queries and representative query candidates 114 extracted may be one or more.

[0033] 1, of the multiple queries Q1 to Q5 that do not have a correct answer 402, query Q3 that constitutes pair 113 is extracted as a representative query. Furthermore, of the multiple queries Q1 to Q5 that do not have a correct answer 402, the remaining queries Q1, Q2, Q4, and Q5 are extracted as representative query candidates 114.

[0034] The input unit 302 accepts input of a correct answer 402 by developer D operating the input device 203 or terminal 130. In the example of FIG. 1, the correct answer A3 that constitutes the pair 113 is input in the prompt 101. Also, in the example of FIG. 1, the correct answer A2 of query Q2, which is the selected representative query 121, is input in the answer 102. When the correct answer 402 is input, the input unit 302 associates query Q2, which is the representative query 121, with the accepted correct answer 402, and updates the dataset DB 310. As a result, the correct answer A2 of query Q2 is updated from "null".

[0035] The generation unit 303 generates a query. Specifically, for example, the generation unit 303 generates the prompt 101 shown in FIG. 1 by setting a pair 113 and a representative query candidate 114 in a template prompt having a first fixed phrase 111 and a second fixed phrase 112.

[0036] [Figure 5 Example of prompt generation] 5 is an explanatory diagram showing an example of prompt generation. In FIG. 5, the extraction unit 301 extracts query Q3 as a representative query from the dataset DB 310. The extraction unit 301 also extracts queries Q1, Q2, Q4, and Q5 from the dataset DB 310 as representative query candidates 114. When queries Q1 to Q5 are extracted, their correct answers 402 are "null."

[0037] The input unit 302 accepts the correct answer 402 (A3) for the query Q3, which is "beginner", associates the query Q3 with the accepted correct answer A3, and updates the dataset DB 310. As a result, the correct answer 402 for the query Q3 is updated from "null" to "beginner".

[0038] In a template prompt 500 including a first fixed phrase 111 and a second fixed phrase 112, the generation unit 303 sets a query Q3 and its correct answer A3 (beginner level) to form a pair 113 for the first fixed phrase 111, and sets queries Q1, Q2, Q4, and Q5 as representative query candidates 114 for the second fixed phrase 112. In this way, the prompt 101 is generated.

[0039] 3, the acquiring unit 304 acquires the answer 102 to the inquiry generated by the generating unit 303. Specifically, for example, the acquiring unit 304 outputs the prompt 101 to the language model 100, and acquires the answer 102 output from the language model 100.

[0040] Calculation unit 305 calculates the appearance probability and answer variance of query 401. Details of the calculation process by calculation unit 305 will be described later with reference to FIG.

[0041] The selection unit 306 selects a representative query from the representative query candidates 114 based on the calculation result calculated by the calculation unit 305. Details of the selection process by the selection unit 306 will be described later with reference to FIG.

[0042] <Figure 6 Dataset generation example 1> 6 is a flowchart showing a detailed example of a processing procedure of Dataset Generation Example 1. In Dataset Generation Example 1, a generation device 300 uses an extraction unit 301, an input unit 302, a generation unit 303, and an acquisition unit 304 to perform dataset generation.

[0043] (Step S601) The generating device 300 causes the extracting unit 301 to extract a representative query (e.g., query Q3) and representative query candidates 114 (e.g., queries Q1, Q2, Q4, and Q5) from the dataset DB 310. Through the extraction, the representative query (e.g., query Q3) and representative query candidates 114 (e.g., queries Q1, Q2, Q4, and Q5) are displayed on a display, which is an example of the output device 204. Then, the process proceeds to step S602.

[0044] (Step S602) The generation device 300 receives, via the input unit 302, an input of the correct answer 402 (e.g., "beginner") of the representative query extracted in step S601, and updates the dataset DB 310 by associating the representative query extracted in step S601 with the received correct answer 402. That is, developer D refers to the representative query (e.g., query Q3) displayed on a display, which is an example of the output device 204, and inputs the correct answer 402 (e.g., "beginner"). As a result, the correct answer 402 of query Q3, which is the representative query, is updated from "null" to "beginner". Then, the process proceeds to step S603.

[0045] (Step S603) 5, the generation device 300 generates a prompt 101 by using the generation unit 303 to set the representative query extracted in step S601, its correct answer 402 input in step S602, and the representative query candidate 114 extracted in step S601 to the template prompt 500. Then, the process proceeds to step S604.

[0046] (Step S604) The generation device 300 outputs the prompt 101 generated in step S603 to the language model 100 via the acquisition unit 304, and acquires the answer 102 generated by the language model 100. Then, the process proceeds to step S605.

[0047] (Step S605) The generation device 300 receives, via the input unit 302, an input of the correct answer 123 for the representative query 121 (query Q2 in FIG. 1) in the answer 102 acquired in step S604, and updates the dataset DB 310 by associating the representative query 121 (query Q2 in FIG. 1) with the received correct answer 402 (correct answer A2 in FIG. 1). As a result, the correct answer A2 for query Q2, which is the representative query 121, is updated from "null" to either intermediate or advanced. Then, the process proceeds to step S606.

[0048] (Step S606) The generating device 300 determines whether the termination condition is satisfied. Specifically, for example, the generating device 300 determines that the termination condition is satisfied when the number of loops returning to step S601 reaches a predetermined number, when the number of representative queries reaches a predetermined number, or when a query 401 whose correct answer 402 is “null” does not exist in the dataset DB 310 (step S606: Yes).

[0049] If the termination condition is not satisfied (step S606: No), the process proceeds to step S607. If the termination condition is satisfied (step S606: Yes), the generating device 300 ends the processing of data set generation example 1.

[0050] (Step S607) The generation device 300 extracts the representative queries and their correct answers in steps S601 and S605, and extracts representative query candidates from a group of queries other than the representative queries. Then, the process proceeds to step S603. As a result, steps S603 to S606 are repeatedly executed while increasing the number of representative queries and their correct answers.

[0051] In this way, according to Dataset Generation Example 1, the required specifications of Developer D are semi-automatically determined, a query 401 where the boundaries between classes are ambiguous or difficult to distinguish is identified as a representative query 121, and a correct answer 123 can be set based on the answer 102 from the language model 100. Therefore, automatic prompt generation with higher performance than existing technologies becomes possible.

[0052] <Figure 7 Dataset generation example 2 (first half)> 7 is a flowchart showing a detailed example of a processing procedure of dataset generation example 2 (first half). In dataset generation example 2, generation device 300 executes dataset generation using extraction unit 301, input unit 302, generation unit 303, acquisition unit 304, calculation unit 305, and selection unit 306.

[0053] (Step S701) The generating device 300 causes the extracting unit 301 to extract a comparison query from the dataset DB 310, either randomly or through an operation by developer D. The comparison query is a query 401 whose correct answer 402 is "null", and is displayed on a display, which is an example of the output device 204, after extraction. Here, as an example, the comparison query is assumed to be query Q4. Then, the process proceeds to step S702.

[0054] (Step S702) The generating device 300 receives, via the input unit 302, the input of the correct answer 402 of the comparison query extracted in step S701, associates the comparison query with the received correct answer 402, and updates the dataset DB 310. That is, developer D refers to the comparison query displayed on a display, which is an example of the output device 204, and inputs the correct answer 402. As a result, the correct answer A4 of query Q4, which is the comparison query, is updated from "null" to "advanced." Then, the process proceeds to step S703.

[0055] (Step S703) The generating device 300 causes the extracting unit 301 to extract a representative query (e.g., query Q3) and representative query candidates 114 (e.g., queries Q1, Q2, and Q5) from the dataset DB 310. Through the extraction, the representative query (e.g., query Q3) is displayed on a display, which is an example of the output device 204. Then, the process proceeds to step S704.

[0056] (Step S704) The generating device 300 receives, via the input unit 302, an input of the correct answer 402 (e.g., "beginner") of the representative query extracted in step S703, and updates the dataset DB 310 by associating the representative query extracted in step S703 with the received correct answer 402. That is, developer D refers to the representative query (e.g., query Q3) displayed on a display, which is an example of the output device 204, and inputs the correct answer 402 (e.g., "beginner"). As a result, the correct answer A3 of query Q3, which is the representative query, is updated from "null" to "beginner". Then, the process proceeds to step S705.

[0057] (Step S705) The generation device 300 determines whether the number of correct answers 402 that are not "null" is equal to or greater than a threshold. If it is equal to or greater than the threshold (step S705: Yes), data set generation example 2 ends. If it is not equal to or greater than the threshold (step S705: No), the process proceeds to step S801 in FIG. 8.

[0058] <Figure 8 Dataset generation example 2 (second half)> FIG. 8 is a flowchart showing a detailed example of the processing procedure of the second data set generation example (second half).

[0059] (Step S801) The generation device 300 generates, via the generation unit 303, a prompt 101 for each representative query candidate 114. Specifically, for example, the generation device 300 sets, via the generation unit 303, a known query 401 to which a correct answer 402 has been assigned, that correct answer 402, and the representative query candidate 114 as a template prompt 500, and generates a prompt for each representative query candidate 114. The known query 401 to which a correct answer 402 has been assigned is the query 401 (comparison query and representative query) to which the correct answer 402 has been assigned in steps S702 and S704 before step S801 is executed.

[0060] [Figure 9 Prompts and example answers] 9 is an explanatory diagram showing prompts and example answers in dataset generation example 2. Prompt 900 has a first template 901 and a second template 902. First template 901 is an instruction statement that specifies a pair of a known query and its correct answer. Second template 902 is an instruction statement that instructs the generation of a correct answer for representative query candidate 114, such that knowing the correct answer makes it possible to predict correct answers for other queries.

[0061] Pair 911 is a combination of query Q4, the comparison query set in steps S701 and S702, and its correct answer A4, "advanced." Pair 912 is a combination of query Q3, the representative query set in steps S703 and S704, and its correct answer A3, "beginner." Also, in FIG. 9, of queries Q1, Q2, and Q5 for which the correct answer 402 has not been determined, query Q1 is set as the representative query candidate 114. Although not shown, other prompts 900 are also generated in which the representative query candidates 114 are queries Q2 and Q5.

[0062] Answer 920 is data generated by language model 100 when prompt 900 is input to language model 100. Answer 920 is query Q1, which is representative query candidate 114, and its correct answer candidate 921. In FIG. 9, correct answer candidate 921 is correct answer A1: intermediate.

[0063] (Step S802) 8, the generation device 300, via the generation unit 303, determines whether or not there is an unselected prompt 900. If there is an unselected prompt 900 (step S802: Yes), the process proceeds to step S803. If there is no unselected prompt 900 (step S802: No), the process proceeds to step S810.

[0064] (Step S803) The generation device 300 selects one of the unselected prompts 900 via the generation unit 303. Then, the process proceeds to step S804.

[0065] (Step S804) The generating device 300 generates a predetermined number (for example, 10) of seed values ​​using the generating unit 303. The seed value is, for example, an integer value randomly selected from integer values ​​from 0 to 9. When the language model 100 generates a sentence, the seed value is used by the language model 100 to randomly select the next word in accordance with a conditional probability distribution. Specifically, for example, the seed value is used to set the initial state of a random number generator included in the language model 100. Then, the process proceeds to step S805.

[0066] (Step S805) The generating device 300 determines whether there are any unselected seed values ​​among the predetermined number of seed values ​​generated in step S804 by the generating unit 303. If there are any unselected seed values ​​(step S805: Yes), the process proceeds to step S806. If there are no unselected seed values ​​(step S805: No), the process proceeds to step S808.

[0067] (Step S806) The generating device 300 selects one unselected seed value via the generating unit 303. Then, the process proceeds to step S807.

[0068] (Step S807) The generating device 300 outputs the selection prompt 900 in step S803 and the selection seed value in step S806 to the language model 100 via the obtaining unit 304, and obtains an answer 920 generated by the language model 100. In this case, when generating the answer 920, the language model 100 randomly selects the next word according to a conditional probability distribution based on the selection seed value, specifically, for example, by Top_K Sampling, Top_p Sampling, or random control of temperature. Then, the process proceeds to step S805.

[0069] (Step S808) The generation device 300 causes the calculation unit 305 to calculate the appearance probability for each classification class of the representative query candidate 114 in the selection prompt 900. In this example, the classification class is the difficulty level (beginner, intermediate, advanced) indicated in the second template 902. Then, the process proceeds to step S809.

[0070] When step S808 is executed, the predetermined number of correct answer candidates 921 in step S804 have been acquired for the representative query candidates 114 of the selection prompt 900 (step S807). For example, if the predetermined number in step S804 is 10, there are 10 correct answer candidates 921, and therefore the generating device 300, using the calculation unit 305, calculates the appearance probability P(x) for each difficulty level (beginner, intermediate, advanced), for example, as follows: x indicates the difficulty level (beginner, intermediate, advanced).

[0071] P(Beginner)=4 / 10 P(Intermediate)=5 / 10 P (advanced)=1 / 10

[0072] That is, when the combination of the selection prompt 900 and the selection seed value was given to the language model 100 ten times, the correct answer A1 of the query Q1, which is the representative query candidate 114, was classified as beginner four times, intermediate five times, and advanced once out of the correct answer candidates 921 obtained in the answer 920.

[0073] (Step S809) The generation device 300 uses the calculation unit 305 to calculate the answer variance for the representative query candidate 114 of the selection prompt 900. Calculating the answer variance is a process of quantifying the variance of the representative query candidate 114 of the selection prompt 900 using entropy H. Entropy H is calculated using the following formula (1). Then, the process returns to step S802. By returning to step S802, the occurrence probability P(x) and entropy H are calculated for each of the representative query candidates 114, queries Q2 and Q5.

[0074] The magnitude of the entropy H indicates the likelihood of predicting the correct answers 402 of other representative query candidates 114. In other words, the higher the entropy H of a representative query candidate 114, the more likely it is that the correct answers 402 of other representative query candidates 114 can be predicted by knowing the correct answers 402 of that representative query candidate 114.

[0075]

number

[0076] Note that while the calculation of answer variance when the task is document classification has been described above, when the task is sentence generation, the calculation unit 305 inputs the predetermined number of answers 920 acquired in step S807 into a deep language model such as BERT to calculate a feature vector. Then, the calculation unit 305 calculates the average value of the feature for each dimension of the feature vector of each answer 920. The calculation unit 305 calculates the square of the deviation from the calculated average value of the feature for each dimension. The calculation unit 305 calculates the average value of the squared deviations for each dimension. This average value becomes the variance for each dimension. Then, the calculation unit 305 calculates the answer variance by summing the variances for all dimensions.

[0077] (Step S810) The generation device 300 uses the selection unit 306 to select a new representative query 121 from among the representative query candidates 114 based on the answer variances (e.g., entropy H) of all the representative query candidates 114. Specifically, for example, the generation device 300 uses the selection unit 306 to sort the representative query candidates 114 in descending order of answer variance (e.g., entropy H), and selects the top n (n is a preset integer of 1 or greater) representative query candidates 114 as the representative queries 121.

[0078] For example, if n=2 and the entropy H of query Q1 is highest (H(Q1)), followed by the entropy H(Q2) of query Q2 and the entropy H(Q5) of query Q5, queries Q1 and Q2 are newly selected as representative queries 121, and answer 920 is displayed on a display, which is an example of output device 204. Then, the process returns to step S704.

[0079] In step S704, developer D refers to the predetermined number of answers 920 to the newly selected representative query 121 displayed on the display and inputs the correct answer 402. Specifically, for example, the generation device 300 receives input of the correct answer 402 for the newly selected representative query 121 in step S810 via the input unit 302, associates the newly selected representative query 121 with the received correct answer 402, and updates the dataset DB 310. For example, if the newly selected representative query 121 is query Q1, when "intermediate" is input as the correct answer A1 for query Q1, the correct answer 402 for query Q1 is updated from "null" to "intermediate." Then, the process proceeds to step S705.

[0080] In this way, according to Dataset Generation Example 2, the required specifications of Developer D are semi-automatically determined, a representative query 121 in which the boundaries between classes are ambiguous or difficult to distinguish is selected from the group of representative query candidates 114, and the correct answer 123 can be set based on the answer 920. Therefore, it is possible to automatically generate prompts with higher performance than existing technologies.

[0081] As described above, according to this embodiment, the generation device 300 generates a query that can infer a new representative query 121 and its correct answer 123 from the representative query candidate 114 by combining the representative query and its correct answer 402. In this way, it is possible to present a representative query that includes more boundary cases where it is difficult to uniquely determine an answer, and to obtain generalizability from a smaller data set.

[0082] This allows developer D to easily determine which class the correct answer 123 should be assigned to when the correct answer candidate falls into a boundary case. Furthermore, this determination makes it easier for the language model 100 to infer the correct answer of the representative query candidate 114.

[0083] In the above-described embodiment, for the sake of convenience, the correct answer 402 in the dataset DB 310 is set to "null" by default, but the correct answer 402 may be set. In this case, a list of queries 401 may be displayed for representative queries and representative query candidates, and developer D may select the query by performing an operation. This allows the correct answer 402 that has already been set to be modified.

[0084] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0085] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0086] Information such as programs, tables, files, etc. that realize each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0087] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0088] 100 language models 101 prompt 102 answers 114 Representative Query Candidates 121 Representative Queries 123 Correct Answer 201 processor 202 Storage Devices 300 generator 301 Extraction part 302 Input section 303 Generation part 304 Acquisition Department 305 Calculation Unit 306 Selection Section

Claims

1. A generation device that can access a database that stores a group of queries and a language model that, when a natural sentence that is text data in a natural language is input, interprets the natural sentence and probabilistically predicts a response sentence to the natural sentence, an extracting unit that extracts a first query and a second query from the group of queries; an input unit that receives an input of a correct answer to the first query extracted by the extraction unit and associates the correct answer with the first query; a generation unit that generates a query requesting the language model to present a specific second query and a correct answer candidate for the specific second query, the specific second query being capable of inferring a correct answer to another second query from the second query by referring to a combination of the first query and a correct answer to the first query and knowing the correct answer; an acquisition unit that acquires, from the language model as a result of the inquiry generated by the generation unit being output to the language model, an answer including the specific second query and the correct answer candidate for the specific second query; A generating device comprising:

2. The generating device of claim 1 , the input unit accepts an input of a correct answer to the specific second query included in the answer acquired by the acquisition unit, and associates the correct answer with the specific second query; A generating device characterized by:

3. 3. The generating device of claim 2, the generation unit further refers to a combination of the specific second query and a correct answer to the specific second query in the language model, and generates the inquiry for the remaining second queries excluding the specific second query for which the input of the correct answer has been accepted from the second queries. A generating device characterized by:

4. The generating device of claim 1 , the generation unit generates the inquiry for each of the second queries; the acquisition unit randomly controls the inquiry to acquire a plurality of answers to the second query for each inquiry; a calculation unit that calculates, for each of the inquiries, an appearance probability of the classification class of the second query included in the answer, and calculates, for each of the second queries, an answer variance of the second query based on the appearance probability; a selection unit that selects the specific second query from the second queries based on the response variance calculated for each second query by the calculation unit; and A generating device comprising:

5. 5. The generating device of claim 4, the input unit accepts an input of a correct answer to the specific second query based on the answer including the specific second query selected by the selection unit, and associates the correct answer with the specific second query. A generating device characterized by:

6. 6. The generating device of claim 5, the generation unit references a combination of the specific second query and the correct answer for the specific second query, which is further associated with the language model by the input unit, and generates the inquiry for each of the remaining second queries, excluding the specific second query for which the input of the correct answer has been accepted. A generating device characterized by:

7. A generation method executed by a generation device having a processor that executes a program and a storage device that stores the program, and that can access a database that stores a group of queries, and a language model that, when a natural sentence that is text data in a natural language is input, interprets the natural sentence and probabilistically predicts a response sentence to the natural sentence, The processor: an extraction process of extracting a first query and a second query from the group of queries; an input process of accepting an input of a correct answer to the first query extracted by the extraction process and associating the correct answer with the first query; a generation process in which the language model generates a query requesting the language model to present a specific second query and a correct answer candidate for the specific second query, the specific second query being capable of predicting a correct answer to another second query from the second query by referring to a combination of the first query and a correct answer to the first query and knowing the correct answer; A generating method characterized by executing the following.

8. a processor of a generation device that can access a database that stores a group of queries and a language model that, when a natural sentence that is text data in a natural language is input, interprets the natural sentence and probabilistically predicts a response sentence to the natural sentence; an extraction process of extracting a first query and a second query from the group of queries; an input process of accepting an input of a correct answer to the first query extracted by the extraction process and associating the correct answer with the first query; a generation process in which the language model generates a query requesting the language model to present a specific second query and a correct answer candidate for the specific second query, the specific second query being capable of predicting a correct answer to another second query from the second query by referring to a combination of the first query and a correct answer to the first query and knowing the correct answer; A generating program characterized by executing the above.

Images