Data conversion device, data conversion method, and data conversion program

Abstract

To rationalize a form of a question sentence to an evaluation object.SOLUTION: A data conversion device having a processor for executing a program and a storage device for storing therein the program executes: input processing for inputting a plurality of choices and a first question sentence for taking one of the plurality of choices as a correct answer; conversion processing for converting the first question sentence to a second question sentence having a form different from that of the first question sentence, on the basis of the plurality of choices and the first question sentence inputted by the input processing; and output processing for outputting the second question sentence converted by the conversion processing and the plurality of choices.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a data conversion device, a data conversion method, and a data conversion program for converting data. [Background technology]

[0002] In recent years, there has been remarkable progress in language models, such as Generative Pretrained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT). These language models are also called foundational models and are used for various downstream tasks such as text generation, document classification, translation, image captioning, and grammar analysis.

[0003] A language model can be evaluated for downstream tasks as an integrated system that includes external knowledge utilization, pre- and post-processing, and additional supervised learning. However, building an integrated system is costly, and the quality of parts other than the language model has a significant impact on the performance of downstream tasks. For this reason, it is desirable to be able to evaluate a language model independently.

[0004] Currently, evaluation data and evaluation methods specifically designed for evaluating individual language models exist. In general domains, these evaluation data and evaluation methods can be used to evaluate individual language models. However, when creating a new language model for a specific domain, these evaluation data and evaluation methods alone are insufficient.

[0005] Furthermore, because these evaluation data and methods were created by humans, building evaluation data and methods for specific domains using similar procedures would be costly. Meanwhile, even within those specific domains, tests and practice questions for humans may exist. However, while language models are generally constructed statistically through fill-in-the-blank and completion questions in text, questions for humans may also include multiple-choice questions. For this reason, in order to evaluate a language model through questions for humans, it is necessary to incorporate external knowledge, pre- and post-processing, and additional supervised learning into the language model, as mentioned above. Therefore, it is difficult to evaluate a language model on its own.

[0006] Non-Patent Document 1 below discloses a method for evaluating a language model. Non-Patent Document 2 below discloses a method for evaluating a language model based on backward completion. Non-Patent Document 3 below discloses a machine learning system that formulates a task as predicting transition labels between text fragments that map the text fragments to a tree. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. 2018. Language Models are Unsupervised Multitask Learners. OpenAI Blog. [Non-patent document 2] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics. [Non-patent document 3] Yuta Koreeda and Christopher Manning. 2021. Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 144-154, Punta Cana, Dominican Republic. Association for Computational Linguistics. Summary of the Invention [Problem to be solved by the invention]

[0008] The evaluation method based on multiple-choice questions disclosed in Non-Patent Document 1 is known to depend on the method of formatting the question sentence (method of creating the prompt), and currently, various performances have been reported even for the same language model. The evaluation method disclosed in Non-Patent Document 2 is an evaluation method that was constructed manually over a long period of time, and the device or method for constructing the evaluation method is not disclosed.

[0009] The present invention aims to optimize the format of questions for evaluation targets. [Means for solving the problem]

[0010] A data conversion device according to one aspect of the invention disclosed in this application is a data conversion device having a processor that executes a program and a storage device that stores the program, wherein the processor executes an input process for inputting a plurality of options and a first question sentence in which one of the plurality of options is the correct answer; a conversion process for converting the first question sentence into a second question sentence in a format different from that of the first question sentence based on the plurality of options and the first question sentence input by the input process; and an output process for outputting the second question sentence converted by the conversion process and the plurality of options. [Effects of the Invention]

[0011] According to the exemplary embodiment of the present invention, it is possible to optimize the format of questions for evaluation targets. Problems, configurations, and effects other than those described above will become clear from the following description of the examples. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an example of a hardware configuration of a data conversion device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram of an example of a functional configuration of the data conversion device according to the first embodiment. [Figure 3] FIG. 3 is a flowchart of an example of an input question structuring process performed by the input question structuring unit according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram showing an example of input question data. [Figure 5] FIG. 5 is an explanatory diagram showing an example of structured input question data. [Figure 6] FIG. 6 is a flowchart of an example of a question format conversion process performed by the question format conversion unit according to the first embodiment. [Figure 7] FIG. 7 is an explanatory diagram showing an example of a prompt template. [Figure 8] FIG. 8 is an explanatory diagram showing an example of a prompt. [Figure 9] FIG. 9 is a flowchart showing an example of a detailed processing procedure for generating text (step S602). [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of a token dictionary. [Figure 11] FIG. 11 is an explanatory diagram showing an example of generated text. [Figure 12] FIG. 12 is an explanatory diagram illustrating an example of backward completion question data. [Figure 13] FIG. 13 is a block diagram of an example of a functional configuration of a data conversion device according to the second embodiment. [Figure 14] FIG. 14 is a flowchart illustrating an example of a detailed processing procedure of the generation score calculation process performed by the generation score calculation unit. [Figure 15] FIG. 15 is a flowchart illustrating a detailed example of a process procedure of generating score calculation (step S1404) according to the first embodiment. [Figure 16] FIG. 16 is an explanatory diagram illustrating an example of a backward completion question prediction result. [Figure 17] FIG. 17 is a flowchart of an example of a question format conversion process performed by the question format conversion unit according to the third embodiment. [Figure 18] FIG. 18 is an explanatory diagram illustrating an example of the first REST query. [Figure 19] FIG. 19 is an explanatory diagram illustrating an example of the first response. [Figure 20] FIG. 20 is a flowchart illustrating a detailed example of a process procedure of generating score calculation (step S1404) according to the second embodiment. [Figure 21] FIG. 21 is an explanatory diagram illustrating an example of the second REST query. [Figure 22] FIG. 22 is an explanatory diagram illustrating an example of the second response. [Figure 23] FIG. 23 is a flowchart of an example of a question format conversion process performed by the question format conversion unit according to the fourth embodiment. [Figure 24]FIG. 24 is an explanatory diagram showing an example of syntax analysis. [Figure 25] FIG. 25 is an explanatory diagram showing an example of an input question information summary. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, examples of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be modified within the scope of the idea or intent of the present invention. In the configuration of the invention described below, the same or similar configurations or functions are designated by the same reference numerals, and redundant explanations will be omitted.

[0014] The designations "first," "second," "third," etc. in this specification are used to identify components and do not necessarily limit the number or order. The position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings, etc. [Example]

[0015] A data conversion device according to a first embodiment will be described with reference to the drawings. The data conversion device is a computer that converts existing question data used in existing human evaluation tests into complementary question data suitable for evaluating a language model. In the first embodiment, a process of converting a collection of multiple-choice English questions into complementary questions suitable for evaluating a language model will be described as an example. A complementary question is an explanatory sentence with a missing part in a complementary format that completes the sentence by selecting the correct answer from multiple options.

[0016] For example, a question “Which of the following defines the guarantee of the levels of availability? and its multiple options a.NDA b.SLI c. SLA d.SLO In this case, the completion question is a multiple choice question. a.NDA b.SLI c. SLA d.SLO A missing explanation in the form of completion, where the sentence is completed by filling in the correct answer. “The guarantee of the levels of availability is defined by” This completion question is called a backward completion question because the correct answer is missing at the end of the sentence.

[0017] The input questions are not limited to English or natural languages, but can take any form that is understandable to humans, including natural languages ​​from different language families such as Japanese and Chinese, programming and markup languages ​​such as Java (registered trademark), C, Mathematica, and HTML, domain-specific languages ​​(DSLs) created by users, diagrams, tables, audio, numeric strings, mathematical formulas, and combinations of these.

[0018] <Figure 1: Example of hardware configuration for data conversion device> FIG. 1 is a block diagram illustrating a hardware configuration example of a data conversion device according to a first embodiment. The data conversion device 100 includes one or more computers. The data conversion device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, the storage device 102, the input device 103, the output device 104, and the communication IF 105 are connected via a bus 106. The processor 101 controls the data conversion device 100. The storage device 102 serves as a working area for the processor 101. The storage device 102 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 102 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 103 inputs data. The input device 103 includes, for example, a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 104 outputs data. The output device 104 includes, for example, a display, a printer, and a speaker. The communication IF 105 connects to a network and transmits and receives data.

[0019] <Figure 2: Example of functional configuration of data conversion device 100> 2 is a block diagram illustrating an example of a functional configuration of the data conversion device 100 according to the first embodiment. The data conversion device 100 includes an input question structuring unit 201, a question format conversion unit 202, and a complementary question output unit 203. Specifically, the input question structuring unit 201, the question format conversion unit 202, and the complementary question output unit 203 are functions that are realized by, for example, causing the processor 101 to execute a program stored in the storage device 102 illustrated in FIG.

[0020] The input question structuring unit 201 receives input of question data including one or more questions and divides each piece of question data into a question body and options. The question format conversion unit 202 receives each structured piece of question data and creates complementary question data. The complementary question output unit 203 outputs the complementary question data.

[0021] <Figures 3 to 5: Input question structuring process> Fig. 3 is a flowchart illustrating an example of an input question structuring process procedure by the input question structuring unit 201 according to the first embodiment. Fig. 4 is an explanatory diagram illustrating an example of input question data. Fig. 5 is an explanatory diagram illustrating an example of structured input question data.

[0022] (Step S301) The input question structuring unit 201 accepts input of input question data 400. The input question data 400 includes one or more multiple-choice questions. The input question data 400 may be composed of multiple documents, such as a document including question data 410 and a document including answer data 411, or the question data 410 and the answer data 411 may be included in a single document. The input question data 400 may include elements other than questions and answers, such as explanatory texts or instructions, or may consist mostly of elements other than questions and answers, as seen in example sentences in textbooks.

[0023] In FIG. 4, the question data 410 is "They are part of behavioral diagrams that show how objects interact with each other with focus on the messages passed between the objects where each node represents a message object. a. Class diagram b. Communication diagram c. Sequence diagram d. Use case diagram The question is in the form of a declarative sentence, with the correct answer expressed at the beginning of the sentence using the demonstrative pronoun "They."

[0024] Those skilled in the art will readily understand that the medium on which the input question data 400 is expressed may be paper, dictation, encrypted data, etc., and that each may be digitized by means of an Optical Character Recognition / Reader (OCR), voice recognition, decryption, etc. Furthermore, those skilled in the art will readily understand that the input question data 400 expressed on an electronic medium may be plain text as exemplified in the first embodiment, or may be a table or column-formatted document, a Portable Document Format (PDF) document, an Extensible Markup Language (XML) document, or the like, as long as it can be processed by existing technology.

[0025] Furthermore, those skilled in the art will easily understand that even if the input question data 400 includes diagrams or images, it is possible to process them without deviating from the spirit of Example 1 by invoking Layout Language Model (LayoutLM) or GPT-4V, respectively.

[0026] (Step S302) Input question structuring unit 201 extracts one or more structured input question data 500 from input question data 400. Structured input question data 500 includes question identifier 501, question body 502, options 503, and correct answer 504. Question identifier 501 is identification information that uniquely identifies question body 502. Question body 502 is a character string indicating a question. Question body 502 may be a question asking which of multiple options 503 is the correct answer, or may be a declarative sentence indicating one of multiple options 503. In the example of FIG. 5, it is a declarative sentence. Options 503 are multiple answer candidates to question body 502. Correct answer 504 is an answer candidate selected from options 503.

[0027] The input question structuring unit 201 scans each line of the question data 410, and extracts the question body 502 by taking the line that satisfies the first regular expression / ^([1-9][0-9]*)\. / in Perl notation as the start of the question body 502 and taking the line immediately before the line that satisfies the regular expression / ^[az]\. / as the end of the question body 502.

[0028] Furthermore, the input question structuring unit 201 stores the character string stored in the first match group of the first regular expression (the character string that matches [1-9][0-9]* enclosed in parentheses) as the question identifier 501.

[0029] In addition, the input question structuring unit 201 stores in options 503 the character string stored in the first match group (the character string that matches .* enclosed in parentheses) of the line that matches the Perl regular expression / ^[az]\. among the lines existing from one question body 502 to the next question body 502.

[0030] The input question structuring unit 201 also performs a similar scan on the answer data 411, and for each line that matches the Perl regular expression / ^([1-9][0-9]*)\.[az]) / , it stores the degree of the option 503 that matches the string stored in the second match group (the string that matches [az] enclosed in parentheses) in the correct answer 504 of the structured input question data 500 where the string stored in the first match group (the string that matches [1-9][0-9]* enclosed in parentheses) matches the question identifier 501.

[0031] The processing of step S302 may be realized by extraction processing using different regular expressions depending on the format of the input question data 400, rule-based extraction processing, machine learning based extraction processing, or a combination of these as in Non-Patent Document 3.

[0032] Furthermore, in step S302, if structured input question data 500 is generated in which one or more of question identifier 501, question body 502, option(s) 503, and correct answer 504 are missing, inconsistent, or duplicated due to incompleteness of input question data 400 or incompleteness of the extraction process, the processing in step S302 may include processing to remove structured input question data 500 or request the user to correct structured input question data 500.

[0033] (Step S303) The input question structuring unit 201 starts loop processing of the structured input question data 500 (step S303). Specifically, for example, the input question structuring unit 201 selects an unselected structured input question data 500.

[0034] (Step S304) The input question structuring unit 201 performs preprocessing on the selected structured input question data 500. In the preprocessing (step S304), the input question structuring unit 201 deletes spaces at the beginning and end of sentences from the question body 502 and normalizes the spaces. Note that the input question structuring unit 201 may also normalize Unicode and correct OCR conversion errors.

[0035] (Step S305) The input question structuring unit 201 determines whether there is unselected structured input question data 500. If it is determined that there is unselected structured input question data 500, the process returns to step S302. If it is determined that there is no unselected structured input question data 500, the input question structuring unit 201 ends the input question structuring process for the structured input question data 500.

[0036] <Figures 6 to 8: Question format conversion process> Fig. 6 is a flowchart showing an example of a question format conversion process procedure by the question format conversion unit 202 according to the first embodiment. Fig. 7 is an explanatory diagram showing an example of a prompt template. Fig. 8 is an explanatory diagram showing an example of a prompt. The question format conversion unit 202 executes the processes of steps S601 to S603 for each structured input question data 500.

[0037] (Step S601) The question format conversion unit 202 combines the question body 502 and the prompt template 700 to create a prompt 800, which is an instruction statement for generating complementary question data.

[0038] As shown in FIG. 7, a prompt template 700 includes a question body placeholder 701 , a choice placeholder 702 , an instruction prompt 703 , a conversion specification explanation prompt 704 , and an example prompt 705 .

[0039] The question body placeholder 701 is an area where the question body 502 is set. The option placeholder 702 is an area where the option 503 is set. The instruction prompt 703 is a character string indicating a sentence that explains the conversion instruction in natural language. The conversion specification explanation prompt 704 is a character string indicating a sentence that explains the detailed specifications of the conversion in natural language. In particular, the conversion specification explanation prompt 704 specifies that the generated completion-format question data should be a declarative sentence that ends with the option 503.

[0040] The example prompt 705 is an example of an input / output format. In the example prompt 705, the question body 502 is a question asking which of the multiple choices 503 is the correct answer. “Which of the following defines the guarantee of the levels of availability?” and its options “a.NDA b.SLI c. SLA d.SLO” is shown as an example of the input format, and a partially missing explanation (complementary question) is completed by filling in the correct answer. “The guarantee of the levels of availability is defined by” and its options “a.NDA b.SLI c. SLA d.SLO” is shown as an example of the output format. Therefore, the user can refer to the example prompt 705 and input the question body 502 into the question body placeholder 701 and the options 503 into the option placeholders 702 according to the input format.

[0041] The question format conversion unit 202 stores the question body 502 in the question body placeholder 701, stores the options 503 in the option placeholder 702, and creates a prompt 800. Note that the prompt template 700 may include any content that suits the field, language, and other characteristics of the question body 502, as long as it specifies that the generated completion-style question data should end with the option 503.

[0042] The prompt template 700 does not need to have the above configuration, and may include any content that matches the field, language, and other characteristics of the question body 502, as long as it is intended to generate complementary question data by utilizing the question body 502. Furthermore, such information may be separately input from the user through an interactive question, a setting file, or the like.

[0043] (Step S602) The question-form conversion unit 202 generates text based on the prompt 800. The text generation (step S602) will be described with reference to FIGS.

[0044] <Figure 9 Text Generation (Step S602)> FIG. 9 is a flowchart showing an example of a detailed processing procedure for generating text (step S602).

[0045] (Step S901) The question format conversion unit 202 divides the prompt 800 into a prompt token string consisting of one or more tokens. A "token" generally means the smallest unit that constitutes a programming language, such as a variable name, reserved word, or operator, but in the first embodiment, division may be performed using a unit other than a "token" in a programming language, such as byte pair encoding.

[0046] (Step S902) The question format conversion unit 202 uses a token dictionary to convert the prompt token sequence into a prompt token index sequence {t0, t1, . . . , t(P-1)} consisting of P integers.

[0047] <Figure 10 Token Dictionary> 10 is an explanatory diagram showing an example of a token dictionary. The token dictionary 1000 is information that associates a token 1001 with an index 1002. The index 1002 is identification information that uniquely identifies the token 1001. The token dictionary 1000 has V entries (V is an integer equal to or greater than 1).

[0048] 9, the question format conversion unit 202 is realized by matching each token constituting the prompt token string with a token 1001 in the token dictionary 1000 and obtaining an index 1002 that matches the matching token 1001. If there is no token 1001 that matches the token, the token is associated with a predetermined index 1002.

[0049] (Step S903) The question format conversion unit 202 starts a loop process of the variable i, setting the variable i to 0 as its initial value.

[0050] (Step S904) The question format conversion unit 202 predicts the (i+1)th generated token index using a probability distribution model for the token index sequence, such as GPT. The question format conversion unit 202 inputs the prompt token index sequence and the generated token index sequences up to the (i-1)th token into the probability distribution model, and calculates the probability distribution p(xi|θ,t0,t1,...,t(P-1),x0,x1,...,x(i-1)) of the i-th generated token index xi, where θ is a setting value (parameter) of the probability distribution model.

[0051] The question format conversion unit 202 acquires the generated token index with the maximum probability value of the probability distribution p(xi|θ,t0,t1,...,t(P-1),x0,x1,...,x(i-1)) as the i-th generated token index.

[0052] argmaxxi p(xi|t0,t1,…,t(P-1),x0,x1,…,x(i-1))

[0053] Instead of selecting the token index with the maximum probability value, the question format conversion unit 202 may perform sampling based on the probability distribution p(xi|θ,t0,t1,...,t(P-1),x0,x1,...,x(i-1)), or may select the token index using beam search, etc.

[0054] In addition, the probability distribution model may be realized using a model based on the Transformer architecture such as T5 or Llama-2, a model based on recursive calculation such as Long Short Term Memory (LSTM) or Receptance Weighted Key Value (RWKV), or a model based on distributed calculation such as GShard.

[0055] The above-mentioned probability distribution model may be a model that has been trained by self-supervised learning of word probability distributions using various documents, or a model that has been trained by supervised learning using a corpus consisting of pairs of manually or mechanically created prompts and backward-completion question data. The requirement of a probability distribution does not need to be met as long as an input token index can be generated based on the input token index sequence.

[0056] (Step S905) The question format conversion unit 202 determines whether the i-th generated token index is a token index indicating the end of the document, and whether the variable i is i=L-1 (L is a predetermined number of generated tokens L).

[0057] If it is determined that the condition is not met, i.e., if the i-th generated token index is not a token index indicating the end of the document and the variable i is not i=L-1, the question format conversion unit 202 increments the variable i and returns to step S904.

[0058] If it is determined that the condition is met, i.e., if the i-th generated token index is a token index indicating the end of the document, or if the variable i is i=L-1, the question format conversion unit 202 ends the loop processing of the variable i and proceeds to step S906.

[0059] (Step S906) The question-form conversion unit 202 converts the generated token index sequence {x0, x1, ..., x(L-1)} into generated text. Specifically, for example, the question-form conversion unit 202 matches each token index constituting the generated token index sequence {x0, x1, ..., x(L-1)} with an index 1002 in the token dictionary 1000.

[0060] The question-form conversion unit 202 reads the token 1001 associated with the matched index 1002 from the token dictionary 1000 to obtain a generated token string. The question-form conversion unit 202 generates a generated text by combining the obtained generated token string, and proceeds to step S603.

[0061] <Figure 11 Generated text> 11 is an explanatory diagram showing an example of generated text. Generated text 1100 includes a generated question body 1101 and generated options 1102. Generated text 1100 is stored in prompt 800.

[0062] Instead of data conversion device 100 generating text (step S602), data conversion device 100 may send the prompt created in step S601 to a language model implemented in an external computer. In this case, the language model implemented in the external computer executes the process shown in Fig. 9 to generate generated text 1100. Data conversion device 100 then receives generated text 1100 generated by the language model implemented in the external computer.

[0063] (Step S603) Returning to FIG. 6, the question format conversion unit 202 creates backward complement question data from the generated text 1100.

[0064] <Figure 12 Backward Completion Question Data> 12 is an explanatory diagram showing an example of backward completion question data. The backward completion question data 1200 includes a question identifier 1201, a generation context 1202, a generation option 1203, and a correct answer 1204.

[0065] The value of question identifier 501 in structured input question data 500 is set in question identifier 1201. The generated question text 1101 is set in generation context 1202. The generated option 1102 is set in generation option 1203. The value of correct answer 504 is set in correct answer 1204.

[0066] The backward completion question data 1200 is output from the complementary question output unit 203. Specifically, for example, the complementary question output unit 203 displays the backward completion question data 1200 on a display, which is an example of the output device 104, prints it out on a printer, or transmits it to the outside via the communication IF 105.

[0067] According to the first embodiment, the data conversion device 100 constructs data suitable for evaluating a language model from an existing human evaluation test, thereby enabling standalone evaluation of a language model without the need to manually create evaluation data dedicated to the language model. [Example]

[0068] Next, Example 2 will be described. Example 2 will be described by taking as an example a process in Example 1 in which a collection of multiple-choice English questions is input and the collection of multiple-choice questions is converted into backward-completion questions suitable for evaluating a language model. Example 2 will be described mainly with respect to differences from Example 1, and therefore a description of commonalities with Example 1 will be omitted.

[0069] <Fig. 13: Example of functional configuration of data conversion device 100> 13 is a block diagram illustrating an example of a functional configuration of a data conversion device 100 according to a second embodiment. The data conversion device 100 includes an input question structuring unit 201, a question format conversion unit 202, a complementary question output unit 203, a generated score calculation unit 1301, and an index calculation unit 1302. Specifically, the generated score calculation unit 1301 and the index calculation unit 1302 are functions realized by, for example, causing the processor 101 to execute a program stored in the storage device 102 illustrated in FIG.

[0070] The generation score calculation unit 1301 receives the backward completion question data 1200 and calculates a generation score indicating the likelihood that the backward completion question data 1200 will be generated for each backward completion question data 1200. Specifically, for example, the generation score calculation unit 1301 inputs the backward completion question data 1200 and calculates the likelihood for each option 903.

[0071] The index calculation unit 1302 calculates an evaluation index for one or more pieces of input question data 400 as a whole based on the backward completion question data 1200 associated with each piece of input question data 400 and the generated score for the backward completion question data 1200 .

[0072] <Figure 14 Generation score calculation process> FIG. 14 is a flowchart showing a detailed example of a procedure for generating score calculation processing by the generating score calculation unit 1301.

[0073] (Step S1401) The generation score calculation unit 1301 receives the backward completion question data 1200 from the completion question output unit 112 .

[0074] (Step S1402) The generation score calculation unit 1301 starts loop processing of the backward completion question data 1200. Specifically, for example, the generation score calculation unit 1301 selects unselected backward completion question data 1200. The selected backward completion question data 1200 is referred to as selected backward completion question data 1200.

[0075] (Step S1403) The generation score calculation unit 1301 starts loop processing of the generation option 1203 for the selected backward completion question data 1200 (step S1403). Specifically, for example, the generation score calculation unit 1301 selects an unselected generation option 1203. The selected generation option 1203 is referred to as a selected generation option 1203.

[0076] (Step S1404) The generation score calculation unit 1301 receives the generation context 1202 and the selected generation option 1203 of the selected backward completion question data 1200 as input, and calculates the likelihood of the selected generation option 1203 .

[0077] <Figures 15 and 16: Generation score calculation (step S1404)> 15 is a flowchart illustrating a detailed example of a processing procedure of generating score calculation (step S1404) according to the first embodiment. FIG. 16 is an explanatory diagram illustrating an example of a backward completion question prediction result. A backward completion question prediction result 1600 includes a question identifier 1601, an option score 1603, and a correct answer 1604.

[0078] The value of the question identifier 501 is stored in the question identifier 1601. The option score 1603 is calculated for each generated option (0 to 3). The correct answer 1604 stores the generated option for which the option score 1603 is the maximum value.

[0079] (Step S1501) The generation score calculation unit 1301 divides the generation context 1202 and the selected generation option 1203 of the selected backward completion question data 1200 into a token string of the generation context 1202, each of which is composed of one or more tokens, and a token string of the generation option 1203. This process may divide the token strings based on a criterion different from that of step S901.

[0080] (Step S1502) The generation score calculation unit 1301 refers to the token dictionary 1000 and converts the token sequence of the generation context 1202 and the token sequence of the generation option 1203 into a token index sequence {t0, t1, ..., t(P-1)} of the generation context 1202 consisting of P integers and a token index sequence {x0, x1, ..., x(Q-1)} of the generation option 1203 consisting of Q integers, respectively. This conversion is the same as in step S902, but a token dictionary 1000 different from that in step S902 may be used.

[0081] (Step S1503) The generated score calculation unit 1301 starts loop processing of the variable i, setting the variable i to i=0 as the initial value.

[0082] (Step S1504) The generation score calculation unit 1301 uses a probability distribution model for the token index sequence such as GPT-3 to input the token index sequence of the generation context 1202 and the token index sequences of the first (i-1)th generation options 1203, and calculates the generation score lnp(x i |θ,t0,t1,…,t P ,x0,x1,…,x i-1 ) is calculated.

[0083] where θ is a setting value (parameter) of the probability distribution model. The generated score to be calculated does not have to be the natural logarithm of the generated probability value; it can be the logarithm of an arbitrary base for the generated probability value, the probability value, the reciprocal of the probability value, or a score that no longer satisfies the requirements of the probability distribution after applying an arbitrary correction to the probability value.

[0084] (Step S1505) The generation score calculation unit 1301 determines whether the i-th generated token index is a value obtained by subtracting 1 from the length Q of the generated choice token index sequence, where i is the variable of the document.

[0085] If it is determined that the variable i is not the value obtained by subtracting 1 from the length Q of the token index sequence of the generation option 1203, the generation score calculation unit 1301 increments the variable i, returns to step S1504, and executes the same processing. If it is determined that the variable i is the value obtained by subtracting 1 from the length Q of the token index sequence of the generation option 1203, the generation score calculation unit 1301 ends the loop processing of the variable i, and proceeds to step S1506.

[0086] (Step S1506) The generation score calculation unit 1301 calculates the sum of the generation scores calculated in step S1504, and stores the sum as the option score 1603 corresponding to the selected generation option 1203. Specifically, the value y i,k is calculated by the following formula (1).

[0087]

Number

[0088] However, j (0 ≦ j < N) is the index of the backward-completion question data 1200, and k is the index of the generated option 1203. Note that, instead of the sum of the generation scores, an arbitrarily corrected score such as a value normalized by the likelihood of how easily the generated option 1203 itself is generated, as shown in the following formula (2), may be utilized.

[0089]

Number

[0090] As a result, the generation score calculation (step S1404) is completed, and the process proceeds to step S1405.

[0091] Note that instead of the data conversion device 100 executing the generation score calculation (step S1404), the data conversion device 100 may transmit the selected backward-completion question data 1200 to a language model implemented on an external computer. In this case, the language model implemented on the external computer executes the process shown in FIG. 14 and calculates the sum of the generation scores for the selected backward-completion question data 1200. Then, the data conversion device 100 receives the sum of the generation scores calculated by the language model implemented on the external computer (the same applies to FIG. 20 described later).

[0092] (Step S1405) Returning to FIG. 14, the generation score calculation unit 1301 determines whether there is an unselected generated option 1203 in the selected backward-completion question data 1200 (step S1405).

[0093] If it is determined that there is at least one unselected generated option 1203, the generation score calculation unit 1301 returns to step S1404 and executes the same process.

[0094] If it is determined that there is no unselected generation option 1203, the generation score calculation unit 1301 ends the loop processing of the generation option 1203 and proceeds to step S1406.

[0095] (Step S1406) The generation score calculation unit 1301 determines whether or not there is any unselected backward completion question data 1200. If it is determined that there is at least one unselected backward completion question data 1200, the generation score calculation unit 1301 returns to step S1403 and executes the same processing. If it is determined that there is no unselected backward completion question data 1200, the generation score calculation unit 1301 ends the loop processing of the backward completion question data 1200, and ends the processing of the generation score calculation unit 1301.

[0096] Then, the index calculation unit 1302 calculates the performance of the probability distribution model for the entire one or more input question data 400 based on the backward completion question data 1200 associated with each input question data 400 and the generated score for the backward completion question data 1200.

[0097] The accuracy of the probability distribution model is calculated by the choice score 1603(y j,k ), Correct answer 1604(z j ), is calculated using the number N of backward completion query data 1200 by the following formula (3).

[0098]

number

[0099] However, the 1[ ] on the right side of the above formula (3) is a function that becomes "1" when the condition of the argument is met and becomes "0" when it is not met.

[0100] In addition, the mean average precision of the probability distribution model is j,k ), correct answer 904(z j ), is calculated using the number N of backward completion query data 1200 by the following formula (4).

[0101]

number

[0102] However, rank[ ] on the right side of the above formula (4) is a function that represents the degree of element i when the set of arguments is sorted in descending order.

[0103] The index calculation unit 1302 may calculate precision, recall, ROC-AUC, etc. instead of or in addition to the accuracy of the above formula (3) or the mean average precision of the above formula (4). Furthermore, although the second embodiment shows a case where there is one correct answer, it will be easily understood by those skilled in the art that a similar procedure can also be used to handle multiple-choice questions where there are zero or more correct answers.

[0104] According to the second embodiment, the data conversion device 100 constructs data suitable for evaluating a language model from an existing human evaluation test and evaluates the language model, thereby enabling standalone evaluation of the language model without the need to manually create evaluation data dedicated to the language model. [Example]

[0105] In Example 3, as in Example 2, an example of a process of converting an English multiple-choice question set into backward-completion questions suitable for evaluating a language model will be described. Example 3 describes an example in which the data conversion device 100 generates text and calculates a generation score using an external calculation service, instead of generating text and calculating a generation score using the question format conversion unit 202 and the generation score calculation unit 1301, respectively. Note that Example 3 will mainly describe differences from Examples 1 and 2, and therefore will not describe commonalities with Examples 1 and 2.

[0106] <Figures 17 to 19 Question format conversion process> Fig. 17 is a flowchart illustrating an example of a question format conversion process procedure performed by the question format conversion unit 202 according to the third embodiment. Fig. 18 is an explanatory diagram illustrating an example of a first REST query. Fig. 19 is an explanatory diagram illustrating an example of a first response.

[0107] (Step S1701) The question format conversion unit 202 creates a first REST query 1800 based on the prompt 800 created in step S601. The question format conversion unit 202 performs JSON escaping on the prompt 800 and inserts the escaped prompt 800 into a query prompt 1802. The first REST query 1800 includes an external calculation service endpoint 1801 that generates text and a setting value 1803 for text generation.

[0108] External computing services that can be used include OpenAI API and Anthropic Claude, but other services or compatible services provided by the developer on their intranet are also acceptable. Communication methods can also be different, such as Remote Procedure Call (RPC) or GraphQL.

[0109] (Step S1702) The question format conversion unit 202 executes the first REST query 1800. Specifically, for example, the question format conversion unit 202 transmits the first REST query 1800 to an external calculation service and obtains a first response 1900 from the external calculation service.

[0110] (Step S1703) The question-form conversion unit 202 extracts generated text 1901 from the first response 1900. Then, the process proceeds to step S603.

[0111] <FIGS. 20 to 22 Generation score calculation (step S1404)> Fig. 20 is a flowchart illustrating a detailed example of a process procedure of generating a score (step S1404) according to Example 2. Fig. 21 is an explanatory diagram illustrating an example of a second REST query. Fig. 22 is an explanatory diagram illustrating an example of a second response.

[0112] (Step S2001) The generation score calculation unit 1301 creates a second REST query 2100 based on the selected backward completion question data 1200. The question format conversion unit 202 combines the generation context 1202 and the selected generation option 1203 of the selected backward completion question data 1200 with a space, performs JSON escaping processing, and inserts the combined data, which is obtained by combining the generation context 1202 and the selected generation option 1203 of the selected backward completion question data 1200 with a space and performing JSON escaping processing, into the query prompt 2102.

[0113] The second REST query 2100 includes an external calculation service endpoint 2101 that generates text. To obtain the score, the generation score calculation unit 1301 sets max_tokens to "1," logprobs to "0," and echo to "true" in the text generation setting values ​​2103 in the case of the OpenAI API, for example. As with the question format conversion unit 202, various external calculation services and communication means may be used, and external calculation services and communication means different from those of the question format conversion unit 202 may be used.

[0114] (Step S2002) The generation score calculation unit 1301 executes the second REST query 2100. Specifically, for example, the generation score calculation unit 1301 transmits the second REST query 2100 to an external calculation service and obtains a second response 2200 from the external calculation service.

[0115] (Step S2003) The generation score calculation unit 1301 calculates the generation score of the selected generation option 1203 from the second response 2200. To extract the generation score corresponding to the selected generation option 1203, the generation score calculation unit 1301 calculates the degree of the token offset 2201 that is equal to or greater than the number of characters in the generation context 1202 and that is obtained by excluding, from the end, the item specified by max_tokens in the text generation setting value 2103.

[0116] The generation score calculation unit 1301 extracts a numerical value corresponding to this degree from the generation score list 2202 as a generation option generation score list 2203, and outputs the sum of the generation option generation score list 2203 as a generation score. After that, the process proceeds to step S1505.

[0117] According to the third embodiment, the data conversion device 100 can perform standalone evaluation of a language model without performing internal processing that requires high computational costs. In addition, the data conversion device 100 can evaluate a language model that is implemented as an external computer. [Example]

[0118] In Example 4, as in Example 1, an example will be described in which a collection of multiple-choice English questions is input and the collection of questions is converted into backward-completion questions suitable for evaluating a language model. In Example 4, an example will be described in which, when generating text in the data conversion device 100 in the question format conversion unit 202, conversion is performed based on the syntax tree and rules of the evaluation data instead of directly generating text using a probability distribution model. Note that in Example 4, differences from Examples 1 to 3 will be mainly described, and therefore descriptions of common parts with Examples 1 to 3 will be omitted.

[0119] <Figures 23 to 25 Question format conversion process> Fig. 23 is a flowchart illustrating an example of a question format conversion process procedure by the question format conversion unit 202 according to the fourth embodiment. Fig. 24 is an explanatory diagram illustrating an example of syntax analysis. Fig. 24 shows an input sentence 2401 and a syntax tree 2402. Fig. 25 is an explanatory diagram illustrating an example of an input question sentence information summary.

[0120] The input question information summary 2500 includes a subject slot 2501, a predicate slot 2502, and a conjunction slot 2503. The subject slot 2501, the predicate slot 2502, and the conjunction slot 2503 are initially set to blank. FIG. 23 will now be described.

[0121] (Step S2301) The question format conversion unit 202 performs syntactic analysis on the question body 502. In Example 4, an example is shown in which phrase structure analysis in Penn Treebank format is performed on an English question sentence. In particular, it is assumed that Stanford CoreNLP is used as the analyzer, and the notation of part-of-speech tags below conforms to the notation of Stanford CoreNLP. Note that it will be easily understood by those skilled in the art that question format conversion is possible by following Example 4 even when syntactic analysis is performed on a grammar other than English, when dependency structure analysis is used instead of phrase structure analysis, when a grammar other than Penn Treebank is adopted, or when an analyzer other than Stanford CoreNLP is used.

[0122] For example, when a question body 502 such as the input sentence 2401 is input to a parser, the parser outputs a syntax tree 2402. In Example 4, a sentence having SBARQ (a direct question introduced by a wh word or wh phrase) as null constituents is assumed as a 5W1H question sentence typical of multiple-choice questions.

[0123] The main clause of an SBARQ consists of a wh-adverb phrase (WHADVP) or a wh-noun phrase (WHNP) and an SQ (a subordinate component of the SBARQ that excludes the wh-word or wh-phrase).

[0124] (Step S2302) The question format conversion unit 202 determines whether SBARQ=WHNP+SQ and WHNP=WDT (wh determiner)+NN (noun). For example, "What components stopped network?" satisfies this condition, and its syntax tree is (SBARQ (WHNP (WDT What) (NN component)) (SQ (VP (VBD stopped) (NP (NN network)))) (. ?)) This becomes:

[0125] If this condition is met, the process proceeds to step S2303, otherwise, the process proceeds to step S2304.

[0126] (Step S2303) When WHNP includes NN (noun), the question format conversion unit 202 replaces WHNP with WDT (wh determiner) + be verb + the + NN + that. At this time, the be verb is determined to be either "is" or "are" depending on whether NN (noun) is plural or not.

[0127] Whether NN (noun) is singular or plural is determined by referring to a dictionary (not shown) prepared in advance. In other words, "What components stopped network?" is converted to "What are the components that stopped network?"

[0128] (Step S2304) The question format conversion unit 202 sets the connection slot 2133 based on the head (main part) of the WHNP (wh noun phrase) (step S2304). If the head of the WHNP is "why", it is set to "because", if it is "how", it is set to "by", and otherwise it is left empty.

[0129] (Step S2305) The question format conversion unit 202 checks whether the syntax tree 2402 includes a modal auxiliary verb, that is, SQ=MD[+RB]+NP[+RB]+VP[+*] (Step S2305) where MD is a modal verb, RB is an adverb, NP is a noun phrase, VP is a verb phrase, [+X] indicates that X is a non-essential component, and [+*] indicates that any number of any components may be included.

[0130] An example of a syntax tree 2402 that meets this condition is: (SBARQ (WHADVP (WRB How)) (SQ (MD can) (NP (PRP I)) (VP (VB reset) (NP (NN network)))) (. ?)) There is.

[0131] If this condition is met, the process proceeds to step S2306, otherwise, the process proceeds to step S2307.

[0132] (Step S2306) The question format conversion unit 202 sets the subject slot 2131 and the predicate slot 2132 based on the value of SQ. Specifically, for example, the question format conversion unit 202 SQ=MD[+RB]+NP+VP[+*] For the above, NP (noun phrase) is stored in the subject slot 2131 and MD[+RB]+VP[+*] is stored in the predicate slot 2132 .

[0133] For example, for the above-mentioned "How can I reset network," "I" is stored in the subject slot 2131 and "can reset network" is stored in the predicate slot 2132.

[0134] (Step S2307) The question format conversion unit 202 sets the subject slot 2131 and the predicate slot 2132 based on the value of SQ. Specifically, for example, the question format conversion unit 202 SQ=VP[+RB]+NP[+*] For this, [+*] is stored in the subject slot 2131 and VP[+RB] is stored in the predicate slot 2132 .

[0135] For example, the syntax tree 2402 (SBARQ (WHNP (WP What)) (SQ (VP (VBP are) (NP (NP (DT the) (NNS components)) (SBAR (WHNP (WDT that)) (S (VP (VBD stopped) (NP (NN network))))))) (. ?)) For the sentence, "the components that stopped network" is stored in the subject slot 2131 and "are" is stored in the predicate slot 2132.

[0136] For "Why isn't network stopped?", "network" is stored in the subject slot 2131 and "isn't stopped" is stored in the predicate slot 2132.

[0137] Furthermore, for "How did network stop?", "network" is stored in the subject slot 2131 and "did stop" is stored in the predicate slot 2132.

[0138] (Step S2308) The question format conversion unit 202 creates a backward-complement format explanatory sentence from the subject slot 2131, the predicate slot 2132, and the conjunction slot 2133. The first character of the subject slot 2131 is converted to uppercase, and the subject slot 2131 is combined with the predicate slot 2132 and the conjunction slot 2133 with spaces to create the explanatory sentence.

[0139] For example, input sentence 2401 shown in FIG. “Which is the diagram that shows how objects interact with each other with focus on the messages passed between the objects where each node represents a message object?” In contrast, ”The diagram that shows how objects interact with each other with focus on the messages passed between the objects whereas each node represents a message object is” A backward-completion style description like this is created:

[0140] The question format conversion unit 202 described in the fourth embodiment does not have a function to convert all questions into backward-complementary explanations. For example, according to the above process, the question "What causes network delay?" is converted into "Network delay causes," which loses the original meaning of the question.

[0141] However, since human evaluation tests generally tend to be written in a consistent and controlled language for each evaluation test, providing exception handling for each evaluation test does not pose any major practical problems.

[0142] (Step S2309) The question format conversion unit 202 creates backward-complementation question data 1200 from the created backward-complementation question sentence. Specifically, the question format conversion unit 202 inserts the created text into the generation context 1202 of the backward-complementation question data 1200. For example, the question format conversion unit 202 inserts question identifier 501 of structured input question data 500 into question identifier 901 and correct answer 504 into correct answer 904. This completes the question format conversion process.

[0143] According to the fourth embodiment, the data conversion device 100 enables a language model to be evaluated independently without processing a probability distribution model, which requires high computational costs. Furthermore, the data conversion device 100 is configured mainly based on deterministic rules, which facilitates quality control and problem correction.

[0144] As described above, according to the above-mentioned first to fourth embodiments, it is possible to optimize the format of questions for evaluation targets such as language models. Specifically, for example, by converting questions used in existing human evaluation tests into questions in a format suitable for evaluating language models, it is possible to evaluate language models individually without manually creating questions in a format suitable for evaluating language models. This reduces the workload and optimizes the evaluation of language models.

[0145] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0146] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0147] Information such as programs, tables, files, etc. that realize each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0148] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0149] 100 Data conversion device 101 processors 102 Storage Devices 112 Complementary Question Output Unit 201 Input Question Structuring Unit 202 Question Format Conversion Unit 203 Complementary Question Output Unit 400 Input question data 410 Problem Data 411 Response Data 500 Structured Input Question Data 501 Question Identifier 502 Question body 503 Choices 504 Correct Answer 700 prompt templates 700 Backward Completion Question Data 701 Question Body Placeholder 702 Choice Placeholder 703 Instruction Prompt 704 Conversion specification explanation prompt 705 Case Prompts 800 prompt 901 Question Identifier 903 Choices 904 Correct Answer 1000 token dictionary 1100 Generated Text 1200 Backward Completion Question Data 1202 Creation Context 1203 Generated Choices 1301 Generation score calculation unit 1302 Indicator calculation section 1600 Backward Completion Question Prediction Results

Claims

1. A data conversion device having a processor that executes a program and a storage device that stores the program, The processor: an input process for inputting a plurality of options and a first question sentence in which one of the plurality of options is a correct answer; a conversion process for converting the first question into a second question having a format different from that of the first question, based on the plurality of options and the first question input by the input process; an output process for outputting the second question sentence converted by the conversion process and the plurality of options; A data conversion device characterized by executing the above.

2. 2. The data conversion device according to claim 1, The second question sentence is an explanatory sentence in a completion format in which a part is missing and is completed by completing it with the correct answer. A data conversion device characterized by:

3. 2. The data conversion device according to claim 1, the first question is a question asking which of the plurality of options is the correct answer; A data conversion device characterized by:

4. 2. The data conversion device according to claim 1, the first question sentence is a declarative sentence in which the correct answer is expressed using a demonstrative pronoun; A data conversion device characterized by:

5. 2. The data conversion device according to claim 1, In the conversion process, the processor generates the second question sentence by executing a prediction process that predicts the next word in a string of words from the beginning of the first question sentence. A data conversion device characterized by:

6. 2. The data conversion device according to claim 1, have access to a computer that implements a language model that predicts the next word in a word sequence; In the conversion process, the processor receives the second question sentence generated by the computer as a result of transmitting the first question sentence to the computer. A data conversion device characterized by:

7. 2. The data conversion device according to claim 1, The processor: performing an acquisition process of acquiring a score indicating the likelihood that the second question sentence will be generated for each of the options based on the word string of the second question sentence and the word string of the options; In the output process, the processor outputs the score for each of the options acquired in the acquisition process. A data conversion device characterized by:

8. 8. The data conversion device according to claim 7, a computer that implements a language model that predicts the likelihood of an answer to a question generated based on a word sequence of the question and a word sequence of an answer to the question; In the acquisition process, the processor transmits the word string of the second question sentence and the word string of the options to the computer, and as a result, sets a score for each option that indicates the likelihood of the option generating the second question sentence, calculated by the computer. A data conversion device characterized by:

9. 8. The data conversion device according to claim 7, The processor: performing a calculation process of calculating an index value indicating the performance of a language model for the second question sentence based on the score; A data conversion device characterized by:

10. 2. The data conversion device according to claim 1, In the conversion process, the processor analyzes a syntax tree of the first question, matches each subtree of the syntax tree with a predetermined rule, rearranges the order of words in the first question based on an operation method associated with the matched rule, and supplements a predetermined word to generate the second question. A data conversion device characterized by:

11. A data conversion method executed by a data conversion device having a processor that executes a program and a storage device that stores the program, comprising: The processor: an input process for inputting a plurality of options and a first question sentence in which one of the plurality of options is a correct answer; a conversion process for converting the first question into a second question having a format different from that of the first question, based on the plurality of options and the first question input by the input process; an output process for outputting the second question sentence converted by the conversion process and the plurality of options; A data conversion method comprising:

12. The processor an input process for inputting a plurality of options and a first question sentence in which one of the plurality of options is a correct answer; a conversion process for converting the first question into a second question having a format different from that of the first question, based on the plurality of options and the first question input by the input process; an output process for outputting the second question sentence converted by the conversion process and the plurality of options; A data conversion program characterized by executing the above.

Images