Information processing apparatus and computer readable medium

Abstract

Provided are an information processing apparatus and a computer readable medium. The information processing apparatus has a processor that performs processing of causing a question generator that generates a question sentence corresponding to an answer to generate a question sentence corresponding to an article and a plurality of answer candidates extracted from the article by inputting the article and the plurality of answer candidates to the question generator, and to output generation data including a group of the plurality of answer candidates and the question sentence.

Description

TECHNICAL FIELD

[0001] The present application relates to an information processing apparatus and a computer readable medium. BACKGROUND

[0002] A technique of extracting an appropriate answer from a related text when a question and the related text are given is proposed.

[0003] An information processing system that automatically generates teaching data for machine learning by a device is disclosed in Japanese Patent Application Publication No. 2019-16025. A storage section stores a verification rule that is set in advance, the verification rule indicating a condition for judging an output value of a learning model section with respect to an input value as true. A training section inputs a plurality of input values to the learning model section, acquires a plurality of output values of the learning model section with respect to the plurality of input values, judges whether the plurality of output values are respectively true with respect to the plurality of input values with reference to the verification rule, and stores a pair of an output value judged as true among the plurality of output values and a corresponding input value as new training data for teaching learning to the storage section.

[0004] A question generation device that achieves high answer accuracy for a question is disclosed in Japanese Patent Application Publication No. 2019-215841. The question generation device has a generation unit that inputs a question sentence and a related document containing an answer to the question sentence, generates a modified question sentence obtained by modifying a part of the question sentence in which a word is potentially missing with a word included in a predetermined set of words, using a machine learning model that is learned in advance. The generation unit has a collation unit that generates matching information indicating a coincidence relationship of each word included in the question sentence with each word included in the related document, and a question restoration unit that generates each word constituting the modified question sentence from the set of words using the matching information generated by the collation unit, thereby generating the modified question sentence.

[0005] A question answering device that achieves high accuracy of a question answer is disclosed in Japanese Patent Application Publication No. 2019-191827. The question answering device has an answer generation unit that inputs a document and a question, uses a word included in a union set of a predetermined first vocabulary and a second vocabulary constituted by words included in the document and the question, and performs a process of generating an answer sentence to the question by a learned model. The learned model includes a learned neural network learned in advance on whether a word included in the answer sentence is included in the second vocabulary, and increases or decreases a probability that the word included in the second vocabulary is selected as the word included in the answer sentence by the learned neural network at the time of generating the answer sentence.

[0006] When a machine reading comprehension model is used to extract answers to questions that require multiple answers from analysis target data, a multiple answer model needs to be learned in advance using a data set composed of question sentences and multiple answers to extract multiple answers from analysis target data. However, a multiple answer data set is more complex than a single answer data set, and thus it is not easier to prepare a multiple answer data set than a single answer data set. SUMMARY

[0007] An object of the present application is to generate a data set composed of question sentences and multiple answers using a single answer data set composed of question sentences and single answers.

[0008] According to a first aspect of the present application, there is provided an information processing apparatus including a processor that performs processing of causing a question generator that generates a question sentence corresponding to an answer to generate a question sentence corresponding to an article and a plurality of answer candidates extracted from the article by inputting the article and the plurality of answer candidates to the question generator, and causing the question generator to output generation data including a group of the plurality of answer candidates and the question sentence.

[0009] According to a second aspect of the present application, the processor uses the generation data as teaching data for a multiple answer reading comprehension model that extracts a plurality of answers from a question and analysis target data and outputs the extracted plurality of answers.

[0010] According to a third aspect of the present application, the processor performs processing of learning the multiple answer reading comprehension model using the generation data as the teaching data.

[0011] According to a fourth aspect of the present application, the processor further performs processing of learning the multiple answer reading comprehension model using a single answer data set as teaching data.

[0012] According to a fifth aspect of the present application, the processor performs processing of judging the appropriateness of the generation data and outputting only the generation data judged to be appropriate.

[0013] According to a sixth aspect of the present application, the processor judges the appropriateness of the generation data using a verification model that verifies the generation data.

[0014] According to a seventh aspect of the present application, the processor learns the verification model using appropriate question answer data extracted from a single answer data set and inappropriate question answer data generated from the single answer data set.

[0015] According to a ninth aspect of the present application, the processor causes the question generator to learn using a single answer data set prepared in advance.

[0016] According to a ninth aspect of the present application, the processor causes the question generator to learn using a single answer data set prepared in advance.

[0017] According to a tenth aspect of the present application, there is provided an information processing apparatus including a processor that performs processing of causing a question generator to generate a question corresponding to an article and at least one answer candidate extracted from the article by inputting the article and the answer candidate to the question generator, and outputting generation data including a group of the article and the question; and inputting the generation data to a machine reading comprehension model that extracts a plurality of answers corresponding to the question from the article and outputs the extracted answers, causing the machine reading comprehension model to generate the plurality of answers to the question in the input generation data, and causing the machine reading comprehension model to add the plurality of answers to the generation data and output.

[0018] According to an eleventh aspect of the present application, the processor performs processing of, in a case where the plurality of answers output by the machine reading comprehension model include different answers, causing the generation data output by the machine reading comprehension model to be used as teaching data for a multi-answer reading comprehension model that extracts a plurality of answers from a question and analysis target data and outputs the extracted plurality of answers.

[0019] According to a twelfth aspect of the present application, the processor performs processing of causing the machine reading comprehension model to learn the multi-answer reading comprehension model using the generation data output by the machine reading comprehension model as teaching data.

[0020] According to a thirteenth aspect of the present application, the processor further performs processing of causing the multi-answer reading comprehension model to learn using a single answer data set prepared in advance as teaching data.

[0021] According to a fourteenth aspect of the present application, the processor substitutes the learned multi-answer reading comprehension model for the machine reading comprehension model, and inputs output from the question generator to the machine reading comprehension model.

[0022] According to a fifteenth aspect of the present application, the processor performs processing of judging the appropriateness of the generation data, and using only the generation data judged to be appropriate as teaching data for the multi-answer reading comprehension model.

[0023] According to a 16th aspect of the present application, the processor judges the appropriateness of the generated data using a verification model that verifies the generated data.

[0024] According to a 17th aspect of the present application, the processor learns the verification model using appropriate question answer data extracted from a single answer data set and inappropriate question answer data generated from the single answer data set.

[0025] According to an 18th aspect of the present application, the processor judges the appropriateness of each of a plurality of the answer candidates for the generated data by the verification model.

[0026] According to a 19th aspect of the present application, the machine reading comprehension model extracts answers from the input question sentence using a plurality of single answer models prepared in advance.

[0027] According to a 20th aspect of the present application, the machine reading comprehension model extracts upper N answers from the input question sentence using 1 single answer model prepared in advance, where N is an integer of 1 or more.

[0028] According to a 21st aspect of the present application, the processor causes the question generator to learn using a single answer data set prepared in advance.

[0029] According to a 22nd aspect of the present application, there is provided a computer-readable medium storing a program that causes a computer to execute processing in which a question generator that generates a question sentence corresponding to an answer is caused to generate a question sentence corresponding to an article and a plurality of answer candidates extracted from the article by inputting the article and the plurality of answer candidates extracted from the article to the question generator, and the question generator is caused to output generated data including a group of the plurality of answer candidates and the question sentence.

[0030] According to a 23rd aspect of the present application, there is provided a computer-readable medium storing a program that causes a computer to execute processing in which a question generator that generates a question sentence corresponding to an answer is caused to generate a question sentence corresponding to an article and at least one answer candidate extracted from the article by inputting the article and the at least one answer candidate extracted from the article to the question generator, and the question generator is caused to output generated data including a group of the article and the question sentence, and a machine reading comprehension model that extracts a plurality of answers corresponding to the question sentence from the article and outputs the extracted answers is inputted with the generated data, and outputs data including a group of the plurality of answers with respect to the question sentence in the inputted generated data and the question sentence.

[0031] (EFFECTS)

[0032] According to the first aspect, a multiple answer data set composed of a question and a group of multiple answers can be generated using a single answer data set composed of a question and a single answer.

[0033] According to the second aspect, a multiple answer data set generated using a single answer data set composed of a question and a single answer can be used as teaching data for a multiple answer reading comprehension model.

[0034] According to the third aspect, a multiple answer reading comprehension model can be learned using a multiple answer data set generated using a single answer data set.

[0035] According to the fourth aspect, a multiple answer reading comprehension model can be learned using a single answer data set in addition to a multiple answer data set generated using a single answer data set.

[0036] According to the fifth aspect, the accuracy of a multiple answer data set can be improved compared to a case where the appropriateness of generated data is not judged.

[0037] According to the sixth aspect, the accuracy of the judgment of appropriateness can be improved compared to a case where a model that judges the appropriateness of generated data is not learned.

[0038] According to the seventh aspect, the learning accuracy of a multiple answer reading comprehension model can be improved compared to a case where the appropriateness of generated data is not judged.

[0039] According to the eighth aspect, the accuracy of a multiple answer data set can be improved compared to a case where the appropriateness of generated data is not judged.

[0040] According to the ninth aspect, the generation accuracy of an answer candidate generated by a question generator can be improved compared to a case where the question generator is not caused to learn using a single answer data set.

[0041] According to the tenth aspect, a multiple answer data set composed of a question and a group of multiple answers can be generated using a single answer data set composed of a question and a single answer.

[0042] According to the eleventh aspect, a multiple answer data set having different answers for a question generated using a single answer data set composed of a question and a single answer can be used as teaching data for a multiple answer reading comprehension model.

[0043] According to the twelfth aspect, a multiple answer reading comprehension model can be learned using a multiple answer data set generated using a single answer data set.

[0044] According to the 13th aspect, the multi-answer reading comprehension model can be learned from the single-answer dataset in addition to the multi-answer dataset generated using the single-answer dataset.

[0045] According to the 14th aspect, the multi-answer dataset can be generated using the multi-answer model learned from the multi-answer dataset.

[0046] According to the 15th aspect, the accuracy of the dataset of multiple answers can be improved compared to a case where the appropriateness of the generated data is not determined.

[0047] According to the 16th aspect, the determination accuracy of the appropriateness can be improved compared to a case where the model that determines the appropriateness of the generated data is not learned.

[0048] According to the 17th aspect, the learning accuracy of the multi-answer reading comprehension model can be improved compared to a case where the appropriateness of the generated data is not determined.

[0049] According to the 18th aspect, the accuracy of the dataset of multiple answers can be improved compared to a case where the appropriateness of the generated data is not determined.

[0050] According to the 19th aspect, the generation accuracy of the answer candidates generated by the question generator can be improved compared to a case where the question generator is not learned using the single-answer dataset.

[0051] According to the 20th aspect, the multi-answer dataset having multiple answers can be generated using the answers respectively output by the multiple single-answer models.

[0052] According to the 21st aspect, the multi-answer dataset having multiple answers can be generated using the multiple answers output by one single-answer model.

[0053] According to the 22nd aspect, the multi-answer dataset composed of a group of a question sentence and multiple answers can be generated using the single-answer dataset composed of a group of a question sentence and a single answer.

[0054] According to the 23rd aspect, the multi-answer dataset composed of a group of a question sentence and multiple answers can be generated using the single-answer dataset composed of a group of a question sentence and a single answer.

[0055] In a case where a question and analysis target data related to the question are given, multiple answers to the question can be extracted from the analysis target data with high accuracy compared to a case where multiple single-answer models are used to extract multiple answers. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a diagram illustrating an example of an extractive machine reading comprehension.

[0057] Figure 2 FIG. 6 is a graph showing an example of a probability distribution output by the answer model.

[0058] Figure 3 FIG. 7 is a block diagram showing a hardware structure of the information processing apparatus.

[0059] Figure 4 FIG. 8 is a block diagram showing an example of a functional structure of the information processing apparatus.

[0060] Figure 5 FIG. 9 is a flowchart showing a flow of the learning processing of the information processing apparatus.

[0061] Figure 6 FIG. 10 is a graph explaining an outline of the learning processing of the information processing apparatus.

[0062] Figure 7 FIG. 11 is a flowchart showing a flow of the learning processing of the information processing apparatus.

[0063] Figure 8 FIG. 12 is a graph explaining an outline of the learning processing of the information processing apparatus.

[0064] Figure 9 FIG. 13 is a graph explaining an outline of the learning processing of the information processing apparatus.

[0065] Figure 10 FIG. 14 is a graph showing an outline of the learning processing of the verification model.

[0066] Figure 11 FIG. 15 is a graph showing an outline of the verification processing of the verification model.

[0067] Figure 12 FIG. 16 is a graph explaining an outline of the learning processing of the information processing apparatus.

[0068] Figure 13 FIG. 17 is a graph explaining an outline of the re-construction of the question by the question re-construction section.

[0069] Figure 14 FIG. 18 is a graph showing a score of the multi-answer QA based on the multi-answer model.

[0070] Figure 15 FIG. 19 is a graph showing a score of the multi-answer QA based on the multi-answer model. DETAILED DESCRIPTION

[0071] Hereinafter, an example of an embodiment of the present application will be explained with reference to the attached drawings. In each drawing, the same or equivalent structural elements and parts are attached with the same reference numerals. Further, the size ratio of the drawings is sometimes exaggerated for the sake of explanation, and is not necessarily the same as the actual ratio.

[0072] Before the embodiments of the present application are explained in detail, the general principles thereof will be explained.

[0073] A machine reading comprehension task is a task for reading comprehension of given text and answering a question. The machine reading comprehension task is also an important task in benchmark testing of reading comprehension ability of a model for extracting an answer. For example, in an extractive machine reading comprehension such as SQuAD (Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.), a most appropriate answer is extracted from a context given a question and the context (context).

[0074] In the extractive machine reading comprehension, a technique for a single answer model corresponding to a correct answer to a question and a question answer based on a single answer range (hereinafter, "single answer QA") is being studied. However, the extractive machine reading comprehension requires a correct answer to a question corresponding not only to the single answer QA but also to a question answer based on a plurality of answer ranges (hereinafter, "multiple answer QA").

[0075] Figure 1 is a diagram illustrating an example of the extractive machine reading comprehension, and is a diagram illustrating an example of a question (Question) and a context (Context). In Figure 1 In the extractive machine reading comprehension illustrated in Figure 1 is illustrated in the case of extracting a single answer, "47-yard TD run" is extracted as an appropriate answer to a question in a context. Further, in Figure 1 is illustrated in the case of extracting a plurality of answers, "Brown" and "Gould" are extracted as appropriate answers to a question in a context.

[0076] An answer model performing the extractive machine reading comprehension outputs a probability distribution according to a context with respect to a range of answers considered to correspond to a question, and extracts an answer according to the probability distribution. Figure 2 is a diagram illustrating an example of a probability distribution output by the answer model. Figure 2 The content illustrated in is a probability distribution of a start word and an end word of an answer. In Figure 2In the example, in the response model, the word with the highest probability of being the start word of the response is calculated as "Josh", and the word with the highest probability of being the end word of the response is calculated as "Brown". Therefore, the response model outputs "Josh Brown".

[0077] The inventors of this application have discovered that when multiple answers are contained within related text, the answers output by a single-answer model are often inconsistent. Therefore, by using multiple single-answer models, it is possible to extract multiple answers from the question and related text. However, the inventors of this application have also found that there is room for improvement in the accuracy of answers when preparing multiple single-answer models to extract multiple answers based on the question. Therefore, it is preferable to extract multiple answers from the question and related text using a multi-answer model corresponding to a multi-answer QA.

[0078] To improve the accuracy of responses based on multi-response models, a multi-response model is learned. Learning a multi-response model for a multi-response QA requires a complex dataset (multi-response dataset) capable of generating multiple answers from a single question, compared to learning a single-response model for a single-response QA. Preparing such a complex dataset manually is very time-consuming. Therefore, instead of relying on a single-response model for multi-response QA, it is preferable to automatically generate a multi-response dataset from a dataset that does not require significant preparation from the dataset itself.

[0079] As explained below, it is possible to generate a multi-answer dataset from a single-answer dataset. Furthermore, the inventors of this application have developed a technique for learning a multi-answer model and extracting answers using the generated multi-answer dataset, which improves the accuracy of multiple answer extraction compared to preparing multiple single-answer models and extracting multiple answers based on a question.

[0080] Figure 3 This is a block diagram showing the hardware structure of the information processing device 10. Figure 3 The information processing device 10 shown is a device for extracting one or more answers based on a question using a multi-answer model corresponding to a single-answer QA and a multi-answer QA.

[0081] like Figure 3 As shown, the information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a memory 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. All these components are connected via a bus 19 to enable communication between them.

[0082] The CPU 11 is a central operation processing unit that executes various programs or controls each section. That is, the CPU 11 reads out a program from the ROM 12 or the memory 14, and executes the program using the RAM 13 as a work area. The CPU 11 performs the control of each structure and various operation processing according to the program recorded in the ROM 12 or the memory 14. In the present embodiment, a generation program for generating a multiple answer data set from a single answer data set, a learning program for learning a multiple answer model using a multiple answer data set, and an answer extraction program for extracting a plurality of answers to a question by a multiple answer model are stored in the ROM 12 or the memory 14.

[0083] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The memory 14 is configured by a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory, and stores various programs including an operating system and various data.

[0084] The input section 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs.

[0085] The display section 16 is, for example, a liquid crystal display that displays various information. The display section 16 can adopt a touch panel method, and function as the input section 15.

[0086] The communication interface 17 is an interface for communicating with other devices such as a user terminal, and standards such as Ethernet (registered trademark), FDDI, Wi-Fi (registered trademark), and the like can be used.

[0087] The information processing apparatus 10 realizes various functions using the above-described hardware resources when executing the above-described generation program, learning program, and answer extraction program. The functional structure realized by the information processing apparatus 10 will be described.

[0088] Figure 4 is a block diagram showing an example of the functional structure of the information processing apparatus 10.

[0089] As shown in Figure 4 , the information processing apparatus 10 has an input section 101, a question generator learning section 102, a question generator storage section 103, a model learning section 104, a model storage section 105, and an output section 106 as a functional structure. Each functional structure is realized by reading out and executing the generation program, the learning program, and the answer extraction program stored in the ROM 12 or the memory 14 by the CPU 11.

[0090] The input unit 101 inputs information for causing the model learning unit 104 to learn a multi-answer model and information for causing the multi-answer model to extract an answer. Specific examples of the information input by the input unit 101 will be described later. Further, the input unit 101 inputs an article and a question sentence for causing the learned multi-answer model to extract a plurality of answers.

[0091] The question generator learning unit 102 causes a question generator to learn, the question generator outputting generation data including a group of an article (context) for learning a multi-answer model, a question, and an answer. The question generator is stored in the question generator storage unit 103 and is caused to learn using a single-answer data set generated by a person. The kind of the single-answer data set is arbitrary. The question generator can be constructed by, for example, a neural network. The single-answer data set is a data set having a plurality of groups including an article, a question related to the article, and an answer to the question. The question generator, when input with an article and an answer candidate in the article, generates a question in which the answer candidate becomes an answer, and outputs output data including a group of the generated article and the question. The generation data output from the question generator becomes a basis of learning data for learning a multi-answer model. That is, the generation data output from the question generator becomes a basis of a multi-answer data set generated from a single-answer data set.

[0092] The model learning unit 104 learns a multi-answer model using generation data including a group of an article, a question, and an answer output by the question generator. The multi-answer model is stored in the model storage unit 105. The model learning unit 104 learns the multi-answer model using, in addition to the generation data output from the question generator, a single-answer data set used in the learning of the question generator.

[0093] The output unit 106 outputs a result of machine reading comprehension based on a multi-answer model, with respect to an article and a question sentence input by the input unit 101.

[0094] Next, the operation of the information processing apparatus 10 will be described.

[0095] (1st generation processing and learning processing)

[0096] Figure 5 is a flowchart showing the flow of the generation processing and the learning processing of the information processing apparatus 10. Further, Figure 6 is a diagram showing an outline of the learning processing of the information processing apparatus 10. The CPU 11 reads out and expands in the RAM 13 a generation program and a learning program from the ROM 12 or the memory 14, and executes the generation program and the learning program, whereby the learning processing is performed.

[0097] The CPU 11 causes the question generator to learn using a prepared single-answer data set (step S101). The single-answer data set used in the learning of the question generator can use any single-answer data set.

[0098] Using Figure 6 The processing of step S101 is described. The CPU 11 causes the question generator 122 to learn by inputting a prepared single-answer data set 121 into the question generator 122. The single-answer data set 121 is a data set having a plurality of groups as described above, the group containing an article, a question related to the article, and an answer to the question. The learning method of the question generator 122 is not limited to a particular method. For example, the CPU 11 can also update the parameters of the question generator 122 in such a way that the correct question is reduced from the error of the question generated by the question generator 122, thereby causing the question generator 122 to learn.

[0099] After step S101, the CPU 11 inputs an article and an answer candidate in the article to the learned question generator 122 (step S102). In step S102, an article for which there are a plurality of answer candidates and the answer candidates are input to the question generator 122.

[0100] Using Figure 6 The processing of step S102 is described. The CPU 11 extracts a group X' containing an article and an answer candidate in the article from the article data 123 in which a plurality of groups are stored, and inputs the group X' to the question generator 122. X' input to the question generator 122 from the article data 123 contains an article c and an answer candidate a* contained in the article c. The answer candidate a* for one article c is at least two. The answer candidate a* can be generated by previously analyzing the article c, or can be generated by a person reading the article c and specifying a sentence that can be an answer candidate from the article c. In the article data 123, articles extracted from various document sources can be stored in advance. The document sources can be, for example, web pages on the Internet. Furthermore, the article data 123 can be stored in the same device as the information processing apparatus 10, or can be stored in a different device from the information processing apparatus 10.

[0101] After step S102, the CPU 11 causes the question generator 122 to generate a question based on the article and the answer candidate input to the question generator 122, and causes it to output generated data consisting of a group of the article, the question, and the answer candidate (step S103).

[0102] Using Figure 6The processing in step S103 is explained below. When a group X' consisting of article c and answer candidates a* contained in that article is input from article data 123 to question generator 122, question generator 122 generates a question q* such that answer candidate a* becomes an answer based on article c and answer candidate a*. Since there are at least two answer candidates a* for one article c, the question q* generated by question generator 122 is a question where multiple answer candidates a* become answers. Then, question generator 122 outputs generated data X' consisting of article c, the generated question q*, and answer candidate a*. The generated data X' output by question generator 122 becomes a multi-answer dataset.

[0103] After step S103, CPU 11 uses the generated data generated by question generator 122 to learn a multi-answer model (step S104).

[0104] use Figure 6 The processing in step S104 will be explained. The generated data X' produced by the question generator 122 is input into the multi-answer model 124. The CPU 11 uses the generated data X' to learn the multi-answer model 124. While learning the multi-answer model 124, the CPU 11 further inputs the single-answer dataset 121 into the multi-answer model 124. The learning method for the multi-answer model 124 is not limited to a specific method. For example, the CPU 11 can also update the parameters of the multi-answer model 124 in a manner that reduces the error between the answer candidate a* extracted in step S102 and the answer predicted by the multi-answer model 124, thereby learning the multi-answer model 124.

[0105] CPU 11 executes Figure 5 The series of processes shown enables the question generator to generate generated data X' containing multiple answers. Furthermore, CPU 11 executes... Figure 5 The series of processes shown enables the learning of a multi-answer model using generated data X' produced by a question generator, which outputs multiple answers. The CPU 11 learns the multi-answer model based on the generated data X' produced by the question generator, which is learned using a single-answer dataset. This improves the accuracy of multiple answer extraction compared to preparing multiple single-answer models and extracting multiple answers based on the question.

[0106] (Second generation and learning processes)

[0107] This is another example illustrating the generation and learning processes of the information processing device 10. Figure 7 This is a flowchart illustrating the learning process of the information processing device 10. Furthermore, Figure 8FIG. 1 is a diagram illustrating an overview of the learning processing of the information processing apparatus 10. The CPU 11 reads out the generation program and the learning program from the ROM 12 or the memory 14 and expands them in the RAM 13 to execute the generation program and the learning program, thereby performing the learning processing.

[0108] The CPU 11 causes the question generator to perform learning using the prepared single-answer dataset (step S111).

[0109] After step S111, the CPU 11 inputs an article and a candidate answer in the article to the learned question generator 122 (step S112). In step S112, an article for which there is one candidate answer for one article and the candidate answer are input to the question generator 122.

[0110] After step S112, the CPU 11 causes the question generator 122 to generate a question based on the article and the candidate answer input to the question generator 122 and to output generation data composed of an article, a question, and a candidate answer (step S113).

[0111] Using Figure 8 The processing of step S113 will be described. When the question generator 122 is input with a group X' composed of an article c and a candidate answer a* included in the article from the article data 123, the question generator 122 generates a question q* based on the article c and the candidate answer a* such that the candidate answer a* becomes an answer. The candidate answer a* for one article c is one, and thus the question q* generated by the question generator 122 is a question in which one candidate answer a* becomes an answer. Then, the question generator 122 outputs generation data X' composed of the article c, the generated question q*, and the candidate answer a*.

[0112] After step S113, the CPU 11 inputs the generation data generated by the question generator 122 to the unsupervised multiple-answer model 125 (step S114).

[0113] Using Figure 8 The processing of step S114 will be described. The generation data X' generated by the question generator 122 is input to the unsupervised multiple-answer model 125. The unsupervised multiple-answer model can be composed of a plurality of single-answer models learned in advance or can be composed of one single-answer model learned in advance.

[0114] In a case where the no-teaching multi-answer model 125 is constituted by a plurality of single-answer models learned in advance, the no-teaching multi-answer model 125 causes each single-answer model to extract an answer from a question according to input of the generated data X'. Further, in a case where the answers extracted by each single-answer model are different, the no-teaching multi-answer model 125 outputs a plurality of answers according to the generated data X'. In a case where the answers extracted by each single-answer model are all the same, the no-teaching multi-answer model 125 outputs a single answer according to the generated data X'. In a case where each single-answer model cannot extract an answer from a question, the no-teaching multi-answer model 125 outputs a result like an answer cannot be extracted from the generated data X'.

[0115] In a case where the no-teaching multi-answer model 125 is constituted by one single-answer model learned in advance, the no-teaching multi-answer model 125 causes the single-answer model to attempt to extract N upper answers from a question according to input of the generated data X'. Then, in a case where the N upper answers extracted by the single-answer model are different, the no-teaching multi-answer model 125 outputs a plurality of answers according to the generated data X'. In a case where the N upper answers extracted by the single-answer model are the same, the no-teaching multi-answer model 125 outputs a single answer according to the generated data X'. In a case where the single-answer model cannot extract an answer from a question, the no-teaching multi-answer model 125 outputs a result like an answer cannot be extracted from the generated data X'.

[0116] In a case where a plurality of answers have been extracted from the generated data X', the no-teaching multi-answer model 125 replaces the answer candidate of the generated data X' with the extracted answers. If explained using Figure 8 , in a case where a plurality of answers have been extracted from the generated data X', the no-teaching multi-answer model 125 replaces the input answer candidate a* with the extracted answers a^.

[0117] After the step S114, the CPU 11 learns a multi-answer model using the generated data output by the no-teaching multi-answer model 125 (step S115).

[0118] Using Figure 8The processing of step S115 will be described. The generated data X' output by the no-teaching multiple answer model 125 is input to the multiple answer model 124. The CPU 11 learns the multiple answer model 124 using the generated data X' output by the no-teaching multiple answer model 125. The CPU 11 further inputs the single answer data set 121 to the multiple answer model 124 while learning the multiple answer model 124. The learning method of the multiple answer model 124 is not limited to a particular method. For example, the CPU 11 can also update the parameters of the multiple answer model 124 in a manner that reduces the error of the answer replaced by the no-teaching multiple answer model 125 from the answer predicted by the multiple answer model 124, thereby learning the multiple answer model 124.

[0119] In addition, in a case where the generated data output by the no-teaching multiple answer model 125 contains only one answer or the no-teaching multiple answer model 125 does not extract an answer, the generated data X' output by the no-teaching multiple answer model 125 is not used in the learning of the multiple answer model 124.

[0120] The multiple answer model 124 learned using the generated data X' can also be replaced by the no-teaching multiple answer model 125 to which the generated data generated by the question generator 122 is input. By replacing the multiple answer model 124 learned using the generated data X' by the no-teaching multiple answer model 125, it is possible to improve the learning accuracy of the multiple answer model 124.

[0121] After step S115, the CPU 11 determines whether the learning of the multiple answer model 124 has been performed a predetermined number of times (step S116).

[0122] If the determination result of step S116 is that the number of times of learning of the multiple answer model 124 has not reached the predetermined number of times (step S116: No), the CPU 11 returns to step S112 and inputs the article and the answer candidate in the article to the learned question generator 122. On the other hand, if the determination result of step S116 is that the number of times of learning of the multiple answer model 124 has reached the predetermined number of times (step S116: Yes), the CPU 11 ends the series of processing.

[0123] The CPU 11 can learn the multiple answer model that outputs multiple answers from an article, a question generated by a question generator, and an answer extracted from the question by executing the series of processing illustrated in FIG. 12. Figure 7 The CPU 11 can learn the multiple answer model from a question generated by a question generator learned using a single answer data set, thereby improving the extraction accuracy of multiple answers compared to a case where multiple single answer models are prepared and multiple answers are extracted from a question.

[0124] (Verification of Generated Data)

[0125] The CPU 11 can also verify whether the question and answer contained in the generation data output by the question generator 122 are appropriate question and answer. Then, only in a case where it is verified that they are appropriate question and answer, the CPU 11 can use the generation data output by the question generator 122 in the learning of the multi-answer model 124. The verification of the generation data can be applied to either one of the first generation processing and the second generation processing described above.

[0126] Figure 9 is a diagram illustrating an outline of the learning processing of the information processing apparatus 10. The verification model 126 is a model that verifies whether the question and answer contained in the generation data X' output by the question generator 122 are appropriate question and answer. Only the generation data X' judged by the verification model 126 to be appropriate question and answer is used in the learning of the multi-answer model 124.

[0127] A specific example of the learning processing of the verification model 126 is described. The verification model 126 is learned using appropriate question and answer (Gold QA) output by the single-answer data set 121 and inappropriate question and answer (Pseudo QA) generated by the single-answer data set 121. Figure 10 is a diagram illustrating an outline of the learning processing of the verification model 126. For the Gold QA, True (appropriate) is assigned as a teaching label, and for the Pseudo QA, False (inappropriate) is assigned as a teaching label.

[0128] Next, a specific example of the verification processing based on the verification model 126 is described. The verification model 126 verifies the answer candidates contained in the generation data X' output by the question generator 122 one by one. Figure 11 is a diagram illustrating an outline of the verification processing of the verification model 126. In Figure 11 , the verification model 126 verifies the N1 answer candidates contained in the generation data X' output by the question generator 122 respectively.

[0129] In a case where the verification model 126 judges that all the answer candidates are appropriate as a result of verifying the N1 answer candidates respectively, the generation data X' output by the question generator 122 can also be used in the learning of the multi-answer model 124. Then, in a case where the verification model 126 judges that none of the answer candidates is appropriate, the generation data X' output by the question generator 122 can also not be used in the learning of the multi-answer model 124.

[0130] Further, in a case where the result of the verification model 126 verifying the answers of the Nl answer candidates respectively is judged to be a predetermined threshold Ml or more (Ml < Nl) of the answer candidates being proper, the generated data X' output by the question generator 122 can also be used in the learning of the multiple answer model 124. Then, in a case where the verification model 126 judges that only less than Ml of the answer candidates are proper, the generated data X' output by the question generator 122 can also not be used in the learning of the multiple answer model 124.

[0131] Further, after the verification model 126 verifies the answers of the Nl answer candidates respectively, the generated data X' output by the question generator 122 can also be set as the teaching data of the multiple answer model 124 only for the answer candidates judged to be proper as the answer candidates a^. In a case where the verification model 126 judges that the number of the answer candidates judged to be proper is one or less, the generated data X' output by the question generator 122 is not used in the learning of the multiple answer model 124.

[0132] (Reconstruction of answers)

[0133] The CPU 11 can also attempt to reconstruct the answers included in the generated data X' output by the question generator 122. The CPU 11 can also perform the reconstruction of the answers by inputting to a plurality of single answer models. The reconstruction of the answers can also be applied to either of the first generation processing and the second generation processing described above.

[0134] Figure 12 is a diagram that outlines the learning processing of the information processing apparatus 10. The question reconstruction section 127 is constituted by a plurality of single answer models, and the generated data X' output by the question generator 122 is input to the question reconstruction section 127. Then, the question reconstruction section 127 performs the reconstruction of the question for the generated data X', and outputs only the generated data X' that has been reconstructed as the teaching data of the multiple answer model 124.

[0135] Figure 13 is a diagram that outlines the reconstruction of the question by the question reconstruction section 127. The generated data X' output by the question generator 122 is input to M2 single answer models. M2 is M2 or more of the number N2 of the answer candidates included in the generated data X'. The M2 single answer models generate and output predicted answers a~M2 respectively.

[0136] Then, the question reconstruction section 127 appends the generated data X' output by the question generator 122 to the teaching data of the multiple answer model 124 only in a case where all of the predicted answers coincide with the answer candidates. The question reconstruction section 127 evaluates whether the predicted answers coincide with the answer candidates according to the exact match (EM) or the partial match (Fl) of the words or the character strings of each answer.

[0137] In Figure 13 the example, the answer candidate a2 does not coincide with any of the predicted answers, and therefore, the question reformulation unit 127 regards the generation data X' as a case where reformulation of the answer candidate has failed. Therefore, the question reformulation unit 127 does not add the generation data X' as the teaching data of the multiple answer model 124 and discards it as an inappropriate example.

[0138] The information processing apparatus 10 adds the generation data X' output by the reformulation by the question reformulation unit 127 as the teaching data of the multiple answer model 124, and thereby can improve the accuracy of the teaching data used in the learning of the multiple answer model 124 compared to the case where the question reformulation unit 127 does not reformulate.

[0139] (Effect of the generation processing and the learning processing of the present embodiment)

[0140] The effect of the generation processing and the learning processing of the present embodiment will be described.

[0141] Figure 14 is a graph showing scores of multiple answer QA based on a multiple answer model learned by a learning processing using generation data generated by the generation processing of the present embodiment. In Figure 14 , a multiple answer model learned by the 2nd learning processing was used. As a comparative example, a multiple answer model using multiple single answer models (no teaching model) to extract multiple answers was illustrated. In the single answer data set, QUOREF (Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, Matt Gardner. Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning. In EMNLP-IJCNLP, 2019.) was used. As Figure 14 shown in FIG. 8, regarding the scores of the multiple answer QA based on the multiple answer model learned by the learning processing of the present embodiment, both the exact match (EM) and the partial match (F1) showed higher scores than the case where multiple answers were extracted using multiple single answer models.

[0142] Figure 15 is a graph showing scores of multiple answer QA based on a multiple answer model learned by a learning processing using generation data generated by the generation processing of the present embodiment. In Figure 15In this study, a multi-answer model learned through the second learning process was used. As a comparative example, a multi-answer model learned using a manually generated dataset (Gold) and a multi-answer model extracting multiple answers using multiple single-answer models (untaught models) are shown. Figure 14 Similarly, QUOREF was used in a single-response dataset.

[0143] like Figure 15 As shown, the model with the highest score is the multi-answer model learned using a dataset (Gold) manually generated by humans. However, regarding the scores of multi-answer QA based on the multi-answer model learned through the learning process of this embodiment, both perfect agreement (EM) and partial agreement (F1) scores are higher than those obtained by using multiple single-answer models to extract multiple answers.

[0144] Thus, the information processing apparatus 10 of this embodiment can use a multi-answer model learned from a multi-answer dataset generated from a single-answer dataset to extract multiple answers from an article. The information processing apparatus 10 of this embodiment can generate a multi-answer dataset from a single-answer dataset. Furthermore, the information processing apparatus 10 of this embodiment uses a multi-answer model learned from the generated multi-answer dataset to extract answers, thereby improving the accuracy of multiple answer extraction compared to preparing multiple single-answer models to extract multiple answers based on a question.

[0145] Alternatively, the generation, learning, and response extraction processes, which are read into and executed by the CPU in the above embodiments, can also be performed by various processors other than the CPU. An example of a processor in this case is an FPGA (Field-Programmable Gate Array).

[0146] These include PLDs (Programmable Logic Devices), such as Field-Programmable Gate Arrays (FPGAs), whose circuit structure can be modified after manufacturing, and ASICs (Application Specific Integrated Circuits), which are processors with specially designed circuit structures to perform specific processes, i.e., dedicated circuits. Furthermore, generation, learning, and response extraction processes can be performed by a single processor from these various types, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, and combinations of CPUs and FPGAs). More specifically, the hardware structure of these various processors is a circuit obtained by combining circuit elements such as semiconductor components.

[0147] Further, in each of the above-described embodiments, a manner in which programs of the generation processing, the learning processing, and the answer extraction processing are stored (installed) in advance in a ROM or a memory is described, but the present technology is not limited thereto. The programs can also be provided in a manner in which they are recorded in a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory. Further, the programs can also be provided in a manner in which they are downloaded from an external device via a network.

[0148] In each of the above-described embodiments, the processor refers to a broad processor, and includes a general-purpose processor (for example, a CPU (Central Processing Unit) or the like), a dedicated processor (for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a programmable logic device, or the like), and the like.

[0149] Further, the actions of the processor in each of the above-described embodiments are not only completed by one processor, but can also be cooperatively completed by a plurality of processors existing at physically remote locations. Further, the order of each action of the processor is not limited to the order described in each of the above-described embodiments, but can also be appropriately changed.

Claims

1. An information processing device, wherein, The information processing device has a processor. The processor performs the following process: by inputting an arbitrary article and multiple answer candidates extracted from the article into a question generator, the question generator generates a question corresponding to the article and the multiple answer candidates, and the question generator outputs generated data containing a group of the multiple answer candidates and the question. The processor uses appropriate question-answer data extracted from a single-answer dataset and inappropriate question-answer data generated based on the single-answer dataset to learn a validation model for the generated data. The processor uses the verification model to determine the appropriateness of the generated data. The processor only outputs the generated data that it deems appropriate.

2. The information processing apparatus according to claim 1, wherein, The processor uses the generated data as teaching data for a multi-answer reading comprehension model, which extracts multiple answers from the question and analysis object data and outputs the extracted multiple answers.

3. The information processing apparatus according to claim 2, wherein, The processor performs the process of using the generated data as teaching data to learn the multi-answer reading comprehension model.

4. The information processing apparatus according to claim 3, wherein, The processor also performs the process of using a single-answer dataset as teaching data to learn the multi-answer reading comprehension model.

5. The information processing apparatus according to claim 1, wherein, The processor uses the verification model to determine the appropriateness of each of the multiple candidate answers in the generated data.

6. The information processing apparatus according to claim 1, wherein, The processor enables the question generator to learn using a pre-prepared single-answer dataset.

7. An information processing apparatus, wherein, The information processing device has a processor. The processor performs the following processes; By inputting any article and at least one answer candidate extracted from the article into the question generator, the question generator generates the question corresponding to the article and the answer candidate, and outputs generated data containing the article and the question. as well as The generated data is input into a machine reading comprehension model that extracts multiple answers corresponding to interrogative sentences from an article and outputs the extracted answers. The machine reading comprehension model then generates multiple answers to the interrogative sentences in the input generated data, adds the multiple answers to the generated data, and outputs the results. When the multiple answers output by the machine reading comprehension model contain different answers, the processor uses the generated data output by the machine reading comprehension model as teaching data for a multi-answer reading comprehension model. This multi-answer reading comprehension model extracts multiple answers from the question and analysis object data and outputs the extracted multiple answers. The processor uses appropriate question-answer data extracted from a single-answer dataset and inappropriate question-answer data generated based on the single-answer dataset to learn a validation model that validates the generated data output by the machine reading comprehension model. The processor uses the verification model to determine the appropriateness of the generated data output by the machine reading comprehension model. The processor uses only the generated data output by the machine reading comprehension model that it deems appropriate as teaching data for the multi-answer reading comprehension model.

8. The information processing apparatus according to claim 7, wherein, The processor performs the process of using the generated data output by the machine reading comprehension model as teaching data to learn the multi-answer reading comprehension model.

9. The information processing apparatus according to claim 8, wherein, The processor also performs the process of learning the multi-answer reading comprehension model by using a pre-prepared single-answer dataset as teaching data.

10. The information processing apparatus according to claim 9, wherein, The processor uses the learned multi-answer reading comprehension model as the machine reading comprehension model and inputs the output from the question generator into the machine reading comprehension model.

11. The information processing apparatus according to claim 7, wherein, The processor uses the verification model to determine the appropriateness of each of the multiple candidate answers in the generated data.

12. The information processing apparatus according to claim 7, wherein, The machine reading comprehension model uses multiple pre-prepared single-response models to extract answers from the interrogative sentences in the input generated data.

13. The information processing apparatus according to claim 7, wherein, The machine reading comprehension model uses a pre-prepared single-response model to extract N upper-level answers from the input question, where N is an integer greater than or equal to 1.

14. The information processing apparatus according to claim 7, wherein, The processor uses a pre-prepared single-answer dataset to enable the question generator to learn.

15. A computer-readable medium storing a program that causes a computer to perform processing, wherein... In the process, by inputting an arbitrary article and multiple answer candidates extracted from the article into a question generator, the question generator generates a question corresponding to the article and the multiple answer candidates, and outputs generated data containing a group of the multiple answer candidates and the question. A validation model for the generated data is learned using appropriate question-and-answer data extracted from a single-answer dataset and inappropriate question-and-answer data generated from the single-answer dataset. The verification model is used to determine the appropriateness of the generated data. Only output the generated data that is deemed appropriate.

16. A computer-readable medium storing a program that causes a computer to perform processing, wherein, In the process, By inputting any article and at least one answer candidate extracted from the article into the question generator, the question generator generates the question corresponding to the article and the answer candidate, and outputs generated data containing the article and the question. as well as The generated data is input into a machine reading comprehension model that extracts multiple answers corresponding to interrogative sentences from an article and outputs the extracted answers. The output is generated data containing multiple answers to the interrogative sentences in the input generated data and groups of the interrogative sentences. When the multiple answers output by the machine reading comprehension model contain different answers, the generated data output by the machine reading comprehension model is used as teaching data for a multi-answer reading comprehension model. This multi-answer reading comprehension model extracts multiple answers from the question and analysis object data and outputs the extracted multiple answers. Using appropriate question-and-answer data extracted from a single-answer dataset and inappropriate question-and-answer data generated from the single-answer dataset, a validation model is learned to validate the generated data output by the machine reading comprehension model. The verification model is used to determine the appropriateness of the generated data output by the machine reading comprehension model. Only the generated data output by the machine reading comprehension model that is deemed appropriate is used as the teaching data for the multi-answer reading comprehension model.

17. A computer program product comprising a program that causes a computer to perform processing, wherein, In the process, by inputting an arbitrary article and multiple answer candidates extracted from the article into a question generator, the question generator generates a question corresponding to the article and the multiple answer candidates, and outputs generated data containing a group of the multiple answer candidates and the question. A validation model for the generated data is learned using appropriate question-and-answer data extracted from a single-answer dataset and inappropriate question-and-answer data generated from the single-answer dataset. The verification model is used to determine the appropriateness of the generated data. Only output the generated data that is deemed appropriate.

18. A computer program product comprising a program that causes a computer to perform processing, wherein, In the process, By inputting any article and at least one answer candidate extracted from the article into the question generator, the question generator generates the question corresponding to the article and the answer candidate, and outputs generated data containing the article and the question. as well as The generated data is input into a machine reading comprehension model that extracts multiple answers corresponding to interrogative sentences from an article and outputs the extracted answers. The output is generated data containing multiple answers to the interrogative sentences in the input generated data and groups of the interrogative sentences. When the multiple answers output by the machine reading comprehension model contain different answers, the generated data output by the machine reading comprehension model is used as teaching data for a multi-answer reading comprehension model. This multi-answer reading comprehension model extracts multiple answers from the question and analysis object data and outputs the extracted multiple answers. Using appropriate question-and-answer data extracted from a single-answer dataset and inappropriate question-and-answer data generated from the single-answer dataset, a validation model is learned to validate the generated data output by the machine reading comprehension model. The verification model is used to determine the appropriateness of the generated data output by the machine reading comprehension model. Only the generated data output by the machine reading comprehension model that is deemed appropriate is used as the teaching data for the multi-answer reading comprehension model.

Images