Answer generation apparatus, answer generation method, computer readable medium, and computer program product

Abstract

The present application provides an answer generation device, an answer generation method and a computer readable medium. The answer generation device comprises: an input unit configured to input a question and data to be analyzed, i.e., analysis object data; a processor; and an output unit. The processor performs the following processing by executing a program. In a case where the question and the analysis object data are input, a plurality of single answer models prepared in advance are used to independently extract answers to the question from the analysis object data, the occurrence frequency of each mark of the extracted answers is calculated, and a single answer or a plurality of answers are extracted based on the occurrence frequency and output to the output unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an answer generation apparatus, an answer generation method, and a computer-readable medium. BACKGROUND

[0002] Conventionally, a technology of extracting an appropriate answer from a related text when a question and the related text are given has been proposed.

[0003] In Japanese Patent Application Publication No. 2019-220142, an apparatus capable of accurately answering a question in polarity is described. A machine reading section estimates a start and an end of the range using a reading model learned in advance, which is used to estimate a range in a text that becomes a basis for an answer to a question sentence based on the input text and the question sentence. A determination section determines the polarity of the answer to the question sentence using a determination model learned in advance, which is used to determine whether the polarity of the answer to the question sentence is accurate based on information obtained through processing by the machine reading section.

[0004] In Japanese Patent Application Publication No. 2019-191827, an apparatus for achieving a high-precision question answer is described. The apparatus has an answer generation unit that uses a predetermined first vocabulary and a word included in a set of a second vocabulary composed of words included in a document and a question as input to perform processing of generating an answer sentence to the question using a learned model. The learned model includes a learned neural network that has been learned in advance on whether a word included in the answer sentence is included in the second vocabulary, and the learned neural network increases or decreases a probability of selecting a word included in the second vocabulary as a word included in the answer sentence when generating the answer sentence. SUMMARY

[0005] An object of the present disclosure is to provide a technology capable of extracting a plurality of answers to a question from data that becomes a parsing target, i.e., parsing target data, when the question and a text or the like associated with the question are given as the parsing target data.

[0006] According to a first aspect of the present disclosure, there is provided an answer generation apparatus, comprising:

[0007] The answer generation apparatus includes an input unit that inputs a question and data to be analyzed as an analysis target, a processor that performs processing of extracting answers to the question from the analysis target data using a plurality of single answer models prepared in advance, calculating the frequency of occurrence of each token of the extracted answers, and extracting a single answer or a plurality of answers based on the frequency of occurrence and outputting the extracted answer or answers to an output unit, and the output unit.

[0008] According to a second aspect of the present disclosure, the processor uses a first threshold value and a second threshold value smaller than the first threshold value, in a case where the frequency of occurrence is equal to or greater than the first threshold value, determines tokens having the frequency of occurrence equal to or greater than the first threshold value as a single answer and outputs the tokens, in a case where the frequency of occurrence is smaller than the first threshold value and equal to or greater than the second threshold value, determines tokens having the frequency of occurrence smaller than the first threshold value and equal to or greater than the second threshold value as a plurality of answers and outputs the tokens, and in a case where the frequency of occurrence is smaller than the second threshold value, does not select the tokens as answers.

[0009] According to a third aspect of the present disclosure, the processor, in a case where the determination results for a plurality of tokens in succession are the same, instead of outputting each token as an output result, selects tokens in succession in a manner that the sequence length becomes the largest and outputs the tokens.

[0010] According to a fourth aspect of the present disclosure, the processor, in a case where, after determining a processing target token as a single answer, a next processing target token is determined as a plurality of answers or no answer, outputs only the token determined as a single answer.

[0011] According to a fifth aspect of the present disclosure, the processor uses the plurality of single answer models equal to or greater than the number of answers.

[0012] According to a sixth aspect of the present disclosure, the plurality of single answer models are respectively models that, based on a deep learning method, encode the question and the analysis target data into distributed representations, and based on the encoded distributed representations, extract a range of answers having the highest probability from the analysis target data through a fully connected layer for estimating a start point and an end point of the answers.

[0013] According to a seventh aspect of the present disclosure, the plurality of single answer models are respectively learned using learning data that gives only a single answer to a question.

[0014] According to an eighth aspect of the present disclosure, the analysis target data is text data, and the tokens are words or character strings in the text data.

[0015] According to a ninth aspect of the present disclosure, there is provided a computer-readable medium storing a program causing a computer to execute a process, wherein the process has the steps of: inputting a question and data to be analyzed as analysis target data; extracting answers to the question from the analysis target data independently using a plurality of single answer models prepared in advance; calculating the frequency of occurrence of each token of the extracted answers; and extracting and outputting a single answer or a plurality of answers based on the frequency of occurrence.

[0016] According to a tenth aspect of the present disclosure, there is provided an answer generation method, wherein the answer generation method has the steps of: inputting a question and data to be analyzed as analysis target data; extracting answers to the question from the analysis target data independently using a plurality of single answer models prepared in advance; calculating the frequency of occurrence of each token of the extracted answers; and extracting and outputting a single answer or a plurality of answers based on the frequency of occurrence.

[0017] Effects of Invention

[0018] According to the first aspect, the ninth aspect, or the tenth aspect, when a question and analysis target data are given, a plurality of answers to the question can be extracted from the analysis target data.

[0019] According to the second aspect, the third aspect, or the fourth aspect, any one of a single answer, a plurality of answers, and no answer can be further determined and output.

[0020] According to the fifth aspect or the sixth aspect, the accuracy of the answers can be further improved.

[0021] According to the seventh aspect, learning data can be further ensured.

[0022] According to the eighth aspect, words or strings can be further extracted from text data as answers. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is an explanatory diagram of a text and a question (1).

[0024] Figure 2 is an explanatory diagram of a text and a question (2).

[0025] Figure 3 is a chart showing the average number of differences in answers of a single answer QA and a plurality of answers QA.

[0026] Figure 4 is a structural block diagram of an answer generation device of an embodiment.

[0027] Figure 5 is a functional block diagram of an embodiment.

[0028] Figure 6 is a structural diagram of a machine reading comprehension model of an embodiment.

[0029] Figure 7 is a processing flowchart of an embodiment. DETAILED DESCRIPTION

[0030] Hereinafter, an embodiment of the present disclosure will be described based on the drawings, taking text data as an example of analysis target data that is data to be analyzed.

[0031] <Basic Principle>

[0032] First, the basic principle of the present embodiment will be described.

[0033] The purpose of a machine reading comprehension task is to read and comprehend a given text and answer a question, and it is also an important task in terms of benchmarking the reading comprehension ability of a model. In an extractive machine reading comprehension such as SQuAD (Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.), for example, the most appropriate answer is extracted from the context when a question and a related text (context) are given.

[0034] Figure 1 An example of a question (Question) and a related text (Context) is shown. As an appropriate answer to the question in the related text, "Russel" is extracted.

[0035] However, the extractive machine reading comprehension so far has been limited to a question answering (hereinafter referred to as "single answer QA") in which the correct answer to a question is one, and cannot cope with such a question answering in which multiple answers are simultaneously correct.

[0036] On the other hand, in order to extract answers in multiple ranges, several models for extracting appropriate answers without too much or too little have been proposed. However, these models are all premised on learning using data for machine reading comprehension with multiple answer QAs, and it is not realistic to always prepare such complex data.

[0037] In a case where a model capable of only a single answer (hereinafter referred to as "single answer model") is applied, the answer estimated by the model is always limited to one. The present applicant has found that, when a plurality of single answer models are applied to a multiple answer QA, the answers output by the respective models are scattered.

[0038] Figure 2 is an example of a single answer QA and an example of a multiple answer QA. Figure 2 The upper layer is the relevant text, the middle layer is the answer based on a single answer model, and the lower layer is the answer based on a plurality of (specifically, 10) single answer models. The single answer model extracts "50" as the answer, but the plurality of single answer models extract "100 peso note" and "500 peso notes" as the answers, and the answers are scattered.

[0039] Also, when the degree of agreement of the answers of the respective models in the single / multiple answer QAs is compared, it is known that the degree of agreement of the answers of the single answer QAs tends to be high.

[0040] Figure 3 The degree of agreement of the answers of the single answer QAs and the multiple answer QAs is shown. In Figure 3 In the figure, DROP (single-span: single span) shows a single answer QA, and DROP (multi-span: multi-span) shows a multiple answer QA. In addition, the vertical axis indicates the average number of differences in the answers, and the case where the answers estimated by the respective models are completely identical is set to the number of differences = 1. The respective models are,

[0041] BiDAF: (Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603, 2016.)

[0042] QANet: (Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global

[0043] self-attention for reading comprehension. arXiv preprint arXiv: 1804.09541, 2018.)

[0044] BERT BASE : (Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.)

[0045] BERT LARGE : (Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.)

[0046] The single answer QA model is 10 single answer QA models learned in advance. As shown in Figure 3 Table 1, the average number of different answers of the single answer QA is small, and the consistency of the answers is high. This is due to the constraint that each single answer QA model outputs only one answer that is considered to be the best. Therefore, in a case where multiple answers are dispersed in the context, the answers of each model are often inconsistent.

[0047] In this embodiment, the property that the answers of multiple single answer QA models are not particularly consistent in the multiple answer QA is utilized. Specifically, a model capable of multiple answers is realized by combining multiple models that can only make a single answer.

[0048] The specific model is as follows.

[0049] In SQuAD, which is a typical extractive machine reading comprehension dataset, a most appropriate answer is extracted from a given context for a question as an objective, and it has been reported that more models exceed human reading comprehension ability. Thereafter, in SQuAD2.0 (Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In ACL, 2018.), a question for which an answer does not exist in the context is newly added to answer as "unanswerable". In DROP (Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019.), a question for which extraction of a range of multiple answers (multiple answer QA) that has not been addressed in the past is added. SQuAD2.0 is two patterns of single answer / no answer, but since DROP requires one or more arbitrary number of answers, DROP can be said to be a dataset that requires a more diverse answer pattern.

[0050] Based on the multiple answer models of Hu et al. (Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In EMNLP-IJCNLP, 2019.), it is shown that a model that predicts how many answers there are in the context for a question and repeatedly extracts an answer range based on the predicted number of answers can answer multiple answer QAs of DROP. Efrat et al. (Avia Efrat, Elad Segal, and Mor Shoham. Tag-based multi-span extraction in reading comprehension. arXiv preprint arXiv:1909.13375, 2019.) proposed Multi-span Head that assigns a BIO tag to each token by treating answer extraction as a sequence labeling problem. Here, a "token" is the smallest structural element of a sentence such as a word or a string. These models can handle multiple answers, but they need to be prepared with machine reading data including multiple answer QAs in advance in order to learn.

[0051] To this end, the present embodiment can answer multiple answer QAs by combining multiple simple models that can only handle single answers, using only a dataset of extraction-type machine reading that targets only single answers. In the present embodiment, it can be said that the answers using multiple models that can only handle single answers do not particularly coincide in multiple answer QAs.

[0052] Hereinafter, the present embodiment will be described in more detail.

[0053] <Structure>

[0054] Figure 4 A structural block diagram of the answer generation apparatus in the present embodiment is shown.

[0055] The answer generation apparatus is composed of a computer, and has a processor 10, a ROM 12, a RAM 14, an input section 16, an output section 18, and a model storage section 20.

[0056] The processor 10 reads out a processing program stored in the ROM 12 or other program storage and executes using the RAM 14 as a job memory, thereby implementing machine reading comprehension tasks. The processor 10 extracts, from the associated text, an appropriate single answer or multiple answers to the question, using a learned model stored in the model storage 20, based on the input associated text and the question.

[0057] The input unit 16 is constituted by a keyboard, a communication interface, or the like, and inputs the associated text 22 and the question 24. The associated text 22 and the question 24 are basically text data, but can also be image data. In the case of image data, conversion into text data is performed using an OCR (Optical Character Recognition) technique.

[0058] The output unit 18 is constituted by a display, a communication interface, or the like, and outputs the result of the machine reading comprehension task in the processor 10, that is, the answer extracted from the associated text.

[0059] The model storage 20 stores N different single answer models. The N different single answer models are learned models that are learned in advance using only a data set of single answers QA. The single answer model can use, for example, the publicly known BERT (Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.), but is not limited thereto. The single answer model will be described further later.

[0060] In addition, the processor 10 refers to a broad processor, including a general-purpose processor (such as a CPU: Central Processing Unit, or the like), a dedicated processor (such as a GPU: Graphics Processing Unit, an ASIC: Application Specific Integrated Circuit, an FPGA: Field Programmable Gate Array, a programmable logic device, or the like). Furthermore, the action of the processor can be implemented not only by one processor, but also by a plurality of processors existing in physically separate locations in cooperation.

[0061] Figure 5 The machine reading comprehension processing performed by the processor 10 is functionally illustrated.

[0062] Processor 10 inputs associated text 22 and question 24. Associated text 22 is arbitrary, for example, “In the first quarter, Cincinnati trailed early as Patriots kicker Stephen Gostkowski got a 31-yard field goal, while QB Tom Brady completed a 1-yard TD pass to LBMike Vrabel.(…) QB Carson Palmer completing a 1-yard TD pass to LB MikeVrabel.(…)”.

[0063] Question 24 is also arbitrary, for example,

[0064] "Which two players both completed 1-yard TD pass?".

[0065] Here, the question is not limited to a question with only one correct answer. It should be noted that a correct answer includes questions with multiple answers (in the question above, there are 2 correct answers).

[0066] Machine reading model 26 consists of N different single-response models (M = {m1, ..., m...}). N These single-response models are constructed by pre-learning using only a single-response QA. In the case where the well-known BERT is used as a single-response model, learning is performed to take the question Q and associated text (context) as input and output a probability distribution representing the start and end points of the response. N independent response candidates are obtained from N different single-response models. Each response candidate is a probability distribution representing the start and end points. Figure 5 The image shows the start and end markers (words, strings) output from machine reading model 26, including "Brady", "Tom", "Carson Palmer", "Tom Brady completed", ..., "Palmer". N independent answer candidates from N different single-response models are fed to RC-rewriter 28.

[0067] The RC-rewriter 28 has a single / multiple answer selection unit 30 that selects the final answer from N independent answer candidates. The single / multiple answer selection unit 30 determines whether an object is marked as a single answer if the answer range is concentrated on a specific marker (string, word string) in the associated text (context), or as multiple answers if the answer range is scattered across multiple markers, thereby selecting the answer.Figure 5 In this case, it is shown that the answer range is dispersed in multiple tokens, and multiple answers "Tom Brady" and "Carson Palmer" are output as final answers.

[0068] It is possible to determine whether the answer range is concentrated in a specific token or dispersed in multiple tokens by calculating the consistency of the answer and comparing the consistency value of the answer with a threshold value set in advance.

[0069] Figure 6 The machine reading comprehension model 26 is schematically shown. The machine reading comprehension model 26 is composed of N different single answer models M = {m1, m2, ···, mN}, each of which is learned using only a single answer QA data set. In a BERT-based single answer model, when a question Q and a context C are given, N} are given, the following is calculated.

[0070] x = [CLS] Q [SEP] C [SEP] is used as an input to BERT.

[0071] Here, [CLS] is a token representing the representation of the entire sentence, and [SEP] is a token for explicitly indicating the division of the input pair. Thus, by inputting x to BERT, a distributed representation H of dimension d for a token sequence of length L is obtained.

[0072] H · R d×L .

[0073] In order to extract the answer range, two fully connected layers are used, and the probability distribution for predicting the start point and the end point of the answer is calculated based on the following equation.

[0074] p start = softmax(W s H + b s ),

[0075] p end = softmax(W e H + b e )

[0076] Here, Ws, We are weight matrices, and bs, be are bias terms.

[0077] Finally, the single answer model m i extracts the start point and the end point of the answer am i as tokens for which pstart and pend are the largest.

[0078] In this way, the extraction of the answer is based on the maximum value of the probability distribution, and thus the single answer model m iRegardless of the number of potential answers in context C, only one answer is always extracted. The N different single-answer models of the machine reading model 26 output the N independent answer candidates to the RC-rewriter 28 in the above manner.

[0079] The single / multiple answer selection unit 30 of the RC-rewriter 28 determines that if the answer range is concentrated on a specific marker in the context C, the object is marked as a single answer, and if the answer range is scattered across multiple markers, it becomes multiple answers, thereby selecting the answer.

[0080] Specifically, firstly, in order to calculate the consistency of each answer, for all the tags T = {t1, t2, ...} contained in the context C, the frequency of each tag appearing in the answer candidate a is calculated.

[0081] The mark t is calculated using the following formula. i The frequency fi of appearing in answer candidate a.

[0082]

[0083] By applying this calculation to all the tags in the context C, we obtain a sequence F = {f1, f2, ...} representing the frequency of occurrence of the answer candidate a in terms of tags.

[0084] Next, referring to the frequency F of each labeled candidate answer, and based on two pre-set thresholds, namely threshold u and threshold l, the final answer a is obtained. Here, u > l, u ≥ 0.5 × N.

[0085] More specifically, based on the following operations, select all tags in context C.

[0086] Operation 1: When f is satisfied i In the case of ≧u, t i The decision to mark it as a single answer is appended to the final answer (Single Answer QA).

[0087] Operation 2: When u > f i In the case of ≧l, t i Decide to tag multiple answers and append them to the answer (Multiple Answer QA).

[0088] Operation 3: When l > f i In the case of t i The decision is marked as not an answer.

[0089] Through the above steps 1 to 3, you can obtain a single answer or multiple answers as the final answer.

[0090] For example, when u = 0.9 × N and l = 0.2 × N, the frequency f i A QA is considered to have a single answer if the consistency is ≥0.9×N, meaning a consistency rate of 90% or higher. Furthermore, the frequency f... i 0.9N > f i A QA is considered to have multiple answers if the consistency is ≥0.2N, meaning the consistency is less than 90% but greater than 20%. Furthermore, if the frequency f... i If the agreement is less than 0.2 × N, i.e., the consistency is less than 20%, then it is determined that there is no response.

[0091] In addition, in operation 1 or operation 2, when the same operation is repeated on multiple consecutive tags T, instead of appending each tag to the final answer separately, the tag with the longest sequence is appended to the final answer.

[0092] Furthermore, in the case of t j After performing operation 1, on t j+1 In cases other than operation 1, the label sequence obtained through operation 1 is appended to the final answer as a single answer, thus ending the answer selection process.

[0093] In the single / multiple answer selection unit 30 of the RC-rewriter 28, the answer candidates extracted by the machine reading model 26 are reconstructed based on the frequency of the tag units. Therefore, as Figure 5 As shown, the tokens extracted from "Tom" and "Brady" can be used to construct an answer like "Tom Brady".

[0094] <Processing Flowchart>

[0095] Figure 7 The flowchart of this embodiment is shown. Processes S101 to S103 are model learning processes, and processes S104 to S107 are response generation processes using the learned model.

[0096] First, prepare N single-response models (S101). The number N and types of single-response models are arbitrary; for example, prepare 20 BERT-based single-response models. Furthermore, single-response models can be of the same type or different types.

[0097] Next, for each of the prepared single-answer models, training is performed using only a single-answer QA dataset (S102). That is, training is performed by taking the question and related text (context) as input and outputting a probability distribution representing the start and end points of the answer. The trained model is then stored in the storage unit (S103). After the learning process is complete, the process moves to the answer generation process.

[0098] First, the relevant text and the question are input (S104).

[0099] Next, the completed learning model stored in the storage section, i.e., the machine reading comprehension model 26, is used to output an answer (S105). The N single answer models of the machine reading comprehension model 26 output independent N answer candidates.

[0100] Next, from the N answers output by the machine reading comprehension model 26, a single / multiple answer is selected by the RC-rewriter 28 (S106). That is, based on the following determination operation using the two thresholds u, l, all the tokens t in the context are selected i .

[0101] Operation 1: In the case where f i ≧ u is satisfied, t i is decided as a single answer token and is added to the final answer (single answer QA)

[0102] Operation 2: In the case where u > f i ≧ l is satisfied, t i is decided as a multiple answer token and is added to the answer (multiple answer QA)

[0103] Operation 3: In the case where l > f i is satisfied, t i is decided as a token that is not an answer.

[0104] After the single / multiple answer or no answer is selected by the above method, the selection results are output as the final answer (S107).

[0105] [Embodiment]

[0106] As the data set, the single-span extraction type machine reading comprehension in which the answer in the extraction type machine reading comprehension included in DROP (Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019.) is one, and the multi-span extraction type machine reading comprehension in which the answer is two or more is used as the multiple answer QA (multi-span). In addition, the multiple answer QA of DROP is not used as learning data, and is used only at the time of evaluation. Therefore, only DROP (single-span) is used at the time of learning.

[0107] Table 1 shows the statistical amount of single / multi-answer QA used in the embodiment.

[0108] [Table 1]

[0109]

[0110] As the extraction type machine reading comprehension model of single answer, BiDAF, QA Net , and BERT were used. BERT used BERT BASE and BERT LARGE which were models that were learned in advance.

[0111] In order to be used as the machine reading comprehension model 26, 20 models in which different seeds were set were learned in advance. At the time of learning, optimization based on Adam was performed.

[0112] Table 2 shows the parameters used in each model.

[0113] [Table 2]

[0114]

[0115] In addition, 2 thresholds of the frequency of each token at the time of answer selection in the RC-rewriter 28 were set to u = 0.9N, 1 = 0.2N. In addition, N corresponds to the number of models used in the machine reading comprehension model 26, and in the embodiment, N = 20.

[0116] <Comparison Example>

[0117] On the other hand, as a comparison example, a single answer model that was learned in the same conditions as the models prepared as the machine reading comprehension model 26 was prepared as a baseline. In addition, as an upper limit value of the performance in the single answer model, Oracle single was prepared. In Oracle single, one answer in which the value of F1 was the highest was extracted from a plurality of true answers given to each question.

[0118] Following Dua et al., EM (Exact Match) and F1 (macro-averaged) for evaluation were used in which DROP was changed. In EM, all of the answers included in the correct answer and the answer predicted by the model are completely identical, so that the evaluation value becomes 1. In addition, F1 takes alignment in the correct answer and the predicted answer, and normalizes after F1 is calculated for all combinations, so that the evaluation value is maximized by obtaining a plurality of answers without more or less.

[0119] Table 3 shows the implementation results of DROP multi-span QA (multi-span).

[0120] [Table 3]

[0121]

[0122] As shown in Table 3, it is confirmed that Fl is improved by about 10 points in all models after comparing the single answer model to the baseline and the method using the RC-rewriter 28.

[0123] Further, since the single answer model extracts only one answer, EM = 0 in all models including Oracle single. On the other hand, it is confirmed that the method of the embodiment can accurately extract all answers in the context in multiple answers QA of up to 5.61%.

[0124] As described above, in the present embodiment, the performance of multiple answers QA can be greatly improved regardless of a specific single answer model.

[0125] <Variant Example 1>

[0126] The number and kind of the single answer models used as the machine reading comprehension model 26 in the present embodiment can be arbitrarily set.

[0127] The present applicant confirmed that the performance can be improved by increasing the number of models compared to the condition in which the number of single answer models is set to be the minimum (N = 1). At the same time, it was also confirmed that the performance tends to decrease when the number of models is excessively increased. Therefore, although it is necessary to prepare a certain number of single answer models in order to solve multiple answers QA, it is not expected that the performance will be improved by increasing the number of models without limit, and it is preferable to set an appropriate number. The number is usually the number of desired answers or more, and can be set to be the upper limit number or less according to the number of desired answers.

[0128] Further, in the present embodiment, it is desirable that the single answer models are not the same kind, but single answer models of different kinds are combined.

[0129] Table 4 shows the comparison results of the case in which different single answer models are combined and the case other than this.

[0130] [Table 4]

[0131]

[0132] In Table 4, in order to unify the number of single answer models between the compared models, the number of each model of the combination model was set to 5, and a total of 20 single answer models were used. In multi-span, in the case where the combination model was compared with the same kind of single answer model, it was confirmed that the performance was improved by 0.54 points in EM and by 2.9 points in F1.

[0133] According to the above, with respect to the single answer models used as the machine reading interpretation model 26, it can be said that in improving the correct answer rate of multiple answer QAs, it is desirable not to use only the single answer model with the highest performance, but to combine single answer models of multiple kinds.

[0134] <Variation 2>

[0135] In the present embodiment, as the analysis target data, text data such as related texts was adopted, but it is also possible to apply to other data forms such as image data. In the case where the analysis target data is image data, it is possible to use pixels as tokens.

Claims

1. An answer generation device, wherein the answer generation device comprises: an input section that inputs a question and data that is an analysis target and becomes an analysis target, an analysis target data; a processor; and an output section, the processor performs the following processing by executing a program: in a case where the question and the analysis target data are input, using a plurality of different single answer models prepared in advance that respectively extract a single answer to the question from the analysis target data, a plurality of answer candidates to the question are independently extracted from the analysis target data, for all tokens included in the analysis target data, a frequency of occurrence of each token in the plurality of answer candidates independently extracted from the analysis target data by the plurality of different single answer models is calculated, in a determination processing, using a first threshold value and a second threshold value smaller than the first threshold value, in a case where the frequency of occurrence of a token that is a determination target is the first threshold value or more, the token whose frequency of occurrence is the first threshold value or more is determined to be a token of a single answer and selected, in a case where the frequency of occurrence of a token that is a determination target is less than the first threshold value and is the second threshold value or more, the token whose frequency of occurrence is less than the first threshold value and is the second threshold value or more is determined to be a token of a plurality of answers and selected, in a case where the frequency of occurrence of a token that is a determination target is less than the second threshold value, the token is not selected as a token of an answer, and after the determination processing ends, a single or a plurality of answers composed of tokens selected in the determination processing are output to the output section. 2.The answer generation device according to claim 1, wherein the processor, in a case where determination results for a plurality of tokens that are determination targets in succession are the same and tokens are selected according to the determination results, instead of selecting each token respectively, selects tokens in succession whose sequence length is the largest. 3.The answer generation device according to claim 1, wherein the processor, in a case where, after a token that is a determination target is determined to be a token of a single answer, a next token that is a determination target is determined to be a token of a plurality of answers or is not selected as a token of an answer, selects only the token determined to be the token of the single answer and ends the determination processing. 4.The answer generation device according to any one of claims 1 to 3, wherein the processor uses the plurality of different single answer models in a number of which is more than the number of answers of the plurality of answers. 5.The answer generation device according to any one of claims 1 to 3, wherein the plurality of different single answer models are respectively models that, based on a deep learning method, encode the question and the analysis target data into distributed representations, and based on the distributed representations after encoding, extract a range of answers with the highest probability from the analysis target data by a fully connected layer for estimating a start point and an end point of the answers. 6.The answer generation device according to any one of claims 1 to 3, wherein The plurality of different single answer models are each learned using learning data that gives only a single answer to a question.

7. The answer generation apparatus according to any one of claims 1 to 3, wherein The analysis target data is text data, The token is a word or a character string in the text data.

8. A computer-readable medium storing a program that causes a computer to execute a process, wherein The process has the steps of: inputting a question and data that is a target of analysis, that is, analysis target data; extracting, from the analysis target data, a plurality of answer candidates for the question independently using a plurality of different single answer models that are prepared in advance and that each extract a single answer for the question from the analysis target data; calculating, for each token included in the analysis target data, a frequency of occurrence of the token in the plurality of answer candidates extracted independently from the analysis target data by the plurality of different single answer models, and in a determination process, using a first threshold value and a second threshold value that is smaller than the first threshold value, in a case where the frequency of occurrence of a token that is a target of determination is equal to or higher than the first threshold value, determining that the token whose frequency of occurrence is equal to or higher than the first threshold value is a token of a single answer and selecting the token, in a case where the frequency of occurrence of a token that is a target of determination is lower than the first threshold value and equal to or higher than the second threshold value, determining that the token whose frequency of occurrence is lower than the first threshold value and equal to or higher than the second threshold value is a token of a plurality of answers and selecting the token, and in a case where the frequency of occurrence of a token that is a target of determination is lower than the second threshold value, not selecting the token as a token of an answer; and after the determination process ends, outputting a single or a plurality of answers composed of tokens selected in the determination process.

9. An answer generation method, wherein The answer generation method has the steps of: inputting a question and data that is a target of analysis, that is, analysis target data; extracting, from the analysis target data, a plurality of answer candidates for the question independently using a plurality of different single answer models that are prepared in advance and that each extract a single answer for the question from the analysis target data; The occurrence frequency of each token in the answer candidates independently extracted from the analysis target data by the plurality of different single answer models is calculated for all tokens included in the analysis target data, and in the determination process, a first threshold value and a second threshold value smaller than the first threshold value are used, in a case where the occurrence frequency of a token that is a determination target is the first threshold value or more, the token whose occurrence frequency is the first threshold value or more is determined to be a single answer token and selected, in a case where the occurrence frequency of a token that is a determination target is less than the first threshold value and is the second threshold value or more, the token whose occurrence frequency is less than the first threshold value and is the second threshold value or more is determined to be a multiple answer token and selected, and in a case where the occurrence frequency of a token that is a determination target is less than the second threshold value, the token is not selected as an answer token; and After the determination process ends, a single or multiple answers composed of the tokens selected in the determination process are output.

10. A computer program product including a program that causes a computer to execute a process, wherein the process has the following steps: inputting a question and data that is a target of analysis, that is, analysis target data; extracting answer candidates for the question from the analysis target data independently using a plurality of different single answer models prepared in advance, wherein the plurality of different single answer models extract a single answer for the question from the analysis target data; calculating the occurrence frequency of each token in the answer candidates independently extracted from the analysis target data by the plurality of different single answer models for all tokens included in the analysis target data, and in the determination process, a first threshold value and a second threshold value smaller than the first threshold value are used, in a case where the occurrence frequency of a token that is a determination target is the first threshold value or more, the token whose occurrence frequency is the first threshold value or more is determined to be a single answer token and selected, in a case where the occurrence frequency of a token that is a determination target is less than the first threshold value and is the second threshold value or more, the token whose occurrence frequency is less than the first threshold value and is the second threshold value or more is determined to be a multiple answer token and selected, and in a case where the occurrence frequency of a token that is a determination target is less than the second threshold value, the token is not selected as an answer token; and After the determination process ends, a single or multiple answers composed of the tokens selected in the determination process are output.

10. A computer program product including a program that causes a computer to execute a process, wherein the process has the following steps: inputting a question and data that is a target of analysis, that is, analysis target data; extracting answer candidates for the question from the analysis target data independently using a plurality of different single answer models prepared in advance, wherein the plurality of different single answer models extract a single answer for the question from the analysis target data; calculating the occurrence frequency of each token in the answer candidates independently extracted from the analysis target data by the plurality of different single answer models for all tokens included in the analysis target data, and in the determination process, a first threshold value and a second threshold value smaller than the first threshold value are used, in a case where the occurrence frequency of a token that is a determination target is the first threshold value or more, the token whose occurrence frequency is the first threshold value or more is determined to be a single answer token and selected, in a case where the occurrence frequency of a token that is a determination target is less than the first threshold value and is the second threshold value or more, the token whose occurrence frequency is less than the first threshold value and is the second threshold value or more is determined to be a multiple answer token and selected, and in a case where the occurrence frequency of a token that is a determination target is less than the second threshold value, the token is not selected as an answer token; and After the determination process ends, a single or multiple answers composed of the tokens selected in the determination process are output.

Images