Method and apparatus for generating question and answer dataset based on input paragraph

Abstract

A method for generating a question-and-answer dataset based on input of a paragraph containing a plurality of sentences according to an embodiment of the present invention comprises: a step of generating a machine-readable data representation from the input paragraph through a neural network and storing it in memory; a step of performing sampling on the data representation; a step of calculating the similarity of a data representation associated with the query in the data representation in memory using the sampled data as a query; a step of generating a response based on the similarity; and a step of generating a query based on the response and the similarity.

Description

Technology Field

[0001] The present invention discloses a method and apparatus capable of generating various data sets by simultaneously generating and sampling query and answer pairs that may occur in a single paragraph. Background Technology

[0002] Due to recent technological advancements, various question-answering systems utilizing deep learning have been developed. Recently, systems have been demonstrated where users submit natural language queries to the system, and a QA model is used to automatically provide a response appropriate to the query. However, since these deep learning-based QA models ultimately require a large amount of training data, they were limited to automatically providing responses for specific fields or topics. Furthermore, manually building such training data presented the problem of requiring a significant amount of time and cost. Prior art literature

[0003] Registered Patent Publication No. 10-1605430, March 22, 2016. The problem to be solved

[0004] The problem that the present invention aims to solve is to provide a question-answer generation model that simultaneously generates a question-answer dataset based on input paragraphs even without a given set of response candidates, unlike existing question-answer (QA) models where the user had to manually collect and train the combined distribution of question-answer pairs, and a method and apparatus for generating a question-answer learning model that can be automatically trained using the same. means of solving the problem

[0005] A method for generating a question-and-answer dataset based on input of a paragraph containing a plurality of sentences according to an embodiment of the present invention comprises: a step of generating a machine-readable data representation from the input paragraph through a neural network and storing it in memory; a step of performing sampling on the data representation; a step of calculating the similarity of a data representation associated with the query in the data representation in memory using the sampled data as a query; a step of generating a response based on the similarity; and a step of generating a query based on the response and the similarity. Effects of the invention

[0006] According to one embodiment, the present invention automatically generates a dataset of questions and answers from an input paragraph, so multiple pairs of questions and answers can be sampled with only one paragraph input, thereby enabling the construction of more diverse data than previous models.

[0007] In addition, various query and response datasets can be constructed from data sets provided in the form of information boxes or tables in a standardized format on existing websites.

[0008] Furthermore, since the method for generating a question and answer dataset according to one embodiment extracts a question and answer dataset from data where training or machine translation is completed, it can be used to verify the quality of training or translation by evaluating the accuracy of the response corresponding to the question.

[0009] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0010] FIG. 1 is a conceptual diagram for schematically explaining a question-and-answer dataset generation model based on paragraph input according to one embodiment. FIG. 2 is a flowchart illustrating a method for generating a question-and-answer dataset based on paragraph input according to one embodiment. FIG. 3 is a figure illustrating an example of a method for generating a question-and-answer dataset in a paragraph according to one embodiment. FIG. 4 is a diagram illustrating a method for generating a question-and-answer dataset in a structured infobox according to one embodiment. FIG. 5 is a diagram illustrating a method for generating a question-and-answer dataset from an unstructured page according to one embodiment. FIG. 6 is a diagram illustrating a method for improving the performance of a question-answering model through adversarial learning according to one embodiment. FIG. 7 is a block diagram schematically showing the internal configuration of a question-and-answer dataset generation device according to one embodiment. Specific details for implementing the invention

[0011] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the present invention, and the present invention is defined only by the scope of the claims.

[0012] The terms used in this specification are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" as used in this specification do not exclude the presence or addition of one or more other components in addition to the components mentioned. Throughout the specification, the same reference numerals refer to the same components, and "and / or" includes each of the mentioned components and all combinations of one or more. Although terms such as "first," "second," etc., are used to describe various components, these components are not limited by these terms. These terms are used merely to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the technical scope of the invention.

[0013] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0014] Spatially relative terms such as "below," "beneath," "lower," "above," and "upper" may be used to facilitate the description of the relationship between one component and other components as illustrated in the drawings. Spatially relative terms should be understood as encompassing different orientations of components during use or operation, in addition to the orientations depicted in the drawings. For example, if a component depicted in a drawing is inverted, a component described as "below" or "beneath" of another component may be placed "above" of that component. Therefore, the exemplary term "below" may encompass both the lower and upper directions. Components may also be oriented in other directions, and accordingly, spatially relative terms may be interpreted according to the orientation.

[0016] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0017] FIG. 1 is a conceptual diagram for schematically explaining a question-and-answer dataset generation model based on paragraph input according to one embodiment.

[0018] Referring to FIG. 1, a question-and-answer dataset generation model according to one embodiment may include a memory (110), an encoder (120), an attention (130), a response decoder (140), and a question decoder (150).

[0019] According to a method for generating a question-and-answer dataset according to one embodiment, a passage (passage, 101) containing multiple sentences is input directly into an encoder (120). Here, the passage (101) may include phrases, sections, paragraphs, etc.

[0020] The encoder (120) generates a hidden representation for each word in the input paragraph (101) and stores it in memory (110). Here, the encoder (120) can construct the hidden representation from the input through a neural network. Thus, the hidden representation may refer to a machine-readable data representation learned in a hidden layer of the neural network and may consist of two pairs of a key and a value. For example, the encoder (120) can store information regarding the meaning represented by a component of the sentence (e.g., a word) as a hidden representation.

[0021] Next, the encoder (120) performs sampling (102) based on the hidden representation to generate a latent variable (103), and the generated latent variable (103) is input to the response decoder (140).

[0022] The response decoder (140) generates a weighted sum vector of memory (110) by calculating the importance of each component of memory (110) through attention (130) using the latent variable (103) as a query. For example, at every time step in which the response decoder (140) predicts an output word through attention (130), the entire input sentence from the encoder (120) can be referenced once again. However, the entire input sentence is not referenced at the same rate, but rather the parts of the input word associated with the word to be predicted at that time are referenced with greater attention. Accordingly, the attention function calculates the similarity between the given 'Query' and the 'Key' obtained from all hidden expressions. Then, this calculated similarity is reflected in each 'Value' mapped to the key. And, a weighted sum vector can be obtained by adding up all the 'values' that reflect similarity.

[0023] The response decoder (140) predicts the response span (104, answer span) based on the weighted sum vector. Here, the response span may refer to the starting and ending points where the response is located in the input paragraph.

[0024] Next, the query decoder (150) receives the previously generated response span (104) and performs attention (130) calculations in memory (110) to finally generate a query (105).

[0025] A question-and-answer dataset consisting of pairs of questions and answers generated according to one embodiment may be stored in memory (110), and the stored question-and-answer dataset may be used to machine learn a question-and-answer learning model.

[0027] Therefore, since the dataset generation model according to one embodiment automatically generates a dataset of questions and answers from an input paragraph, multiple pairs of questions and answers can be sampled with only one paragraph input, allowing for the construction of more diverse data than previous manual input models.

[0029] Hereinafter, a method for generating a question-and-answer dataset based on paragraph input according to one embodiment will be described in more detail with reference to FIG. 2.

[0030] FIG. 2 is a flowchart illustrating a method for generating a question-and-answer dataset based on paragraph input according to one embodiment.

[0031] In step S200, a method for generating a question-and-answer dataset according to one embodiment generates a hidden representation for each word in an input paragraph and stores it in memory. Here, the input paragraph may be structured infobox data extracted from data crawled from a website. Alternatively, if infobox data is not extracted from the crawled data, a structured paragraph based on data frequency in a table recognized through column recognition may be input as the paragraph. Furthermore, if the input paragraph is data that has been machine translated, the quality of the machine translation may be evaluated by evaluating the accuracy of the response based on the generated question-and-answer dataset. Specific input paragraphs will be described later with reference to FIGS. 3 and 4.

[0032] In step S210, a method for generating a question-and-answer dataset according to one embodiment samples a latent variable based on a hidden representation.

[0033] In step S220, a method for generating a question-and-answer dataset according to one embodiment generates a weighted sum vector by calculating the importance of a hidden representation stored in memory through attention that uses a sampled latent variable as a query.

[0034] In step S230, a method for generating a question-and-answer dataset according to one embodiment generates a response span based on a latent variable and the weighted sum vector.

[0035] In step S240, a method for generating a question-answer dataset according to one embodiment generates a query based on the response span and attention to the hidden representation in the memory. The generated query and response dataset can be stored in memory. Furthermore, a question-answer model can be machine trained using the stored query and response dataset. Additionally, adversarial queries for adversarial machine learning are automatically generated from the input paragraph, thereby allowing the question-answer model to be machine trained to be closer to real-world boundaries.

[0036] More specifically, the above-described method for generating a question-and-answer dataset can be implemented through the objective function shown in the following mathematical formula 1.

[0037]

[0038] Here, x, y, c, and z correspond to the question, answer, passage, and latent variable, respectively. is the encoder of the question-answering learning model, These are functions used in the response decoder and query decoder, respectively. is the marginal prior distribution of the latent variable, and is the marginal posterior distribution of the latent variable. For example, It generates a distribution of latent variables generated from the input, and It is possible to generate an output close to the input using latent variables. Meanwhile, since it is very difficult to calculate the KL (Kullback-Leibler) divergence for optimization (to reduce the difference between the two distributions mentioned above), it can be approximated as shown in Equation 2 below.

[0039]

[0040] Meanwhile, with reference to FIGS. 3 and 4, it will be explained how paragraph input is performed and how a question-and-answer dataset construction model according to one embodiment can be utilized.

[0041] FIG. 3 is a figure illustrating an example of a method for generating a question-and-answer dataset in a paragraph according to one embodiment.

[0042] Referring to FIG. 3, when a paragraph (301) according to one embodiment is input, a response (303) is first generated through sampling (302). Then, a query (304) paired with the response can be generated using the generated response (303).

[0043] For example, if the input paragraph contains the sentence "Yi Sun-sin (April 28, 1545 – December 16, 1598 (Lunar November 19)) is a military official of the mid-Joseon Dynasty," a response "April 28, 1545" can be generated first through sampling. Then, based on the response generated according to the method of generating a question-answer dataset according to one embodiment, various forms of queries such as "What is the birthday of Admiral Yi Sun-sin?" or "When was Yi Sun-sin born?" can be generated. Therefore, by using a question-answer generation model, various datasets can be constructed for machine learning the question-answer model.

[0045] FIG. 4 is a diagram illustrating a method for generating a question-and-answer dataset in a structured infobox according to one embodiment.

[0046] Referring to FIG. 4, a question-and-answer dataset generation model according to one embodiment can obtain infobox data (400) through crawling from a site famous as an online knowledge encyclopedia site.

[0047] In particular, since the infobox data (500) is structured, it can be easily structured into a corpus of questions and answers. In addition, since these knowledge encyclopedia sites provide detailed classification data for fields or subjects, it is possible to generate questions and answers targeting a desired field and generate input paragraphs in the preprocessing stage for machine learning.

[0048] However, since not all websites provide structured infobox data (500) as illustrated in FIG. 4, the crawled data is generally unstructured. Therefore, we will examine how to generate the paragraph to be entered in such cases.

[0049] FIG. 5 is a diagram illustrating a method for generating a question-and-answer dataset from an unstructured page according to one embodiment.

[0050] Referring to Fig. 5, in step S500, the question-and-answer dataset construction model recognizes the title and body of a webpage from the crawled data. Then, in step S510, it recognizes a table based on column recognition in the body. That is, it recognizes a table to obtain data in a form similar to an infobox obtainable from a structured site.

[0051] Next, paragraphs are formatted by checking for words that satisfy the preset data frequency in step S520. For example, words that appear repeatedly, such as "country name" or "capital," are identified, and paragraphs can be formatted based on this.

[0052] In step S530, the question and answer dataset generation model according to one embodiment receives a structured paragraph as input and generates a question and answer dataset, thereby enabling the construction of various question and answer datasets from data sets provided in the form of information boxes or tables provided in a structured form on existing websites.

[0054] The above-described question-and-answer dataset generation model according to another embodiment may go beyond simply generating a question-and-answer dataset and extract a question and answer dataset from data where training or machine translation is completed, evaluate the accuracy of the response corresponding to the question, and use the evaluation results to verify the quality of the training or translation.

[0055] For example, if a question-and-answer dataset generation model according to one embodiment can generate an accurate question-and-answer dataset from data where machine translation has been completed, it can be evaluated that machine reading comprehension is working well.

[0057] Meanwhile, recent question-answering learning models already have significantly high performance because they perform pre-training using language model objectives on large corpora. Therefore, simply using more data for training does not lead to a linear, dramatic improvement in the performance of the question-answering learning model. In particular, since most of the question-correct pairs automatically generated through the question-answering dataset generation model according to one embodiment are already correct for the question-answering learning model, they may not be of much help, so a method is needed to improve machine learning performance.

[0058] Accordingly, a question-and-answer dataset generation model according to another embodiment discloses a method for generating a question-and-answer dataset that a current question-and-answer learning model cannot match through adversarial learning.

[0059] FIG. 6 is a diagram illustrating a method for improving the performance of a question-answering model through adversarial learning according to one embodiment.

[0060] Referring to FIG. 6, in order to perform adversarial learning, data located in a data manifold that is outside the decision boundary of the current question-answering model must be sampled. Specifically, a method for generating a question-answering dataset based on paragraph input according to one embodiment selects latent variables through an encoder, and the question-answering data pairs generated from the sampled latent variables must be those that the question-answering learning model cannot correctly answer. Then, the question-answering learning model learns the question-answering pairs that did not correctly answer to correct the decision boundary, and by repeating this process, the decision boundary of the question-answering learning model approaches the true decision boundary, and performance can also be improved.

[0061] Meanwhile, referring to Fig. 6, the question-answer dataset for such adversarial learning can be determined by adjusting the loss of the question-answer learning model and the constraints for positioning it on the data manifold.

[0062] For example, the aforementioned adversarial question-answering learning model may have the objective function of the following mathematical equation 3.

[0063]

[0064] Here, x, y, c, and z correspond to the question, answer, passage, and latent variable, respectively. is the preprocessor of the question-answering learning model, is an agent, These are functions used in the response decoder and query decoder, respectively.

[0065] Meanwhile, the encoder is a latent variable It samples, and the questions and answers generated from this are the loss of the question-answering model ( ) must be increased. However, because there may be no correlation at all between the questions generated in this way and the correct answers, Constraints are required in selecting . That is, the previously learned Since invalid question-correct answer pairs are generated as the distance from the distribution increases, the value can be adjusted to prevent this by applying a penalty when the distance between the two distributions increases (e.g., using KL divergence).

[0066] Accordingly, a question-answer learning model according to one embodiment controls the loss and constraints to generate a latent variable that is valid but for which the question-answer learning model fails to guess the correct answer.

[0067] In addition, the performance of a question-answer learning model can be improved by performing adversarial machine learning using a dataset of adversarial questions and answers generated using the generated latent variables.

[0068] FIG. 7 is a block diagram schematically showing the internal configuration of a question-and-answer dataset generation device according to one embodiment.

[0069] Referring to FIG. 7, a question-and-answer dataset generation device (100) according to one embodiment may include one or more memories (110) and a processor (190). The operation of the question-and-answer dataset generation device (100) may be performed by executing a program stored in the memory (110) through the processor (190).

[0070] The operation performed in the processor (190) according to one embodiment performs the operation of generating a hidden representation for each word in the input paragraph and storing it in memory (110).

[0071] Additionally, the processor (190) samples a latent variable based on a latent representation and generates a weighted sum vector by calculating the importance of the latent representation stored in the memory through an attention that uses the sampled latent variable as a query.

[0072] Additionally, the processor (190) can perform operations to generate a response span based on the latent variable and the weighted sum vector, and to generate a query based on the generated response span and attention to the hidden representation of memory.

[0073] Additionally, the processor (190) may further perform operations to store a data set of generated questions and answers in memory (110) and to machine learn a question-answer learning model using the stored data set of questions and answers.

[0074] Additionally, the processor (19) may further perform operations to generate a valid latent variable that the question-answer learning model fails to match by adjusting the loss and constraints of the question-answer learning model, and to perform adversarial machine learning using the generated latent variable and the generated question-answer dataset.

[0075] Meanwhile, the dataset generation device (100) described above may include one or more processors (190) and / or one or more memories (110). Additionally, the processor (190) may include an encoder (120), a response decoder (140), and a query decoder (150), and may perform the operations described above in FIGS. 1 to 7 with respect to each encoder (120), response decoder (150), and query decoder (150).

[0077] Additionally, memory (110) may include volatile and / or non-volatile memory. One or more memories (110) may store instructions that cause one or more processors (190) to perform operations when executed by one or more processors (190). In the present disclosure, a program or instruction is software stored in memory (110) and may include an operating system for controlling the resources of a server (100), an application, and / or middleware that provides various functions to the application so that the application can utilize the resources of the device.

[0078] One or more processors (190) can control at least one component of a device (100) connected to the processor (190) by running software (e.g., a program, a command). Additionally, the processor (190) can perform various operations such as computation, processing, data generation, and processing related to the present disclosure. Additionally, the processor (190) can load data, etc. from memory (110) or store it in memory (110).

[0079] In one embodiment, at least one of the components of the device (100) may be omitted, or another component may be added. Additionally, some components may be implemented by integrating them, or implemented as a single or multiple entities, either additionally or alternatively.

[0081] Meanwhile, the steps of the method or algorithm described in connection with the embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, a hard disk, a removable disk, a CD-ROM, or any form of computer extractable recording medium well known in the art to which the present invention belongs.

[0082] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0084] 100 : Question-and-Answer dataset generator 110 : Memory 120 : Encoder 130 : Attention 140 : Response decoder 150 : Query Decoder

Claims

Claim 1 A method for generating a question-and-answer dataset based on input of a paragraph containing multiple sentences, comprising: a step of generating a machine-readable data representation for each word of the input paragraph through a neural network and storing it in memory—the data representation includes information regarding the meaning represented by each word—; a step of sampling the latent variable from the data representation by referring to constraints associated with the distribution of the latent variable; a step of calculating the similarity of the data representation in the memory associated with the query using the sampled latent variable as a query; a step of generating a response based on the similarity; a step of generating multiple queries of various forms associated with the response based on the response and the similarity; and a step of storing a question-and-answer dataset consisting of pairs of each of the multiple queries and the response. Claim 2 A method according to claim 1, further comprising the step of machine learning a question-answer learning model using the question-answer dataset. Claim 3 The method of claim 2 further comprises the step of generating a latent variable that is valid but for which the question-answer learning model fails to guess the correct answer by adjusting the loss and constraints of the question-answer learning model; and the step of performing adversarial machine learning using the question-answer dataset including pairs of a question and an answer generated using the generated latent variable. Claim 4 A method according to paragraph 3, wherein the generated latent variable is penalized as it deviates from the distribution of the previously learned latent variable. Claim 5 A method according to claim 1, wherein the input paragraph is structured infobox data extracted from data crawled from a website. Claim 6 A method according to claim 5, wherein if infobox data is not extracted from the crawled data, a structured paragraph based on data frequency from a table recognized through column recognition is input as the paragraph. Claim 7 A method according to claim 1, wherein the input paragraph is machine-translated data, and further comprises the step of evaluating the quality of the machine translation by evaluating the accuracy of the response based on the question-and-answer dataset. Claim 8 A question-response dataset generation device comprising: one or more processors; and one or more memories in which instructions are stored to cause the one or more processors to perform operations when executed by the one or more processors, wherein the operations performed by the one or more processors include: an operation in which, when a paragraph containing multiple sentences is input, a machine-readable data representation is generated through a neural network for each word of the input paragraph and stored in the memory—the data representation includes information regarding the meaning represented by each word—; an operation in which the latent variable is sampled from the data representation by referring to constraints associated with the distribution of the latent variable; an operation in which the similarity of the data representation in the memory associated with the query is calculated using the sampled latent variable as a query; an operation in which a response is generated based on the similarity; an operation in which multiple queries of various forms related to the response are generated based on the response and the similarity; and an operation in which a question-response dataset consisting of pairs of each of the multiple queries and the response is stored. Claim 9 A question-and-answer dataset generating device according to claim 8, wherein the operation performed in the processor further includes an operation of machine learning a question-and-answer learning model using the question-and-answer dataset. Claim 10 A question-answer dataset generation device according to claim 9, wherein the operation performed in the processor further comprises: an operation to generate a latent variable that is valid but for which the question-answer learning model fails to guess the correct answer by adjusting the loss and constraints of the question-answer learning model; and an operation to perform adversarial machine learning using the question-answer dataset including pairs of a question and an answer generated using the generated latent variable. Claim 11 A question-and-answer dataset generation device according to claim 10, wherein the generated latent variable is penalized as it deviates from the distribution of the previously learned latent variable. Claim 12 A computer-readable recording medium having a program for executing any one of the methods of paragraphs 1 through 7.

Images