Text classifier and background knowledge representation generator for identifying answers, and training device and computer program

Abstract

To enable the specification of answer candidates for a question with high accuracy by effectively utilizing a background knowledge to the question in order to extract the answer candidates for the question.SOLUTION: An answer specifying text classifier 90 includes a BERT 102 for accepting a question and answer candidates as an input 100, a knowledge integrating transformer 104 for accepting an output of the BERT 102 as an input, a background knowledge representation generator 114 for accepting the question and an answer as the input 100, and outputting a representation vector 116 of a background knowledge to the question, and a vector conversion unit 112 for performing embedded vectorization of the question and the answer candidates and inputting them to the background knowledge representation generator. The knowledge integrating transformer 104 accepts the representation vector 116 of a background knowledge as attention, and outputs a label 102 showing whether or not the answer candidates include a correct answer to the question.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a question answering system, and more particularly to a question answering system that extracts passages containing answers from a text archive to generate answers to questions input in a natural language, and relates to a text classifier for identifying answers that classifies passages into those containing correct answers and those that do not, a background knowledge representation generator used in the text classifier for identifying answers, a training device for the same, and a computer program for training the background knowledge representation generator. [Background technology]

[0002] A why-type question-answering system using natural language processing is disclosed in Patent Document 1, which will be mentioned later. Unlike what-type question-answering systems, why-type question-answering systems often extract from a text archive multiple passages (hereinafter referred to as "passage groups"; each passage contains 5 to 7 sentences) consisting of a collection of sentences that are likely to be the answer, and select from among them the one that is most suitable as the answer to the question. Here, a passage refers to multiple consecutive sentences in a text. The why-type question-answering system described in Patent Document 1 is one such system, but as a prerequisite, it uses an answer candidate search system that, when a question is given, extracts from a text archive multiple passages that are likely to contain the correct answer to the question and outputs them.

[0003] The system described in Patent Document 1 collects and stores a large amount of text available on the web in advance. From this web archive, expressions that are considered to express causal relationships are extracted. The system disclosed in Patent Document 1 recognizes causal relationships in answer passages using clue words such as "because" or specific causal relationship patterns such as "A causes B to occur."

[0004] The why-type question answering system disclosed in Patent Document 1 further extracts features from each answer passage, the extracted causal knowledge, and the question to determine whether the passage is suitable as an answer to the question. Then, the answer passage and the extracted features are input to a pre-trained convolutional neural network (hereinafter referred to as "CNN (Convolutional Neural Network)"). The CNN outputs a likelihood (the probability that the question is a question about the answer passage) indicating whether the answer passage is suitable as an answer to the question as a score. The why-type question answering system ranks the answer passages based on the score calculated for each answer passage in this way, and outputs the answer passage with the highest score as the answer to the question. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-49681 A [Patent Document 2] Special Publication No. 2020-506466 Summary of the Invention [Problem to be solved by the invention]

[0006] In the case of a why-type question, the answer is likely to be the cause part of a causal expression that has the question as its result part. In Patent Document 1, the most appropriate answer passage can be extracted based on the causal expression from a group of answer passages that are answer candidates. Therefore, Patent Document 1 claims that it is possible to select a more appropriate answer to a why-type question than in the past.

[0007] However, in many cases, including the invention disclosed in Patent Document 1, there is still a problem that it is difficult to correctly determine with a high probability whether each passage is an answer to a question. In the case of a human being, when extracting an answer to a question from similarly extracted passages in question answering, the human can utilize background knowledge to accurately extract whether the passage contains the correct answer and the part that will be the correct answer. Even in a question answering system using natural language processing, if such background knowledge can be efficiently utilized, it is believed that the question answering system can provide answers to questions with higher accuracy.

[0008] Until now, such background knowledge was thought to simply refer to passages that are thought to be highly relevant to the question, and it was difficult to say that background knowledge was effectively used when actually identifying the answer. In order to improve the accuracy of answers from question answering systems, it is desirable to use background knowledge to be able to determine with high accuracy whether a passage contains the correct answer or not, and to be able to accurately identify parts that are thought to be particularly relevant to the question. To achieve this, it is also important to consider how to represent background knowledge and how to use it to identify answers.

[0009] Therefore, an object of the present invention is to provide a text classifier for identifying answers that can effectively utilize background knowledge related to a question in order to extract answer candidates for the question and identify answer candidates to the question with high accuracy, a background knowledge representation generator therefor, a training device for the same, and a computer program. [Means for solving the problem]

[0010] A text classifier for identifying an answer according to a first aspect of the present invention includes a language representation model that receives as input a question text and answer candidate text, a knowledge integration transformer that receives as input an output of the language representation model, and a background knowledge representation generator that receives as input the question text and the answer candidate text and outputs a representation vector of background knowledge for the question text, wherein the knowledge integration transformer is configured to receive the representation vector of the background knowledge as attention, and outputs a label indicating whether the answer candidate text includes an answer to the question text.

[0011] Preferably, the knowledge integration transformer includes a plurality of knowledge integration transformer layers, and the background knowledge representation generator outputs a plurality of background knowledge representation vectors corresponding to the plurality of knowledge integration transformer layers, respectively, and the plurality of knowledge integration transformer layers include: Among multiple background knowledge representation vectors, The representation vector corresponding to each knowledge integration transformer layer is received as the information source for attention.

[0012] More preferably, the background knowledge representation generator includes a background knowledge representation generator layer that outputs a representation vector of background knowledge in response to an input vector representing a question and an answer candidate, and an update unit that updates the input vector to the background knowledge representation generator layer using the representation vector of background knowledge output by the background knowledge representation generator layer to set it as a next input vector to the background knowledge representation generator layer, and the update unit updates a preceding input vector to the background knowledge representation generator using the representation vector of background knowledge output by the background knowledge representation generator in response to the preceding input vector. of The expression vector is updated using the degree of association between the expression vector and the preceding input vector.

[0013] A computer program according to a second aspect of the present invention causes a computer to function as any one of the above-described text classifiers for identifying an answer.

[0014] A training device according to a third aspect of the present invention is a training device that uses a plurality of training data items to train a background knowledge representation generator that outputs a vector representation of background knowledge for a question represented by a question text, the plurality of training data items including a question text and background knowledge text related to the question text, the training device including: a genuine expression generator made of a neural network that, when the question text and the background knowledge text are given, generates a genuine expression vector of the same format as the vector representation of background knowledge represented by the background knowledge text; a fake expression generator made of a neural network that, when the question text and a random noise vector representing a vector representation for an arbitrary text are given, outputs a fake expression vector of the same format as the vector representation related to the background knowledge generated from the noise vector; a classifier made of a neural network for classifying the genuine expression vector and the fake expression vector; and a classification device configured to configure the genuine expression generator and the classifier so as to minimize classification errors by the classifier, and a fake expression generator configured to generate a fake expression vector of the same format as the vector representation of background knowledge represented by the background knowledge text, the plurality of training data items including a question text and background knowledge text related to the question text, the training device including Vector and an adversarial training device that trains the fake representation generators by adversarial learning so as to maximize the classification error by a classifier for each of the fake representation generators, and when training by the adversarial training device is completed, the fake representation generator is a trained background knowledge representation generator, and when the actual question text and an arbitrary text are input, the output of the fake representation generator is a representation vector of background knowledge related to the actual question text and the arbitrary text.

[0015] A computer program according to a fourth aspect of the present invention causes a computer to function as the training device described above.

[0016] The object, configuration and effects of the invention of this application will become more apparent by referring to the following description of the preferred embodiment of the invention together with the drawings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an example of an answer passage used in a question answering system. [Diagram 2]FIG. 2 is a diagram showing an example of background knowledge. [Diagram 3] FIG. 3 is a diagram showing the relationship between background knowledge and answer passages. [Figure 4] FIG. 4 is a block diagram showing an outline of the configuration of a text classifier for identifying an answer according to the first embodiment of this application. [Diagram 5] Figure 5 is a schematic diagram to explain how the system obtains a Background Knowledge Representation Generator (BKRG) by training a fake representation generator that uses adversarial learning to create a representation vector of background knowledge from a question. [Figure 6] FIG. 6 is a block diagram showing a basic configuration of an encoder constituting the fake representation generator shown in FIG. [Figure 7] FIG. 7 is a block diagram showing a schematic configuration of a BKRG training system that trains the fake expression generator shown in FIG. 5 to generate causal relationship BKRGs and tool-goal relationship BKRGs. [Figure 8] FIG. 8 is a flowchart showing a control structure of a main routine of a program for implementing adversarial learning by the system shown in FIG. [Figure 9] FIG. 9 is a flowchart showing a control structure of a routine for parameter learning of the fake representation generator in the program shown in FIG. [Figure 10] FIG. 10 is a block diagram showing a schematic configuration of the BERT shown in FIG. [Figure 11] FIG. 11 is a block diagram showing a schematic configuration of a BERT transformer layer. [Figure 12] FIG. 12 is a block diagram showing a schematic configuration of the knowledge integration transformer layer. [Figure 13] FIG. 13 is a block diagram showing a schematic configuration of a multi-head attention unit in the BERT transformer layer shown in FIG. [Figure 14] FIG. 14 is a block diagram showing the configuration of one of the heads shown in FIG. [Figure 15] FIG. 15 is a table showing statistics of the why-type question answering data. [Figure 16] FIG. 16 is a tabular diagram showing statistics of the data for the how-to question answering. [Figure 17] FIG. 17 is a table showing experimental results showing the effectiveness of the text classifier for identifying an answer according to the first embodiment. [Figure 18] FIG. 18 is a schematic diagram showing the process of the question answering system according to the second embodiment of this application. [Figure 19] FIG. 19 is a table showing statistics of experimental data for testing the question answering system according to the second embodiment. [Figure 20] FIG. 20 is a table showing experimental results indicating the effectiveness of the question answering system according to the second embodiment. [Figure 21] FIG. 21 is a diagram showing the appearance of a computer system for implementing each embodiment of the present invention. [Figure 22] FIG. 22 is a block diagram showing the hardware configuration of the computer system whose external appearance is shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] In the following description and drawings, the same parts are given the same reference numbers, and therefore detailed descriptions thereof will not be repeated.

[0019] [First embodiment] [Answer passage and background knowledge] FIG. 1 shows an example of a question 50 and an answer passage 52. Referring to FIG. 1, the answer passage 52 contains suitable answers to the question 50, such as the sentence "The best way to prevent seasonal flu is to get vaccinated every year" and "good health habits like wearing face masks and washing your hands often can help stop the spread of germs." Since humans have background knowledge in advance, they can easily recognize that these are suitable answers to the question 50. However, it is not easy to make a computer perform processing based on such background knowledge.

[0020] [Background Knowledge Representation Generator] For why-type questions, causal relationships can be background knowledge. As a question different from what-type questions, similar to why-type questions, there are how-type questions. For how-type questions, tool-purpose relationships can be background knowledge. Therefore, in the following embodiment, a background knowledge representation generator is trained using semantic relationships such as causal relationships and tool-purpose relationships as training data. Causal relationships are important background knowledge for providing answers to why-type questions, and tool-purpose relationships are important background knowledge for providing answers to how-type questions. By training the background knowledge representation generator using such semantic relationships, when a question is given, it becomes possible to identify parts in an answer passage that are related to the question, and to classify passages that include the correct answer to the question from those that do not with high accuracy.

[0021] To train the background knowledge representation generator, it is necessary to collect a large amount of background knowledge on causal relationships and tool-goal relationships. Although this can be collected manually, in order to train the background knowledge representation generator using machine learning, it is more realistic to automatically collect a large amount of background knowledge from the web. The method for doing this will be described later.

[0022] Figure 2 shows two examples of background knowledge 60, each of which shows a tool-goal relationship, extracted from a different Web document. We prepare many such tool-goal sentences, as well as many other causal sentences. By using these sentences to train a background knowledge representation generator, we make it possible for the question answering system to easily find passages containing answers and notable parts of them when a why-type question or a how-type question is input to the system.

[0023] The concept is explained in Fig. 3. With reference to Fig. 3, when a question 50 and an answer passage 52 are given, the question 50 is given to a background knowledge representation generator, which obtains a background knowledge representation vector that expresses the background knowledge for the question 50 in some way. By examining the magnitude of the relationship between the background knowledge representation vector and each word in the answer passage 52, as shown by arrows 70, 72, and 74, noteworthy words in the answer passage 52 are identified, which become hints for the answer. As a result, it becomes easier to identify the answer. However, in order to do so, the problem is how to obtain the background knowledge representation vector. The configuration for this purpose will be explained below.

[0024] [Overall structure] Fig. 4 shows a configuration of a text classifier 90 for identifying an answer according to the first embodiment as a block diagram. Referring to Fig. 4, the text classifier 90 for identifying an answer includes a Bidirectional Encoder Representation from Transformers (BERT) 102, which is a language representation model arranged to receive an input 100 in which a question and a passage to be compared with the question are concatenated. The BERT 102 is a language representation model described in Reference 1, which will be described later, and is well known for its ease of pre-training and high performance. The configuration of the BERT 102 will be briefly described later.

[0025] The input to BERT 102 consists of a class token [CLS] representing a class classification placed at the beginning, a word string representing a question, a separation token [SEP] placed next, and a word string representing a passage placed after the separation token. The length of the input 100 varies depending on the input sentence. Therefore, in this embodiment, the length of the input 100 is fixed to 512, and a predetermined character string is substituted for the missing part.

[0026] The answer identification text classifier 90 further includes a vector conversion unit 112 that receives an input 100 in parallel with the BERT 102, converts each word in the input 100 for the question and the passage into a trained word embedding vector, and outputs the vector; a BKRG 114 including multiple BKRG layers that receives the word embedding vector sequence q of the question text and the word embedding vector sequence p of the passage text converted by the vector conversion unit 112, and outputs a vector group 116 of background knowledge representation related to the question from each layer; and a plurality of KIs (Knowledge Layers) with the same number of layers as the BKRG 114 that are provided after the output of the BERT 102, and perform language processing on the output of the BERT 102 using the vector group 116 of the background knowledge representation as attention, and finally output 106 including a label 108 indicating whether the passage of the input 100 contains an answer to the question, and a start / end position 110 of the part in the passage that is the answer. and a KI Transformer 104 that includes a Knowledge Integration (Knowledge Integration) transformer layer. The KI Transformer layer is based on the encoder block of a language model called a Transformer described in US Pat. No. 5,399,433, and is modified as described below with reference to Fig. 11 to extract answers based on background knowledge.

[0027] [BKRG114] 4, BKRG 114 includes N BKRG layers 150, 154, ..., BKRG 158, and N-1 update units 152, 156, ... that are provided corresponding to each layer among the N BKRG layers 150, etc., except for the topmost BKRG layer 158, and perform update processing on the input to the corresponding BKRG layer 150, etc., and provide the update to the next layer, BKRG layer 154, etc. The update unit 152 of the first layer receives a word embedding vector sequence (q 1 ,p 1 ) is input. The update unit 152 updates this vector sequence (q 1 ,p 1 ) is updated with the specified update process, and the updated vector sequence (q 2 ,p 2 ) is output. Similarly, the final BKRG layer 158 outputs a vector sequence (q N ,p N ) is given. The output of the BKRG layers 150, 154, ..., 158 is the vector r 1 , r 2 , …, r N , 158, etc. are pre-trained as described below. The answer-identifying text classifier 90 is also trained as a whole, with the parameters of the BKRG layers 150, 154, ..., 158, etc. all being fixed and identical to one another. Thus, the BKRG layers 154, ..., 158 may be created by copying the trained BKRG layer 150, or the BKRG layer 150 may be repeatedly used as the BKRG layers 154, ..., 158.

[0028] The vector group 116 of the background knowledge representation output from the BKRG layers 150, 154, ..., 158 is the vector r 1 , r 2 , …, r N Includes.

[0029] [KI Transformer 104] The KI transformer 104, like the BKRG 114, includes N KI transformer layers 130, 132, ..., 134. The KI transformer layer 130 combines the output of the BERT 102 with a vector r 1 The KI transformer 130 receives the output of the KI transformer layer 130 and the vector r 2 The KI transformer layer in the top layer, the Nth layer, receives the output from the KI transformer layer (not shown) in the lower layer, and the vector r output by BKRG layer 158 in the Nth layer. N and outputs output 106.

[0030] Each component of the answer identifying text classifier 90 will now be described.

[0031] [BKRG] The BKRG layers 150, 154, ..., 158 all have the same configuration and the same parameters. In practice, after training the BKRG layer 150, the BKRG layers 154, ..., 158 can be calculated by copying the BKRG layer 150 or repeatedly using the BKRG layer 150. Therefore, the configuration and training method of only the BKRG layer 150 will be explained below. Considering that the background knowledge should be input to the KI transformer layer 130, it is clear that some kind of vector format is desirable for the background knowledge. However, if the background knowledge is to be expressed as a vector, there is almost no clue as to what content should be expressed in what format.

[0032] Here, as a technology that may be capable of creating a background knowledge representation in vector format by automatic processing by a computer, there is a machine learning technology called adversarial learning (Generative Adversarial Network: GAN) described in Reference 2. GAN is particularly often applied to image generation, and can generate fake images (fakes) that are almost indistinguishable from photographs, for example. Even in question answering processing such as this embodiment, it may be effective in generating background knowledge. In this embodiment, too, we attempt to train BKRG150 using GAN.

[0033] 《BKRG150 Training》 5 shows a basic configuration for training the BKRG layer 150 by the GAN. Referring to FIG. 5, the GAN 180 includes a real representation generator 194 that uses training data items prepared in advance, each of which includes a question 190 and background knowledge 192 related to the question, to generate a vector representation called a real representation 196 from the question 190 and the background knowledge 192, a fake representation generator 200 that samples noise 198 representing any text and generates a fake representation 202 called a fake representation from the sampled noise, and a classifier 204 that receives an input of the real representation 196 or the fake representation 202, classifies them as real or fake, and outputs a classification result 206. The real representation generator 194, the fake representation generator 200, and the classifier 204 are all neural networks.

[0034] Both the real expression generator 194 and the fake expression generator 200 have a similar structure called an encoder. Therefore, only the structure of the fake expression generator 200 will be described below. With reference to Fig. 6, the fake expression generator 200 includes a word embedding vector conversion unit 224 that receives the noise 198 and converts it into a noise word embedding vector sequence 226, and a word embedding vector conversion unit 220 that receives the question 190, converts each word constituting the question 190 into a word vector, and outputs a question word embedding vector sequence 222.

[0035] The fake representation generator 200 further includes an attention summation unit 228 that sums attention from the question word embedding vector sequence 222 for each vector in the noise word embedding vector sequence 226 and outputs an attention-modified word embedding vector sequence 230, and a CNN 232 having an input that receives the attention-modified word embedding vector sequence 230 and is trained to output a fake representation 202, which is a vector representation of the question 190.

[0036] 5 has a similar configuration to the fake expression generator 200, including the CNN 232. The real expression generator 194 differs from the fake expression generator 200 in that the target of encoding is the background knowledge 192, not the noise 198.

[0037] The classifier 204 shown in Fig. 5 is a feedforward network having a logistic regression layer with a sigmoid function as an activation function in the final layer. The real representation 196 or the fake representation 202 is input to the classifier 204. The classifier 204 has two outputs, and is trained so that, for example, when the input is determined to be the real representation 196, one output is true and the other is false, and when the input is determined to be the fake representation 202, one output is false and the other is true.

[0038] These three networks are trained by adversarial learning so that the real representation generator 194 and the classifier 204 are able to distinguish the real representation 196 from the fake representation 202 (to minimize the classification error), and the fake representation generator 200 is able to make the fake representation 202 indistinguishable from the real representation 196 (to maximize the classification error of the classifier 204 against the fake representation 202). That is, these three networks are trained according to the following equations:

[0039]

number

[0040] Training of the fake expression generator 200 is terminated when the classifier 204's judgment finally reaches 50% or when the specified number of iterations of learning is completed. This is based on game theory, and eventually a Nash equilibrium is reached, and the probability that the classifier 204 will correctly classify is 50%. If a fake expression 202 is generated from a question 190 and noise using the fake expression generator 200 trained in this way, the fake expression 202 will be indistinguishable from a real expression 196 generated by the real expression generator 194 from the question 190 and background knowledge 192.

[0041] That is, when a question and an arbitrary text are given to the fake expression generator 200, the fake expression generator 200 outputs an expression as if the question and background knowledge were also given to the fake expression generator 200. Since it is no longer appropriate to call this output a fake expression, hereinafter it will be called a background knowledge representation vector. In addition, the fake expression generator 200 after training is completed will also be called a background knowledge representation generator (BKRG). The number of elements of the background knowledge representation vector is determined in advance so as to be the maximum number of elements of these vectors obtained from the training data.

[0042] As will be described later, when the background knowledge representation generator obtained by such innovation was used to judge whether a passage provided a correct answer to a question, it was found that a clearly higher accuracy could be obtained compared to the conventional method.

[0043] Training by GAN180 is realized by computer hardware and a computer program (hereinafter referred to as "program") executed by the computer hardware. Training data must be prepared for the training. In this embodiment, a question answering system capable of answering both why-type questions and how-type questions is provided. To achieve this, two BKRGs must be prepared: a causal relationship BKRG and a tool-goal relationship BKRG. In this embodiment, the outputs of these are concatenated to produce the output of a single BKRG.

[0044] To train the causal BKRG, we need to collect causal relations, and to train the tool-goal BKRG, we need to collect tool-goal relations. Figure 7 shows the schematic configuration of the training system for collecting training data and training the BKRG.

[0045] Referring to FIG. 7, this BKRG training system 240 includes a causal relationship BKRG training unit 252 that collects a large amount of text representing causal relationships from the Internet 250 and uses it to train a causal relationship BKRG 256, and a tool-goal relationship BKRG training unit 254 that collects a large amount of text representing tool-goal relationships from the Internet 250 and uses it to train a tool-goal relationship BKRG 258.

[0046] The causal relationship BKRG training unit 252 includes a causal relationship extraction unit 270 that extracts text representing causal relationships from the Internet 250, a causal relationship storage device 272 that stores the text extracted by the causal relationship extraction unit 270, a causal relationship learning data generation unit 274 for generating learning data for the causal relationship BKRG 256 by extracting and combining a question portion and a background knowledge portion from each of the causal relationships stored in the causal relationship storage device 272, a causal relationship learning data storage device 276 for storing the learning data thus generated, and a causal relationship BKRG training unit 278 for training the causal relationship BKRG 256 by adversarial learning using the learning data obtained from the causal relationships stored in the causal relationship learning data storage device 276.

[0047] The causal relationship learning data generation unit 274 generates learning data from causal relationships as follows. A causal relationship consists of a cause part and a consequence part that expresses the result. The consequence part is the question part, and the cause part is the background knowledge part. For example, consider the causal relationship "Because most people do not have antibodies against the new influenza virus (cause part), a global pandemic and the associated social impacts could occur (consequence part)." In this case, the question "Why could a global pandemic and the associated social impacts occur?" obtained from the consequence part becomes the question part, and the cause part "Because most people do not have antibodies against the new influenza virus" becomes the background knowledge part.

[0048] The tool-goal relationship BKRG training unit 254 includes a tool-goal relationship extraction unit 280 that extracts text representing tool-goal relationships from the Internet 250, a tool-goal relationship storage device 282 that stores the text extracted by the tool-goal relationship extraction unit 280, a tool-goal relationship learning data generation unit 284 for generating learning data for the tool-goal relationship BKRG 258 by extracting and combining a question portion and a background knowledge portion from each of the relationships stored in the tool-goal relationship storage device 282, a tool-goal relationship learning data storage device 286 for storing the learning data thus generated, and a tool-goal relationship BKRG training unit 288 for training the tool-goal relationship BKRG 258 by adversarial learning using the learning data obtained from the tool-goal relationships stored in the tool-goal relationship learning data storage device 286.

[0049] The tool-goal relationship learning data generation unit 284 generates learning data from the tool-goal relationship as follows. The tool-goal relationship consists of a tool part and a goal part. The goal part is the question part, and the tool part is the background knowledge part. For example, consider the tool-goal relationship, "When going to a crowded place, you must wear a mask (goal part) to prevent influenza infection." In this case, the question "When going to a crowded place, what should you do to prevent influenza infection?" obtained from the goal part becomes the question part, and "wear a mask" in the tool part becomes the background knowledge part.

[0050] In this embodiment, the causal relationship BKRG256 and the tool-goal relationship BKRG258 are trained separately in this way. During testing, the same questions are given to both the causal relationship BKRG256 and the tool-goal relationship BKRG258, and the vectors obtained from these are linked to form a background knowledge representation vector. In other words, the causal relationship BKRG256 and the tool-goal relationship BKRG258 are arranged side by side to form a single BKRG.

[0051] Figure 8 is a flowchart showing the control structure of a program that realizes training by GAN 180. The causal relationship BKRG256 and the tool-goal relationship BKRG258 shown in Figure 7 can be trained separately. Here, a program that realizes the training of the causal relationship BKRG256 is explained, but a similar program can be used to train the tool-goal relationship BKRG258.

[0052] 8, this program includes step 300 of communicatively connecting the computer to the causal relationship learning data storage device 276 shown in FIG. 7; step 304 of training the fake expression generator 200, the real expression generator 194, and the classifier 204 shown in FIG. 5 by adversarial learning between the real expression generator 194, the classifier 204, and the fake expression generator 200; step 306 of judging whether or not a training termination condition is satisfied as a result of step 304 and branching the flow of control according to the judgment result; and step 308 of saving the parameters of the fake expression generator 200 at that time in a predetermined storage device and terminating the execution of this program in response to the judgment that the termination condition is satisfied in step 306. When the judgment in step 306 is negative, the control returns to step 304. Thus, step 304 is repeated until the termination condition is satisfied. In step 308, the user may be allowed to specify a file name and a location to save the parameters.

[0053] 9, the program for realizing the adversarial learning in step 304 in FIG. 8 includes step 320 for sampling a plurality of samples, each of which is a combination of a question and background knowledge corresponding to the question, from training data, step 322 for fixing parameters of the fake expression generator 200, and step 323 for classifying the real expression 196 generated by the real expression generator 194 as a fake, and the fake expression 202 generated by the fake expression generator 200 as a real, using the sample questions and background knowledge sampled in step 320. The method includes a step 324 of training the real representation generator 194 and the classifier 204, respectively, to minimize the probability that the classifier 204 will make a false classification, followed by a step 326 of fixing parameters of the classifier 204 and the real representation generator 194, and a step 328 of learning parameters of the fake representation generator 200 to generate fake representations 202 from the questions 190 and noise and to maximize the probability that the classifier 204 will falsely classify these fake representations 202 as real representations.

[0054] 《BERT102》 Figure 10 shows the schematic configuration of BERT 102. BERT is known to be easy to pre-train and to function as a language representation model that can perform, for example, translation with very high accuracy or extract relevant parts for question answering by fine-tuning.

[0055] The BERT 102 includes multiple BERT transformer layers 602, 604, ..., 606 that are arranged in series with the same configuration. Each of these has the same configuration as the encoder block of the transformer, just like the KI transformer layer 130, and only a small part of the configuration is different from the KI transformer layer 130. Here, they are called BERT transformer layers to distinguish them from the KI transformer layers.

[0056] The BERT transformer layer 602 receives the input word embedding vector sequence 600, the BERT transformer layer 604 receives the output of the BERT transformer layer 602, and so on. The word sequence 608 output by the final BERT transformer layer 606 is input to the KI transformer 104. Note that in FIG. 10, the input word embedding vector sequence 600 is shown as being given to the BERT 102 all at once. However, in reality, the input word embedding vectors constituting the input word embedding vector sequence 600 are input to the BERT 102 in order, and the word sequence 608 is also output in order.

[0057] The multiple BERT transformer layers 602, 604, ..., 606 have the same configuration, but the parameter values ​​are different from each other due to pre-training and fine-tuning of BERT 102. The input to each sub-network and its output are vectors of a fixed length, for example, 512-dimensional vectors. The length of this vector is chosen to be greater than the number of words in the longest input sentence to be processed.

[0058] In the following description, it is assumed that BERT 102 has already been pre-trained in Japanese. The answer identification text classifier 90 shown in FIG. 4 constitutes one neural network as a whole. Fine-tuning of the answer identification classifier 90 as a whole is performed by using training data that includes a pair of a question and an answer candidate passage, a label indicating whether the answer candidate passage contains an answer to the question, and a label indicating the start position and end position of the answer portion, which are created manually or otherwise, and training of the update part of BERT 102, KI transformer 104, and BKRG 114 is performed by a normal backpropagation method. Note that prior to this fine-tuning, training of BKRG 150, 154, ..., 158, etc. is completed, and the parameter values ​​of BKRG layers 150, 154, ..., 158 are fixed during fine-tuning.

[0059] Transformer configuration As described above, the BERT 102 and the KI transformer 104 shown in Fig. 4 use as basic blocks the encoder blocks of a high-performance neural network model called a transformer, which is often used as a language model for natural language (KI transformer layers 130, 132, ..., 134 in Fig. 4 and BERT transformer layers 602, 604, ..., 606 in Fig. 10). Fig. 11 shows an outline of the configuration of the BERT transformer layer, and Fig. 12 shows an outline of the configuration of the KI transformer layer.

[0060] 11 and 12, the BERT transformer layer and the KI transformer layer are almost identical to each other, with only some differences.

[0061] Although not shown, for example, the input stage of BERT102 is provided with a word converter that converts each word in the input sentence into a word embedding vector sequence having a numerical value as an element, thereby converting the input sentence into a word embedding vector sequence, and a position encoder that encodes position information indicating the position of the corresponding word in the input sentence for each word embedding vector in the word embedding vector sequence output by the word converter. By encoding the position information in this way, the transformer can encode the word sequence by utilizing the order of the input word sequence without using an RNN (Recurrent Neural Network) or CNN.

[0062] The output of the BERT transformer layer 602 is the input to the BERT transformer layer 604, and so on. The BERT 102 and the KI transformer 104 are connected so that the output of the final BERT transformer layer 606 of the BERT 102 is the input to the KI transformer layer 130 of the KI transformer 104. Although not shown in FIG. 4, the output stage of the KI transformer 104 is provided with a linear transformation sub-network and a SoftMax layer in that order. The SoftMax layer provides the probability of each output word.

[0063] The BERT transformer layer 602 receives the input 340, which is a word embedding vector sequence, and encodes the input 340 into a vector representation with self-attention. A word embedding vector sequence is a sequence of multiple word embedding vectors with the same configuration, and can be considered as a word embedding vector matrix depending on how you look at it. In the transformer, operations between matrices consisting of vector sequences obtained for multiple words play an important role. Note that self-attention means attention by the same input 340 to an input 340. For example, self-attention for a specific word in the input 340 is calculated using attention from all other words in the input 340.

[0064] 11, the BERT transformer layer 602 includes a two-layer encoder sub-network 436. The encoder sub-network 436 includes a first layer sub-network 440 that receives the input 340 and a second layer sub-network 442 that receives the output of the first layer sub-network 440 and generates the output of the BERT transformer layer 602.

[0065] The first layer sub-network 440 includes a multi-head attention sub-network 450 that receives the word embedding vector sequence 340, calculates self-attention for the corresponding word for each input word embedding vector using the word embedding vector sequence 340, and outputs an attention vector sequence consisting of the attention, and an ADD & Norm sub-network 452 that adds the corresponding word embedding vector in the word embedding vector sequence that is the input of the multi-head attention sub-network 450 to each attention vector in the attention vector sequence output by the multi-head attention sub-network 450, and then performs layer normalization. The multi-head attention sub-network 450 uses a matrix consisting of the word embedding vector sequence that is the input to the BERT transformer layer 602, which is branched into three. These three inputs are called V (value), K (Key), and Q (Query) from the left in FIG. 11. Details of the multi-head attention sub-network 450 will be described later with reference to FIG. 13.

[0066] The second layer subnetwork 442 includes a fully-connected subnetwork 460 including a fully-connected feedforward network provided corresponding to each word position output by the ADD & Norm subnetwork 452, and an ADD & Norm subnetwork 462 that performs the same processing as the ADD & Norm subnetwork 452 on the output of the fully-connected subnetwork 460. The output of the ADD & Norm subnetwork 462 is a word embedding vector sequence of the same length as the input to the BERT transformer layer 602.

[0067] As mentioned above, the word embedding vector sequence can be thought of as a matrix with each row being a word embedding vector. Therefore, in practice, each operation in the BERT transformer layer 602 is performed as a matrix operation.

[0068] The KI transformer layer 130 shown in Fig. 12 has the same configuration as the BERT transformer layer 602 etc. However, the BERT transformer layer 602 etc. differs from the BERT transformer layer 602 in that instead of using a matrix Q obtained by branching the input of the previous stage, the BERT transformer layer 602 etc. uses a background knowledge representation vector r output from a BKRG layer (for example, the BKRG layer 150 shown in Fig. 1) which is the same layer as (corresponding to) the KI transformer layer 130. For V and K, like the BERT transformer layer 602, the output of the previous stage is branched and used.

[0069] 12, the KI transformer layer 130 includes a two-layered encoder sub-network 476. The encoder sub-network 476 includes a first layer sub-network 492 and a second layer sub-network 494, similar to the encoder sub-network 436 of the BERT transformer 602.

[0070] The first layer subnetwork 492 includes a multi-head attention subnetwork 510 that sequentially receives a word embedding vector sequence from the previous stage (in the case of the KI transformer layer 130, the output of the BERT 102), calculates attention for each input word embedding vector using the background knowledge representation vector for the corresponding word, and outputs an attention vector sequence consisting of these attentions, and an ADD & Norm subnetwork 512 that adds the corresponding word embedding vector in the word embedding vector sequence input to the multi-head attention subnetwork 510 to each attention vector in the attention vector sequence output by the multi-head attention subnetwork 510, and then performs layer normalization. Unlike the case of the BERT transformer layer 602, the multi-head attention subnetwork 510 uses a matrix that is input from the previous stage, branched into two, V and K. On the other hand, Q input to the multi-head attention subnetwork 510 is r given by the BKRG 150 (FIG. 4) in the same layer. 1 However, Q is a matrix (a sequence of vectors), while r1 is a vector. Therefore, the desired operation cannot be performed with the configuration as it is. Therefore, in this embodiment, the vector r 1 etc. are converted into matrices of the same format as Q.

[0071] The second layer subnetwork 494 includes a fully-connected subnetwork 520 including a fully-connected feedforward network provided corresponding to each word position output from the ADD & Norm subnetwork 512, and an ADD & Norm subnetwork 522 that performs the same processing as the ADD & Norm subnetwork 462 on the output of the fully-connected subnetwork 520. The output of the ADD & Norm subnetwork 522 is a word embedding vector sequence of the same length as the input to the BERT transformer 602, and is input to the KI transformer 132 (see FIG. 4).

[0072] Fig. 13 shows a schematic configuration of the multi-head attention sub-network 450 shown in Fig. 11 in a general transformer. The multi-head attention sub-network 510 shown in Fig. 12 has a similar configuration, although it uses a vector r instead of a matrix Q and there are differences in the calculation processing shown in the following formulas 2 and 3.

[0073] Referring to FIG. 13, the multi-head attention sub-network 450 includes a header section 540 including multiple parts called headers, a vector concatenation section 542 that concatenates vectors output from each of the multiple headers in the header section 540, and a linear transformation sub-network 544 that performs a linear transformation on the output of the vector concatenation section 542.

[0074] The header section 540 includes a plurality (h pieces) of headers 550, and three linear transformation sub-networks 552 for receiving three inputs branched from a vector sequence that is an input from a lower layer for each header 550, and performing linear transformation on the inputs using a matrix whose element values ​​have been learned in advance to generate three matrices V, K and Q, which are given to each of the h pieces of headers of the header 550. Note that in the KI transformer layer 130 etc. shown in Fig. 12, instead of Q, a background knowledge representation vector r that has been converted into a matrix as described below is used.

[0075] 14, the header 550 includes a matrix multiplication sub-network 560 that performs multiplication between the matrix Q and the matrix K, and a division circuit 562 that divides the matrix obtained by the matrix multiplication sub-network 560 by a predetermined constant. The header 550 further includes a SoftMax layer 566 that performs SoftMax transformation on the matrix scaled by the division circuit 562, and a matrix multiplication sub-network 568 that performs multiplication between the matrix after SoftMax by the SoftMax layer 566 and the matrix V after linear transformation.

[0076] The above transformation is performed in the form of vector and matrix operations as described above. This can be expressed as the following formula. However, d k is the square root of the number of rows of the matrix K after linear transformation. If the number of rows of the matrix K is 64, then d k =8.

[0077]

number

[0078] In addition, in the multi-head attention sub-network 510 such as the KI transformer layer 130 shown in FIG. 12, the vector r is transformed into a matrix of the same format as Q by the following formula, and the above calculation is performed.

[0079]

number

[0080]

number

[0081] 《Update Department》 The update units 152, 156, ... shown in Fig. 4 are part of the entire neural network and have the same configuration. For example, the update unit 152 updates the word embedding vector sequence (q 1 , p 1) is received. This word embedding vector sequence is obtained by performing pre-trained embedding vectorization for each word in the question and the passage, and concatenating the vectors obtained respectively. The BKRG layer 150 receives this vector sequence (q 1 , p 1 ) and process it to answer question q 1 Background knowledge about vector r 1 The update unit 152 then generates the background knowledge vector r 1 and the vector sequence (q 1 , p 1 ) to find the vector sequence (q 1 , p 1 ) to update the vector sequence (q 2 ,p 2 ) and provides it to the next BKRG layer, the BKRG layer 154, and the update unit 156. The BKRG layer 154 generates this vector sequence (q 2 ,p 2 ) and process it to generate a new background knowledge vector r 2 The update unit 156 generates the background knowledge vector r 2 and the vector sequence (q 2 ,p 2 ) to find the vector sequence (q 2 ,p 2 ) to update the vector sequence (q 3 ,p 3 ) is generated and used as the input to the next BKRG layer, and so on.

[0082] In general, a sequence of vectors q i Or p i Let x be the word embedding vector of the jth word in i j If (x=q or p), the background knowledge vector r calculated in the i-th BKRG layer is expressed as follows: i By calculating the relevance of the word to x, we obtain a word embedding vector ￣x i j is obtained.

[0083]

number

[0084] x i+1 (x i+1 q i+1 Or p i+1 (either ￣x i and x i It is calculated in highway network form (Reference 3) using

[0085]

number

[0086] [Operation] The text classifier 90 for identifying an answer, the configuration of which has been described above, operates as follows. In the following description, it is assumed that the BERT 102 has already been pre-trained in Japanese. As described above, the entire text classifier 90 for identifying an answer shown in FIG. 4 is fine-tuned by a normal backpropagation method using training data that includes a pair of a question and a passage, a label indicating whether the passage includes an answer to the question, and a label indicating the start and end positions of the answer, which are created manually or otherwise in advance. Note that prior to this fine-tuning, training of the BKRG layers 150, 154, ..., 158, etc. is completed, and the parameter values ​​of the BKRG layers 150, 154, ..., 158 are fixed during fine-tuning.

[0087] Learning the fake expression generator 200 First, the learning of the fake expression generator 200 shown in FIG. 5 will be described with reference to FIG. 8. The fake expression generator 200 is trained by adversarial learning together with the real expression generator 194 and the classifier 204 shown in FIG. 5. Prior to this training, the causal relationship extraction unit 270 and the tool-purpose relationship extraction unit 280 shown in FIG. 7 collect texts indicating causal relationships and tool-purpose relationships from the Internet 250 and store them in the causal relationship storage device 272 and the tool-purpose relationship storage device 282, respectively. The causal relationship learning data generation unit 274 and the tool-purpose relationship learning data generation unit 284 generate causal relationship learning data and tool-purpose relationship learning data from the causal relationship texts and tool-purpose relationship texts stored in the causal relationship storage device 272 and the tool-purpose relationship storage device 282, respectively, and store them in the causal relationship learning data storage device 276 and the tool-purpose relationship learning data storage device 286, respectively.

[0088] A causal relationship BKRG training unit 278 trains the causal relationship BKRG 256 using the causal relationship learning data stored in the causal relationship learning data memory device 276, and a tool-goal relationship BKRG training unit 288 trains the tool-goal relationship BKRG 258 using the tool-goal learning data stored in the tool-goal relationship learning data memory device 286.

[0089] The training procedures for causal BKRG256 and tool-goal BKRG258 are similar, except for the training data. Therefore, we will only explain the operation of GAN180 for training causal BKRG256.

[0090] 8, in this learning, a computer is connected to the causal relationship learning data storage device 276. Specifically, the computer opens files that contain the causal relationship learning data and are present in the causal relationship learning data storage device 276 realized by an external storage device, reads out the contents of the files, and stores them in memory. Alternatively, the causal relationship learning data is read out from a database that stores the files, and stores them in memory (step 300).

[0091] Next, adversarial learning is performed between the real representation generator 194 and the classifier 204 and the fake representation generator 200 (step 304).

[0092] 9, in the adversarial learning performed in step 304, a plurality of pairs of questions 190 and background knowledge 192 for the questions are sampled from the learning data (step 320). Then, the parameters of the fake expression generator 200 are fixed (step 322). Using the sampled questions 190 and the background knowledge 192 for the questions 190, the real expression generator 194 generates a real expression 196 and provides it to the classifier 204. Meanwhile, the fake expression generator 200 generates a fake expression 202 using the same questions 190 and randomly generated noise and provides it to the classifier 204. The classifier 204 judges whether the real expression 196 and the fake expression 202 are real or not. Using this judgment result for the entire sampled training data, in step 324, the parameters of the classifier 204 and the genuine representation generator 194 are trained by backpropagation while keeping the parameters of the fake representation generator 200 fixed so that the misjudgment of the classifier 204 for this data is minimized, i.e., the probability of misjudging the genuine representation 196 as fake and the fake representation 202 as genuine is reduced.

[0093] Next, the parameters of the discriminator 204 and the real expression generator 194 are fixed (step 326). With the parameters of the discriminator 204 fixed, the fake expression generator 200 is trained using the questions 190 and the randomly generated noise 198 (step 328). Specifically, the fake expression generator 200 generates a fake expression 202. The discriminator 204 judges whether the fake expression 202 is a real expression or not. This judgment is performed for a plurality of questions 190, and the parameters of the fake expression generator 200 are adjusted by fixing the parameters of the discriminator 204 and the real expression generator 194 so that the misjudgment by the discriminator 204 is maximized, i.e., the probability that the discriminator 204 judges the fake expression 202 as a real expression is increased.

[0094] By repeating such processing, the real representation / fake representations generated by the real representation generator 194, the discriminator 204, and the fake representation generator 200 will eventually reach a Nash equilibrium in game theory, and the judgment result by the discriminator 204 will reach a state where 50% of the results are correct and 50% are incorrect, or will reach a certain state close to the Nash equilibrium.

[0095] 8, in step 306, it is determined whether the accuracy of the judgment of the classifier 204 has reached this Nash equilibrium state or a certain state close to the Nash equilibrium. If the result of the determination is negative, control returns to step 304, and adversarial learning of the real representation generator 194, the fake representation generator 200, and the classifier 204 is continued. If the result of the determination in step 306 is positive, in step 308, the parameters of the fake representation generator 200 are saved in a storage device, and the process is terminated. The parameters of the fake representation generator 200 and a program that realizes the network structure of the fake representation generator 200 provide the causal relationship BKRG256 of this embodiment.

[0096] The tool-goal relationship BKRG258 can be obtained in the same way by performing adversarial learning using the tool-goal relationship learning data storage device 286 shown in Fig. 7 on another set of real expression generator 194, fake expression generator 200, and classifier 204. BKRG150, 154, and BKRG158 shown in Fig. 4 are all obtained by concatenating the causal relationship BKRG256 and the tool-goal relationship BKRG258, and the outputs of both are concatenated to obtain the vector r from BKRG150, etc. 1 etc.

[0097] Training BERT102 The pre-training of BERT102 is a well-known method, and the details will not be repeated here. In simple terms, a large number of sentences for pre-training are prepared in advance, and BERT102 pre-trains by deleting any one word from those sentences and predicting that word as input. With this method, there is no need to process data to create so-called training data.

[0098] The pre-training of BERT 102 and the fine-tuning of the answer-specific text classifier 90 are performed as described above, and the details will not be repeated here. In this fine-tuning, the usual backpropagation method can be used, with the parameters of the BKRG layers 150, 154, ..., 158 kept fixed.

[0099] Operation of the Text Classifier 90 for Identifying Answers During Testing Referring to FIG. 4, when testing the answer-identifying text classifier 90, a question and a passage to be determined as to whether or not it contains an answer to the question are provided as input 100 to the BERT 102 and the vector conversion unit 112.

[0100] The BERT 102 processes the input 100 according to the trained parameters and provides an output to the KI transformer layer 130 .

[0101] On the other hand, the vector conversion unit 112 converts the questions and passages included in the input 100 into word embedding vector sequences, concatenates them, and provides them to the BKRG layer 150 and the update unit 152. The BKRG layer 150 converts the word embedding vector sequence q 1 Let p be the question and p be the word embedding vectors of the passage. 1 As shown in Fig. 5, we treat the vector r 1 is generated and provided to the KI transformer layer 130.

[0102] The KI transformer layer 130 converts the output of the BERT 102 into the vector r 1 The calculation is performed using the attention (matrix Q) and the result is given to the KI transformer layer 132. At this time, the vector r 1 is used to direct the KI transformer layer 130 to carefully read likely answers when analyzing the target passage.

[0103] On the other hand, in the BKRG 114, the update unit 152 updates the vector r 1 Using the conversion formula already described, the word embedding vector sequence (q 1 ,p 1 ) into the word embedding vector sequence (q 2 , p 2 ) and provides it to the BKRG layer 154 and the update unit 156.

[0104] The BKRG layer 154 calculates the word embedding vector sequence (q 2 , p 2 )q 2 Question, p 2 As shown in Fig. 5, we treat the vector r 2 is generated and provided to the KI transformer layer 132.

[0105] The KI transformer layer 132, like the KI transformer layer 130, converts the output of the KI transformer layer 130 into the vector r 2 The attention (matrices V and K) is calculated, and the result is given to the subsequent KI transformer layer. At this time, the vector r 2 is used to direct the KI transformer layer 132 to carefully read likely answers when analyzing the target passage.

[0106] The same process is carried out below, and the vector r N is provided to the KI transformer layer 134. The KI transformer layer 134 converts the output of the previous KI transformer layer into a vector r N The processing is performed with attention set to "CLS" and the result is output as output 106. In the portion corresponding to the first token "CLS" of output 106, a label is output indicating whether or not the passage constituting input 100 contains the answer to the question constituting input 100, and among the portions corresponding to each word in the passage of output 106, the start position and end position of the word string that is the answer are each indicated as a probability.

[0107] 4, the answer identifying text classifier 90 for determining the label and the start and end positions of the answer is described as one, but the present invention is not limited to such an embodiment, and these may be trained and used separately.

[0108] [Experimental Results] We conducted an experiment (answer identification experiment) to classify whether a given passage contains a correct answer to a Japanese question using the above-mentioned answer identification text classifier 90. The experiment covers both why-type question answering and how-type question answering tasks.

[0109] As the training data for the adversarial learning of the fake representation generator 200 in FIG. 5 (BKRG150, 154, ..., BKRG158 in FIG. 4), causal relationship data (obtained by the method described in Reference 4 below) and tool-goal relationship data (obtained by the method described in Reference 5) were used. This causal relationship data contains approximately 100 million pieces of causal relationship data automatically obtained from 4 billion web texts. In the experiment, 1 million pieces of these were randomly sampled and used. As the tool-goal data, it contains approximately 120 million pieces of tool-goal relationship data automatically obtained from 400 million web texts. In the experiment, 1 million pieces of these were randomly sampled and used.

[0110] The thus obtained why-type question-answering data and how-type question-answering data were classified into learning data, development data, and evaluation (test) data for the experiment. Statistics of the classified data are shown in Fig. 15 and Fig. 16, respectively.

[0111] As mentioned above, BERT102 has LARGE The number of layers was L = 24, the number of headers in the transformer encoder A = 16, and the number of hidden units in the feedforward network H = 1,024. 2.2 billion sentences were used for training. The batch size during training was 4,096, and the number of training steps was 1.1 million.

[0112] In the experiments, we used the causal BKRG and the tool-goal BKRG trained by the method described in the above embodiment.

[0113] The results of this experiment are shown in table form in Fig. 17. In Fig. 17, the result 636 with "Proposed" written on the fourth line of the results column is the result obtained by the system according to the above embodiment. In Fig. 17, "P@1 (Precision at top answer)" indicates the highest accuracy in the results ranked by question. "MAP (Mean Average Precision)" indicates the average of the average accuracy in the results ranked by question.

[0114] The result 630 in the results column shows the result using the method described in Reference 6 below (CNN and Answer Representation Generator (ARG)). The result 632 in the results column shows the result when only BERT is used. 。 The result 634 in the result column is the result of using the method using BERT in result 632, with the representation output by BKRG in the above embodiment added as the input to the final SoftMax layer.

[0115] In contrast, results 638 and onward in the result column are obtained by removing certain elements from the method according to the above embodiment shown in result 636. Result 638 is obtained by removing the update of the question and passage pair by update units 152, 156, etc. shown in FIG. 4. Result 640 is the result obtained by removing the causal relationship BKRG from the method according to the above embodiment. Result 642 is the result obtained by removing the tool-goal relationship BRKG.

[0116] Comparing the first to third rows and the fourth row in the results column of Figure 17, we can see that the fourth row gives the highest accuracy. In other words, by using the background knowledge representation vector by BKRG and integrating this background knowledge representation vector with the results by BERT using the KI transformer, we can see that high accuracy is obtained not only compared to the basic method in the first row, but also compared to BERT alone (second row) and the BERT+BKRG method.

[0117] Furthermore, the performance of result 636 is higher than that of results 638, 640, and 642. Therefore, it can be seen that the method of updating the input question and passage, such as update unit 152 and BKRG 154 shown in FIG. 4, and the method of not using causal relationships or tool-goal relationships in BKRG 150, 154, ..., BKRG 158, etc., all contribute to the performance improvement of the above embodiment, as shown by result 636.

[0118] [Second embodiment] 〔composition〕 The text classifier 90 for identifying answers according to the first embodiment is an application of the present invention to Japanese why-type questions and how-type questions. However, the present invention is applicable not only to Japanese but also to other languages, for example, English, and also to open-domain question answering systems rather than to a specific domain. The question answering system according to the second embodiment relates to a question answering system that performs an open-domain question answering task in English.

[0119] The task is to receive a question, select a passage that is likely to contain the answer to the question, and then extract the answer from the passage. The main question type is a what-type question. Answers tend to consist of words or noun phrases. Answers to what-type questions tend to be shorter than answers to why-type and how-type questions.

[0120] A task called Distantly supervised open-domain QA (DS-QA) is included in a question answering system for English, and is described in Reference 7 below. Referring to FIG. 18, this task 750 is a task of receiving a question 760, searching 762 a plurality of passages 764, and extracting an answer 774 from the passage 764. More specifically, the question answering system for this task 750 includes a paragraph selector 766 that selects a set 768 of paragraphs that are highly likely to include the answer from the passage, and a paragraph reader 770 that calculates a set 772 of answer candidates that are composed of word strings that are most likely to be an answer to the question 760 when the set 768 of paragraphs selected by the paragraph selector 766 and the question 760 are given. An answer 774 is extracted from the set 772 of answer candidates.

[0121] An exemplary question answering system for performing task 750 described in Reference 7 includes a search 762 that receives a question 760 and searches and retrieves passages 764 from a text archive that may contain potential answers. Let P denote passage 764, where P={p1,...,p N}, where p1,…,p N Each of N represents a paragraph (N=a positive integer). Task 750 further calculates the number of paragraphs p1, ..., p2 in passage 764. N the paragraph selector 766 for selecting a paragraph likely to contain a correct answer from each of the paragraphs in the set 768 and generating a set 768 of paragraphs; a paragraph reader 770 for extracting a portion likely to be an answer from each of the paragraphs in the set 768 and generating a set 772 of answer candidates; and a set 772 of answer candidates for outputting, from the set 772 of answer candidates, the answer candidate that is most likely to be a correct answer to the question 760 as an answer 774.

[0122] In this embodiment, for a question q, P={p i}, the score Score(a|q,P) of each answer a in the set of answer candidates 772 is defined as follows, where ap represents the paragraph containing the answer.

[0123]

number

[0124] In this embodiment, the paragraph selector 766 and the paragraph leader 770 described in the first embodiment can be used. However, the paragraph selector 766 has the same configuration as that described in the first embodiment, and for each combination of a question and each paragraph, the paragraph is used as a label. centre The paragraph leader 770 is a set of a question and a paragraph that contains the correct answer to the question. centre For this combination, two answer-identifying text classifiers 90 can be used, trained using training data with the start of the answer as a label and training data with the end of the answer as a label, respectively.

[0125] 〔effect〕 In order to investigate the performance of the task 750 according to the second embodiment, the following experiment was carried out. Figure 19 shows statistics of the training and evaluation data used in this experiment.

[0126] In the experiment, three publicly known datasets (Quasar-T (Reference 8), SearchQA (Reference 9), and TriviaQA (Reference 10)) were used for comparison, and the proposed method according to the second embodiment was compared with three publicly known methods, OpenQA (Reference 11), TriviaQA (Reference 10), and MBERT (Multi-passage BERT) (Reference 12). The experimental results are shown in FIG. 20.

[0127] 20, results 800, 802, and 804 show results from conventional systems. Result 800 shows a result from a system based on LSTM (Long Short-Term Memory) according to Reference 11. Result 802 is a combination of the system from Reference 11 and ARG introduced in Reference 6, which is also based on LSTM. Result 804 is from MBERT, which is the system considered to have the highest performance in the prior art.

[0128] In contrast, result 806 shows the result using the paragraph selector 766 and paragraph leader 770 of the second embodiment. Result 808 shows the result when BKRG is not used in the second embodiment. Result 810 shows the result when BKRG is used but the question-passage pairs are not updated. Result 812 shows the result when the SQuAD data described in Reference 13 is added to the training data and the paragraph selector 766 and paragraph leader 770 of FIG. 18 are trained. However, BKRG is not used in result 812.

[0129] For all evaluations, we used the EM score and the F1 score. EM indicates the percentage of predictions that matched exactly with any of the ground truths. F1 roughly indicates the average overlap between the predictions and the ground truth.

[0130] From this result, it can be seen that when the paragraph selector 766 and the paragraph reader 770 according to the second embodiment are used, performance is superior to all other conventional methods for all data sets. In particular, it can be seen that the second embodiment provides a significant performance improvement over MBERT, which is the best performing conventional technique. In addition, although the performance of both the result 808 and the result 810 is lower than that of the result 806, they still show higher performance than any of the conventional techniques. This shows that the use of BKRG and Furthermore, by updating the query-passage pairs that are input to BKRG, It can be seen that both of these contribute to the performance improvement of the second embodiment.

[0131] The experimental results of the first and second embodiments show that the background knowledge representation generator of the present invention exhibits higher performance than conventional techniques for different tasks in different languages, and is effective in question answering systems.

[0132] In the above description of the embodiment, BERT is used as the language representation model used in the text classifier for identifying answers, but the language representation model is not limited to BERT. For example, a language representation model based on the encoder block of a transformer or a similar network, such as XLNet (Reference 14), RoBERTa (Reference 15), ALBERT (Reference 16), or StructBERT (Reference 17), may be used.

[0133] [Computer implementation] Fig. 21 is an external view of a computer system for implementing each of the above-mentioned embodiments, and Fig. 22 is a hardware block diagram of the computer system shown in Fig. 21.

[0134] 21, this computer system 950 includes a computer 970 having a DVD (Digital Versatile Disc) drive 1002, and a keyboard 974, a mouse 976, and a monitor 972 for interacting with a user, all of which are connected to the computer 970. Of course, this is just one example of a configuration for user interaction, and any general hardware and software available for user interaction (e.g., a touch panel, voice input, or a general pointing device) can be used.

[0135] 22, the computer 970 includes, in addition to a DVD drive 1002, a CPU (Central Processing Unit) 990, a GPU (Graphics Processing Unit) 992, a bus 1010 connected to the CPU 990, the GPU 992, and the DVD drive 1002, a ROM (Read-Only Memory) 996 connected to the bus 1010 and storing a boot-up program and the like of the computer 970, a RAM (Random Access Memory) 998 connected to the bus 1010 and storing instructions constituting a program, a system program, working data, and the like, and a HDD (Hard Disk Drive) 1000 which is a non-volatile memory connected to the bus 1010. The HDD 1000 is for storing programs executed by the CPU 990 and the GPU 992, and data used by the programs executed by the CPU 990 and the GPU 992, and the like. The computer 970 further includes a network I / F 1008 that provides a connection to a network 986 enabling communication with other terminals, and a USB port 1006 to which a USB memory 984 can be attached / detached and that provides communication between the USB memory 984 and each component within the computer 970.

[0136] The computer 970 further includes an audio I / F 1004 that is connected to the microphone 982, the speaker 980, and the bus 1010, and that reads out audio signals generated by the CPU 990 and stored in the RAM 998 or the HDD 1000 in accordance with instructions from the CPU 990, converts them to analog, amplifies them, and drives the speaker 980, and also digitizes the analog audio signals from the microphone 982 and stores them in the RAM 998 or the HDD 1000 at any address specified by the CPU 990.

[0137] In the above embodiment, data and parameters of the fake representation generator 200, real representation 196, classifier 204, questions 190, background knowledge 192, etc. shown in Fig. 4 are all stored in, for example, the HDD 1000, RAM 998, DVD 978, or USB memory 984 shown in Fig. 22, or a storage medium of an external device (not shown) connected via the network I / F 1008 and network 986. Typically, these data and parameters are written into the HDD 1000 from outside, for example, and loaded into the RAM 998 when the computer 970 is executed.

[0138] A computer program for operating this computer system to realize the functions of the GAN 180 and each part thereof shown in FIG. 8, the training device by adversarial learning therefor, and each component thereof is stored in a DVD 978 mounted in a DVD drive 1002, and transferred from the DVD drive 1002 to a HDD 1000. Alternatively, this program is stored in a USB memory 984, and the USB memory 984 is mounted in a USB port 1006, and the program is transferred to a hard disk 1000. Alternatively, this program may be transmitted to a computer 970 via a network 986 and stored in the HDD 1000. The program is loaded into a RAM 998 at the time of execution. Of course, a source program may be input using a keyboard 974, a monitor 972, and a mouse 976, and an object program after compilation may be stored in the HDD 1000. In the case of a script language, a script input using a keyboard 974 or the like may be stored in the HDD 1000. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in the computer 970 in advance.

[0139] The CPU 990 reads a program from the RAM 998 according to an address indicated by an internal register called a program counter (not shown), interprets the instruction, reads data required for executing the instruction from the RAM 998, the hard disk 1000, or other devices according to an address specified by the instruction, and executes the process specified by the instruction. The CPU 990 stores the execution result data in the RAM 998, the hard disk 1000, a register in the CPU 990, or the like, at an address specified by the program. At this time, the value of the program counter is also updated by the program. The computer program may be directly loaded into the RAM 998 from the DVD 978, the USB memory 984, or via a network. Note that some tasks (mainly numerical calculations) among the programs executed by the CPU 990 are dispatched to the GPU 992 according to instructions included in the program or according to the analysis results obtained when the CPU 990 executes the instructions.

[0140] The program for implementing the functions of each unit according to each embodiment described above by the computer 970 includes a plurality of instructions written and arranged to operate the computer 970 to implement those functions. Some of the basic functions required to execute the instructions are provided by an operating system (OS) or a third-party program running on the computer 970, or by modules of various toolkits installed on the computer 970. Thus, the program does not necessarily include all of the functions required to implement the system and method of this embodiment. The program only needs to include instructions to execute the operations of each of the above-mentioned devices and their components by calling appropriate functions or functions of a "programming toolkit" in a controlled manner to obtain a desired result. The method of operation of the computer 970 for this purpose is well known, and will not be repeated here. The GPU 992 is capable of parallel processing, and can execute a large amount of calculations associated with machine learning in a parallel or pipelined manner simultaneously. For example, parallel computational elements discovered in a program when the program is compiled, or parallel computational elements discovered when the program is executed, are dispatched from the CPU 990 to the GPU 992 at any time and executed, and the results are returned to the CPU 990 directly or via a specified address in the RAM 998 and assigned to a specified variable in the program.

[0141] [Reference 1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs / 1810.04805. [Reference 2] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Joshua Bengio. 2014. Generative adversarial networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14, pages 2672-2680, Cambridge, MA, USA. MIT Press. Section 3 Rupesh K Srivastava, Klaus Greff, and Jurgen Schmidhuber. 2015. Training very deep networks. In Advances in Neural Information Processing Systems 28, pages 2377-2385. 〔Introduction 4〕 Jong-Hoon Oh, Kentaro Torisawa, Chikara Hashimoto, Motoki Sano, Stijn De Saeger, and Kiyonori Ohtake. 2013. Why-question answering using intra- and inter-sentential causal relations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1733-1743. 〔Introduction 5〕 Kazuma Kadowaki, Ryu Iida, Kentaro Torisawa, Jong-Hoon Oh, and Julien Kloetzer. 2019. Event causality recognition exploiting multiple annotators’ judgments and background knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5816-5822. [Reference 6] Jong-Hoon Oh, Kazuma Kadowaki, Julien Kloetzer, Ryu Iida, and Kentaro Torisawa. 2019. Open-domain why-question answering with adversarial learning to encode answer texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4227-4237. [Reference 7] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879. [Reference 8] Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904. 〔Reference 9〕 Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Gueney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs / 1704.05179. 〔Reference 10〕 Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601-1611. 〔Reference 11〕 Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, pages 1736-1745. 〔Reference 12〕 Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5878-5882. Association for Computational Linguistics. [Reference 13] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392. [Reference 14] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le, “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, [ONLINE], https: / / arxiv.org / abs / 1906.08237 [Reference 15] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, [ONLINE], https: / / arxiv.org / abs / 1907.11692 [Reference 16] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut,”ALBERT: A Lite BERT for Self-supervised Learning of Language Representations”, [ONLINE], https: / / arxiv.org / abs / 1909.11942 [Reference 17] Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, Luo Si, “StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding”, [ONLINE], https: / / arxiv.org / abs / 1908.04577 The embodiments disclosed herein are merely examples, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by the claims in the scope of the claims after taking into consideration the description of the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wording described therein. [Explanation of symbols]

[0142] 50, 190, 760 Questions 52 Answer Passage 60, 192 Background knowledge 70, 72, 74 Arrows 90 Text Classifier for Answer Identification 100, 340 input 102 BERT 104 KI Transformer 106 Output 108 Label 110 Start / End Position 112 Vector conversion section 114 BKRG 116 Vectors of Background Knowledge Representation 130, 132, 134 KI transformer layer 150, 154, 158 BKRG tier 152, 156 Update section 180 GAN 194 Realistic Expression Generator 196 Genuine Expression 198 Noise 200 Fake Expression Generator 202 Fake expression 204 Separator 220, 224 Word embedding vector conversion part 222 Word embedding vector sequence for questions 226 Noisy Word Embedding Vector Sequence 228 Attention Addition Section 230 Attentional Word Embedding Vectors 232CNN 240 BKRG Training System 250 Internet 252 Causal Relationship BKRG Training Department 254 Tool-Objective Relationship BKRG Training Department 256 Causal Relationship BKRG 258 Tool-Object Relationship BKRG 270 Causal Relation Extraction Unit 272 Causal Memory Device 274 Causal Relation Learning Data Generation Unit 276 Causal Learning Data Storage Device 278 Causal Relationship BKRG Training Department 280 Tool-Object Relationship Extraction Unit 282 Tool-Object Memory System 284 Tool-Object Relationship Learning Data Generation Unit 286 Tool-Object Relationship Learning Data Storage Device 288 Tool-Object Relationship BKRG Training Department 436, 476 Encoder Sub-Network 440, 492 1st layer subnetwork 442, 494 Layer 2 Subnetwork 450, 510 Multi-headed attention subnetwork 452, 462, 512, 522 ADD&Norm Subnetwork 460 fully connected subnetworks 544, 552 Linear transformation subnetworks 566 SoftMax layer 520 Feedforward Sub-network 540 Header section 542 Vector Connector 550 Header 560, 568 Matrix multiplication subnetwork 562 Division circuit 600 input word embedding vectors 602, 604, 606 BERT transformer layers 608 Word String 750 Tasks 762 Searches 764 Passage 766 Paragraph Selector 768 Paragraph Collection 770 Paragraph Leader 772 Answer Candidate Set 774 answers

Claims

1. A language expression model that receives a question text and an answer candidate text as input; a knowledge integration transformer that receives as input the output of the language expression model; a background knowledge representation generator that receives the question text and the answer candidate text as input and outputs a representation vector of background knowledge for the question text, The knowledge integration transformer is configured to receive the representation vector of the background knowledge as attention, and outputs a label indicating whether the answer candidate text contains an answer to the question text.

2. the knowledge integration transformer includes a plurality of knowledge integration transformer layers; the background knowledge representation generator outputs a plurality of representation vectors of the background knowledge corresponding to the plurality of knowledge integration transformer layers, respectively; The text classifier for identifying an answer according to claim 1 , wherein the plurality of knowledge integration transformer layers receive, as an information source for the attention, a representation vector corresponding to each of the plurality of background knowledge representation vectors.

3. The background knowledge representation generator includes: a background knowledge representation generator layer that outputs a representation vector of the background knowledge in response to input vectors representing questions and candidate answers; an update unit that updates an input vector to the background knowledge representation generator layer using a representation vector of the background knowledge output from the background knowledge representation generator layer to set the input vector to the background knowledge representation generator layer as a next input vector; 3. The text classifier for identifying an answer according to claim 1, wherein the update unit updates a preceding input vector to the background knowledge representation generator by using a degree of relevance between the representation vector of the background knowledge output by the background knowledge representation generator in response to the preceding input vector and the preceding input vector.

4. A computer program causing a computer to function as a text classifier for identifying an answer according to any one of claims 1 to 3.

5. A training device that uses a plurality of training data items to train a background knowledge representation generator that outputs a vector representation of background knowledge for a question represented by a question text, the plurality of training data items including a question text and background knowledge text related to the question text, a real expression generator consisting of a neural network, which generates a real expression vector in the same format as the vector representation of the background knowledge represented by the background knowledge text, when the question text and the background knowledge text are given; a fake expression generator comprising a neural network, which, when given the question text and a random noise vector representing a vector expression for an arbitrary text, outputs a fake expression vector having the same format as the vector expression for the background knowledge generated from the noise vector; A classifier consisting of a neural network for classifying the genuine expression vector and the fake expression vector; an adversarial training device that trains the genuine representation generator and the classifier by adversarial learning so as to minimize a classification error by the classifier, and trains the fake representation generator by adversarial learning so as to maximize a classification error by the classifier for the fake representation vector; A training device, wherein the fake representation generator when training by the adversarial training device is completed is the background knowledge representation generator after training, and the output of the fake representation generator when an actual question text and noise are input is a representation vector of background knowledge related to the actual question text.

6. A computer program causing a computer to function as the training device according to claim 5.

Images