Latent Query Reformulation and Information Accumulation for Multihop Machine Reading

Abstract

To provide question answering systems and methods that answer a question based on information extracted from a document.SOLUTION: A question answering system includes: a first encoder module configured to encode a question Q including a plurality of words into a first vector representation; a second encoder module configured to encode a document D including a plurality of words into a second vector representation; a first reading module configured to generate a third vector representation based on the first and second vector representations; a reformulation module configured to generate a reformulated vector representation based on the first vector representation; a second reading module configured to generate a fifth vector representation based on the second vector representation and the reformulated vector representation; and an answer module configured to determine an answer to the question based on the reformulated vector representation.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] CROSS REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 971,100, filed February 6, 2020. The entire disclosures of the referenced applications are hereby incorporated by reference.

[0002] <Technical field> The present disclosure relates to question answering systems and methods, and more particularly to multi-hop question answering using neural networks, attention mechanisms and bidirectional gated recurrent units. [Background technology]

[0003] The technical description of the background of the invention provided in this specification is intended to generally present the contents of the present disclosure. To the extent described in this technical background section, aspects of the description that are not admitted as prior art at the time of filing, as well as the work of the inventors of the present invention, are not expressly or implicitly admitted as prior art to the contents of the present disclosure.

[0004] The emergence of large-scale datasets such as CNN / Daily Mail, SQuAD, and MSMARCO has encouraged the development of machine-reading models that can outperform human readers. Separately, the widespread use of voice assistants has made question-answering tasks relevant to real-life applications. Unlike screen-based search engines, voice assistants are used to answer user questions based on data retrieved from the Internet.

[0005] The ability to automatically extract relevant information from large corpora is challenging. Question answering tasks can be used as proxy tasks to evaluate the reading comprehension ability of neural architectures. Question answering can be based on single-hop extraction machine reading. This includes the ability to read and extract information from a single text, which may contain several sentences.

[0006] The task of single-hop extraction machine reading may be described as follows: Given a document and an associated question, extract a range of documents that are responsive to the question. This may involve the machine reading model extracting two probability distributions. The first probability distribution may indicate the probability that each word in the document is the start word of a range that is responsive to the question. The second probability distribution may indicate the probability that each word in the document is the last word of a range that is responsive to the question. A prediction is then made as to which ranges of documents are responsive to the question by identifying the ranges that maximize such probabilities without violating certain rules. Example rules include: (a) the range start word must precede the last word, and (b) the range must not be longer than a predetermined length (e.g., a predetermined number of words).

[0007] It can achieve relatively high accuracy for easy questions that can be answered when similarities exist between the question and the text of the document. Although models can aim to detect patterns and named entities, they may lack inference capabilities. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Korean Patent Publication No. 10-2008-0095203 Summary of the Invention [Problem to be solved by the invention]

[0009] The present disclosure provides an improved method and system for multi-hop interrogation. [Means for solving the problem]

[0010] In one feature, a question answering system includes a first encoder module configured to receive a question from a computing device over a network, the question including a first plurality of words, and to encode the question in a first vector representation; a second encoder module configured to encode a document (D) in a second vector representation, the document including the second plurality of words; a first reading module configured to generate a third vector representation based on the first and second vector representations; a first reformulation module configured to generate a first reformulated vector representation for the question based on the first vector representation; a second reading module configured to generate a fifth vector representation based on the second vector representation and the first reformulated vector representation; a second reformulation module configured to generate a second reformulated vector representation for the question based on the first reformulated vector representation; and a response module configured to determine a response to the question based on the second reformulated vector representation and transmit the response to the computing device over the network.

[0011] In one feature, the question answering system includes a first encoder module configured to receive a question including a first plurality of words from a computing device over a network and encode the question in a first vector representation; means for encoding a document (D) in a second vector representation, the document including the second plurality of words; means for generating a third vector representation based on the first and second vector representations; means for generating a first reformulated vector representation for the question based on the first vector representation; means for generating a fifth vector representation based on the second vector representation and the first reformulated vector representation; means for generating a second reformulated vector representation for the question based on the first reformulated vector representation; and means for determining a response to the question based on the second reformulated vector representation and transmitting the response to the computing device over the network.

[0012] In one aspect, a method for answering questions includes: receiving, by one or more processors, a question from a computing device over a network, the question including a first plurality of words and encoding the question in a first vector representation; encoding, by one or more processors, a document (D) in a second vector representation, the document including the second plurality of words; generating, by one or more processors, a third vector representation based on the first and second vector representations; generating, by one or more processors, a first reformulated vector representation for the question based on the first vector representation; generating, by one or more processors, a fifth vector representation based on the second vector representation and the first reformulated vector representation; generating, by one or more processors, a second reformulated vector representation for the question based on the first reformulated vector representation; determining, by the one or more processors, a response to the question based on the second reformulated vector representation; and transmitting, by the one or more processors, the response to the computing device over the network.

[0013] In one aspect, a system for answering questions using documents includes a plurality of sequential hop components, a first component hop being sequentially coupled to a second component hop, (a) the first component hop including a first reading module coupled to a first reformulation module, (i) the first reading module configured to receive a question and the document and output a first question-aware representation of the document to the first reformulation module, (ii) the first reformulation module configured to extract elements from the first question-aware representation of the document to compute a first updated representation of the question, and (b) the second component hop including a second reading module coupled to a second reformulation module, (i) the second reading module configured to read elements of the question and output a first question-aware representation of the document to the first reformulation module, (ii) the first reformulation module configured to extract elements from the first question-aware representation of the document to compute a first updated representation of the question, (ii) a plurality of components configured to receive the first updated representation and the document and output a second question-aware representation of the document to a second reformulation module, the second reformulation module configured to extract elements from the second question-aware representation of the document and compute a second updated representation of the question; (B) a plurality of parallel heads, each of the plurality of parallel heads including an independent instance of a plurality of components of A sequential hops; and (C) a response module; the response module is configured to receive each of the updated representations of the document output by the plurality of parallel heads; and to determine a response to the question and a support factor within the document for the response to the question based on the updated representation of the document.

[0014] Additional areas of applicability of the present disclosure will become apparent from the particular details for carrying out the invention, the claims, and the drawings. The particular details and specific examples for carrying out the invention are intended for illustration only and are not intended to limit the scope of the present disclosure. [Brief description of the drawings]

[0015] The accompanying drawings are incorporated in and form a part of this specification for the purpose of explaining the principles of the embodiments. The drawings should not be construed as limiting the embodiments to only those embodiments shown and described. Additional features and advantages will become apparent from the following description of the embodiments and more particular parts thereof, as illustrated in the accompanying drawings. [Figure 1] FIG. 1 is a block diagram illustrating a neural network-based machine reading system for document-based question answering. [Diagram 2] FIG. 1 is a functional block diagram illustrating a method for answering questions based on documents. [Diagram 3] 1 illustrates an example of a system architecture in which the disclosed methods may be performed. [Figure 4] Two examples of multi-hop queries are included, considering questions (Q) and answers (A). [Diagram 5] FIG. 2 is a functional block diagram of an example implementation of the multi-hop model of FIG. 1. [Figure 6] FIG. 13 is a functional block diagram of an example implementation of a read module in a multi-hop model. [Figure 7] FIG. 13 is a functional block diagram of an example implementation of a multi-hop model reformulation module. [Figure 8] FIG. 2 is a functional block diagram of an example implementation of a response module of a multi-hop model. [Figure 9] Includes example questions, responses, and portions of documents before and after reformulation. [Figure 10] Includes example questions, responses, and portions of documents before and after reformulation. [Figure 11] FIG. 1 is a functional block diagram of an example of a question answering system. [Figure 12] FIG. 2 is a functional block diagram including an example implementation of a question answering module. [Figure 13] 1 includes a flowchart illustrating an example method for receiving a question and providing a response to the question. [Figure 14]1 is a functional block diagram of an exemplary training system.In the drawings, reference numbers may be reused to identify similar and / or same elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <Summary of the Invention> In one aspect, a system architecture is provided that enables a single-hop reader to sequentially read a concatenated set of documents to aggregate and reformulate the information to generate a response. To handle such sequential reasoning capabilities, in one aspect, a new, two-dimensional, fixed-sized latent representation of the question is formulated. During the response process, the representation of the question is updated using an attention mechanism that correlates document information and question reformulations in a latent space, which can also be called a controller.

[0017] In one aspect, a machine reading architecture is defined that includes multiple stacked reading blocks that sequentially collect (i.e., accumulate) information across a document and update their controllers. In some embodiments, an adaptive max pooling hierarchy is provided that compresses information from a variable length text sequence into a fixed size matrix. In one aspect, an extractive reading based attention mechanism is provided that computes an attention vector from the output hierarchy of the extractive machine reading model.

[0018] In one aspect, a neural network-based machine reading system for answering questions based on documents is provided, the system comprising: an input module configured to receive a question (Q) including a first plurality of words; Q and a second encoder configured to encode a document (D) including a plurality of words into a token representation E DEach of the first and second encoders uses one of word level embeddings, character level embeddings, and a combination of word level embeddings and character level embeddings. A predetermined number T of read blocks are provided, where T is an integer greater than 1, and the T read blocks are used to encode the encoded document E D A question E encoded using Q The first read block is configured to repeatedly convert the encoded document E D and the encoded question E Q Receive as input the converted query U 1 where each subsequent read block tt is an integer between 2 and T - is an encoded document E D and the output of the previous read block U t-1 Receive as input the converted query U t The response module is configured to output E D and the output of the last read block, U T and generating as an output a response to Q.

[0019] In one aspect, each read block t of the system generates two sets of probabilities (P t start and P t end ) where the response range is a range of words in the document (D) that are responsive to the question (Q).

[0020] In one aspect, each read block t of the system is t start and P t end Based on,E,, the given token is part of the response range. DAttention vector A, which is a vector of probability values ​​giving the probability for each given token. t The method further includes an extract attention module configured to compute:

[0021] In one aspect, each read block t of the system is E D Each token in corresponds to a probability value A t Weighted by E D Win the U t-1 and weighted E D Based on U t-1 U t The controller further includes a controller update module configured to update the controller.

[0022] In one aspect, the system Q Before being fed to the first read block, E Q an adaptive max pooling hierarchy configured to convert D E Q The controller further includes a further adaptive max pooling hierarchy in the controller update module of each read block t configured to convert the representation of t to a representation having the same fixed number of tokens as the representation of t.

[0023] In one aspect, the controller update module of each read block t in the system includes: D Weighted value tokens and U t-1 We apply the bidirectional gated recurrent unit BiGRU to the combination of the corresponding tokens of U t-1 U t The device is configured to update to .

[0024] In one aspect, the second encoder of the system generates an E based on the combined word-level embeddings and the character-level embeddings by applying BiGRU to the combined word-level embeddings and the character-level embeddings. DThe present invention is configured to calculate

[0025] In one aspect, the response module of the system determines two sets of probabilities (P T start and P T end ) and further comprising a three-way classifier configured to generate probabilities of "yes", "no" and range-based responses.

[0026] According to yet another aspect of the system, the responses generated are a "yes" response, a "no" response, and the highest P T start Starting from the word in the document with the highest P T end , the answer with the highest probability among range-based answers constructed from ranges ending with words in documents having

[0027] In one aspect, each reader t and the response module of the T readers both receive document E D and the transformed query U for each reader t. t-1 or the converted question U for the response module T It is configured to compute bi-attention between the readers t and self-attention of the documents, and the output of each reader t (P t start and P t end ) and the response output by the response module is based on the corresponding bidirectional attention and the corresponding self-attention computed by the reader t and the response module.

[0028] In one aspect, bidirectional attention and self-attention are based on a similarity matrix given by:

[0029]

number

[0030] In one aspect, the received questions that may be received from a user and / or the output responses that may be presented to a user are voice-based and / or the text and questions are related to diagnosing a technical or medical problem and / or the responses trigger automated robotic actions.

[0031] In one aspect, a method for answering questions based on documents is provided, the method including the steps of receiving a question (Q) from a user, the question (Q) including a first plurality of words; converting the question (Q) into a token representation E using one of word level embeddings, character level embeddings, and a combination of word level and character level embeddings; Q Then, the second document (D) containing multiple words is represented as token E. D A sequence of T read blocks is used to encode the data in E D Using E Q wherein the first read block is an encoded document E D and the encoded question E Q Receive as input the converted query U 1 Each subsequent read block t outputs an encoded document E D and the output of the previous read block U t-1 Receive as input the converted query U t where t is an integer between 2 and T. D and the output of the last read block, U T generating a response to Q from the input; and outputting the response to a user.

[0032] In one aspect, the method comprises: Q Before being fed to the first read block, E Q a representation with a fixed number of tokens, and a further adaptive max pooling hierarchy in the controller update module of each read block t to obtain a weighted E D E Q to a representation having the same fixed number of tokens as the representation of

[0033] In one aspect, the iterative transformation step generates, for each read block t, two sets of probabilities (P t start and P t end ) and outputting the answer range, where the answer range is the range of words in the document (D) that answer the question (Q); t start and P t end Based on,E,, the given token is part of the response range. D Attention vector A, which is a vector of probability values ​​giving the probability for each given token. t The stage of calculating E D The probability value A corresponding to each token in t Weighted by E D and U t-1 and weighted E D Based on U t-1 U t This includes updating the

[0034] In one feature, a question answering system includes a first encoder module configured to receive a question from a computing device over a network, the question including a first plurality of words, and to encode the question in a first vector representation; a second encoder module configured to encode a document (D) in a second vector representation, the document including the second plurality of words; a first reading module configured to generate a third vector representation based on the first and second vector representations; a first reformulation module configured to generate a first reformulated vector representation for the question based on the first vector representation; a second reading module configured to generate a fifth vector representation based on the second vector representation and the first reformulated vector representation; a second reformulation module configured to generate a second reformulated vector representation for the question based on the first reformulated vector representation; and a response module configured to determine a response to the question based on the second reformulated vector representation and transmit the response to the computing device over the network.

[0035] In one feature, the question answering system includes a first encoder module configured to receive a question including a first plurality of words from a computing device over a network and encode the question in a first vector representation; means for encoding a document (D) in a second vector representation, the document including the second plurality of words; means for generating a third vector representation based on the first and second vector representations; means for generating a first reformulated vector representation for the question based on the first vector representation; means for generating a fifth vector representation based on the second vector representation and the first reformulated vector representation; means for generating a second reformulated vector representation for the question based on the first reformulated vector representation; and means for determining a response to the question based on the second reformulated vector representation and transmitting the response to the computing device over the network.

[0036] In one aspect, a method for answering questions includes: receiving, by one or more processors, a question from a computing device over a network, the question including a first plurality of words and encoding the question in a first vector representation; encoding, by one or more processors, a document (D) in a second vector representation, the document including the second plurality of words; generating, by one or more processors, a third vector representation based on the first and second vector representations; generating, by one or more processors, a first reformulated vector representation for the question based on the first vector representation; generating, by one or more processors, a fifth vector representation based on the second vector representation and the first reformulated vector representation; generating, by one or more processors, a second reformulated vector representation for the question based on the first reformulated vector representation; determining, by the one or more processors, a response to the question based on the second reformulated vector representation; and transmitting, by the one or more processors, the response to the computing device over the network.

[0037] In one aspect, a system for answering questions using documents includes a plurality of sequential hop components, a first component hop being sequentially coupled to a second component hop, (a) the first component hop including a first reading module coupled to a first reformulation module, (i) the first reading module configured to receive a question and the document and output a first question-aware representation of the document to the first reformulation module, (ii) the first reformulation module configured to extract elements from the first question-aware representation of the document to compute a first updated representation of the question, and (b) the second component hop including a second reading module coupled to a second reformulation module, (i) the second reading module configured to read elements of the question and output a first question-aware representation of the document to the first reformulation module, (ii) the first reformulation module configured to extract elements from the first question-aware representation of the document to compute a first updated representation of the question, (ii) a plurality of components configured to receive the first updated representation and the document and output a second question-aware representation of the document to a second reformulation module, the second reformulation module configured to extract elements from the second question-aware representation of the document and compute a second updated representation of the question; (B) a plurality of parallel heads, each of the plurality of parallel heads including an independent instance of a plurality of components of A sequential hops; and (C) a response module; the response module is configured to receive each of the updated representations of the document output by the plurality of parallel heads; and to determine a response to the question and a support factor within the document for the response to the question based on the updated representation of the document.

[0038] In one feature, each of the first and second encoder modules is configured to perform one of word level encoding, character level encoding, and both word level embedding and character level encoding.

[0039] In an additional feature, the first reformulation module is configured to generate the first reformulated vector representation additionally based on the second vector representation.

[0040] In an additional feature, the second reformulation module is configured to generate a second reformulated vector representation additionally based on the second vector representation.

[0041] In an additional feature, the response module is configured to generate the response additionally based on the second vector representation.

[0042] In additional features, the max pooling module is configured to adjust a dimensionality of the first vector representation to a predetermined dimensionality.

[0043] In additional features, the third reading module is configured to generate a fifth vector representation based on the second vector representation and the second reformulated vector representation; the third reformulation module is configured to generate a third reformulated vector representation to the question based on the second reformulated vector representation, and the response module is configured to determine a response to the question based on the third reformulated vector representation.

[0044] In an additional feature, the first reading module is configured to determine (a) a first set of probabilities and (b) a second set of probabilities representing, respectively, (a) a probability that each word in the document is the beginning of a response range and (b) a probability that each word in the document is the end of the response range, where the response range is a range of words in the document that correctly respond to the question.

[0045] In an additional feature, the extract attention module is configured to determine an attention vector of probabilities that an entry of the second vector representation is part of a response range based on the first set of probabilities and the second set of probabilities.

[0046] In additional features, the first reformulation module is configured to generate a weighted second vector representation by applying weight entries of the second vector representation based on a value of the attention vector, and to determine the first reformulated vector representation based on the weighted second vector representation.

[0047] In additional features, the max pooling module is configured to adjust a dimensionality of the first vector representation to a predetermined dimensionality and the first reformulation module is configured to adjust a dimensionality of the weighted second vector representation to the predetermined dimensionality.

[0048] In additional features, the first reformulation module includes a bidirectional gate recurrent unit (BiGRU) and the second reformulation module includes a BiGRU.

[0049] In additional features, the second encoder module is configured to generate the second vector representation based on a combination of the word level encoding and the character level encoding.

[0050] In an additional feature, the response module is further configured to determine a first probability that the response is yes, a second probability that the response is no, and a third probability that the response is a range of words within a second plurality of words of the document.

[0051] In an additional feature, the response module is configured to set the response to "yes" if the first probability is greater than the second and third probabilities, set the response to "no" if the second probability is greater than the first and third probabilities; and set the response to a range of words if the third probability is greater than the first and second probabilities.

[0052] In an additional feature, the computing device is configured to at least one of display the response on a display and output the response through at least one speaker.

[0053] In an additional feature, the computing device is configured to receive the query through a microphone.

[0054] In an additional feature, the response module includes four bidirectionally gated recurrent units and four fully connected hierarchies.

[0055] In an additional feature, the training system includes: a question answering system; and a training module configured to train the question answering system by minimizing a first loss on support factors of responses to the question, a second loss on word ranges, and a third loss on classification of the responses.

[0056] <Details of the Invention>

[0057] There is a need for multi-hop inference. Multi-hop inference is desirable for answering questions in real-world situations, such as in personal terminals or search engines, which may include voice-based systems (e.g., "Show me movies by the director of Lord of the Rings"), or for the analysis of technical or medical problems. Multi-hop inference may involve the collection and inference of different evidence across multiple paragraphs.

[0058] This application includes systems and methods for answering queries based on documents. For purposes of explanation, numerous examples and specific details are presented in order to provide a thorough understanding of the described embodiments. The embodiments as defined by the claims may include any or all of the features of these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein. The illustrated embodiments will be described with reference to the drawings, in which like elements and structures are labeled with like reference numerals.

[0059] The present disclosure involves a multi-hop inference task, where each question may require information to be collected and integrated from multiple paragraphs to receive an answer. The disclosed system is tested on a dataset named HotpotQA. Natural language questions in this corpus are designed to require reading at least two Wikipedia paragraphs to be answerable. There are two types of questions: elicitation questions, where the answer is a given range of text in an English Wikipedia paragraph, and binary yes / no questions. In addition to the answer, the sentences that were used to generate the correct answer are predicted; these are called support factors.

[0060] Testing of the disclosed system and method may be performed in a setting called the destructor setting, where each question may have 10 relevant clauses, including 2 gold clauses. The gold clauses may be clauses necessary for answering the question. However, the disclosed system and method may also be used in the fullwiki benchmark setting and other settings. The fullwiki benchmark setting evaluates the ability of the model to answer questions given from the entire English Wikipedia dump when combined with an adapted searcher.

[0061] FIG. 1 illustrates an exemplary machine reading architecture. The architecture includes a predetermined number T of stacked reading blocks 115-1, ..., 115-T, and a response module 120 that aims to predict and output a final response 122 (which may be "yes", "no", or a range of responses for the document) together with supporting factors. Each reading block encompasses a reading phase performed through (by) a reader 116, an attention computation in an extract attention module 117, and a controller update in a controller update module 118. In each reading block t, a given document (D) 101 is read sequentially and a controller U tWe look for evidence about the current state of T, where t ∈ [1,T]. This approach can be seen as a sequential reformulation of the question in the latent space with respect to relevant information collected (i.e., accumulated) across documents.

[0062] 1 in more detail, a question (Q) 102 may be received from a user through an input module. The question (Q) 102 may be received from a computing device, such as a cellular phone, a tablet device, a laptop computer, a desktop computer, a home terminal device, or any other suitable type of computing device having a transceiver. A document (D) 101 related to the question may be retrieved from Wikipedia or other suitable databases through a searcher (such as a search engine) or provided to the retrieval architecture in any other manner.

[0063] A sequence of N words in question (Q) 102 and a sequence of L words in document (D) 101 may be defined as follows:

[0064]

number

[0065] <Encoding hierarchy> The encoder first looks up the lookup table

[0066]

number

[0067] Optionally, the encoder performs character-level encoding of each token with a 1D convolutional hierarchy to generate a character embedding for each token.

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072] The BiGRU layer receives a pair (including an input sequence and an output sequence) as input and includes two recurrent neural networks (RNNs). The BiGRU can also include an encoder RNN and a decoder RNN. The first RNN encodes the input sequence of symbols into a fixed-length vector representation (i.e., the final hidden state of the encoder RNN), and the second RNN decodes the fixed-length vector representation again into a variable-length sequence. The encoder RNN sequentially reads each symbol of the input and modifies its hidden state by applying a nonlinear activation function considering the read symbol to its previous hidden state. By using a nonlinear activation function that includes an update gate (which determines whether the hidden state is updated, if any) and a reset gate (which determines whether the previous gate is ignored), the final hidden state of the encoder RNN obtained at the end of the process (i.e., once the entire input sequence of symbols has been read) is a summary of the entire input sequence of symbols. The Decoder RNN is trained to generate an output sequence by predicting the next symbol of the output sequence given the current symbol and the current hidden state of the Decoder RNN, which is initialized to the final hidden state obtained through the Encoder RNN. The entire BiGRU layer generates a score for a given pair of input and output sequences, where the score is the probability of obtaining the output sequence given the input sequence. Cho et al., 2014 provides additional details on the role and architecture of the BiGRU layer.

[0073] <Adaptive Max Pooling Hierarchy> The adaptive max-pooling layer (adaptive max-pool) 113 can be optionally used to convert a sequence of L tokens encoding a query (Q) defined in an h-dimensional space (which can be the result of word encoding, character encoding, or a combination of both, regardless of the use of BiGRU) into a fixed-size representation of dimension (h×C) (where C is a given constant). The adaptive max-pooling layer 113 can be a max-pooling layer having an adaptive kernel size that depends on L, which is the original length of the query (Q). As a result, a matrix of fixed dimension appropriately represents the input without padding dimension, independently of the original length L.

[0074] Input matrix

[0075]

Number

[0076]

Number

[0077]

Number

[0078]

Number

[0079] In one feature, L≧C, and in such a case, the application of the adaptive max-pooling layer 113 leads to compression of the original representation by the adaptive max-pooling layer 113. In an alternative embodiment, when L<C, the sequence representation needs to be extended. Linear approximation can be used to calculate the missing tokens.

[0080] The adaptive max pooling operation dynamically adapts the size of the pooling window depending on the length L of the query (Q).

[0081] <Read block> After the encoding and optional transformations defined above, the input matrix

[0082]

number

[0083]

number

[0084]

number

[0085] <Reading block reader> At stage t, i.e., for a given read block t, the controller state U t-1 (which may be the controller state of the previous hop t-1), the reader 116 and the subsequent extract attention module 117 extract attention A for the input document (D). t Calculate A t is the token that is part of the response range. D is a vector of probability values ​​giving the probability for each given token in

[0086] The reader 116 may be an extractive reader and may include a complete extractive question-answer architecture such as the reader proposed in Clark and Gardner, "Simple and Effective Multi-paragraph Reading Comprehension," (Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL, Melbourne, 2018), the entire contents of which are incorporated herein by reference. Other suitable machine reading models may be used in various implementations. In some embodiments, the extractive architecture may first compute and consider question / document attention (which may be referred to as bidirectional attention) followed by document / document attention (which may be referred to as self-attention).

[0087] Bidirectional attention can be computed by the reader 116 between the (encoded) document (D) and the (encoded and optionally transformed) question (Q) via adaptive max pooling. At a read block t, bidirectional attention is computed by the controller U t-1 (because the state of Q represents the original question (Q) transformed by t-1 previous read blocks), this results in bidirectional attention between the document of read block t-1 and the controller state. Bidirectional attention can be computed using an attention flow hierarchy such as the one proposed in “Bidirectional attention flow for machine comprehension” (CoRR, abs / 1611.01603, 2016) (Seo et al.), the entire contents of which are incorporated herein by reference.

[0088] Specifically, to define bidirectional attention, a similarity matrix is ​​first defined as follows:

[0089]

number

[0090] Given a similarity matrix S, we calculate document-question attention (or context-question attention) c, which indicates which words in the query have the highest similarity to each word in the document. q can be given as:

[0091]

number

[0092]

number

[0093] Query-document attention (or question-context attention) (q c ) can be given as:

[0094]

number

[0095]

number

[0096]

number

[0097] The control-aware representation of a document can be calculated using the following formula:

[0098]

number

[0099] The same attention flow hierarchy can be applied between a document and itself to generate a self-attention Y as described in Wang et al., 2017, the entire contents of which are incorporated herein by reference.

[0100] Additional details regarding the definition and role of the concepts of bidirectional attention and self-attention can be found in Seo et al., 2016 and Wang et al., 2017, respectively, which describe examples for computing bidirectional attention and self-attention that can be used to implement the machine-readable systems disclosed herein.

[0101] Returning to Figure 1, reader t of reader block t calculates two sets of probabilities based on the previously calculated bidirectional attention and self-attention.

[0102]

number

[0103] Extract attention hierarchy for read blocks

[0104] In the extract attention module 117 of the read block t, P t start and P t end calculates the probability that each given word is part of the response range and uses this value as the attention vector A for the given word. t The probability that each word is included in the selected range is the probability that the range starts before this word and ends after:

[0105]

number

[0106] The extract attention module 117 can rescale the attention to be between 0 and 1, for example using the following formula:

[0107]

number

[0108] <Read block controller update module> The document encoding (E D ∈R h×N ), the previous state of the control (U t-1 ∈R h×C ) and attention vector (A t ∈R N), the purpose of the controller update module 118 is to generate an updated version of the controller (U t ∈R h×C The objective is to compute

[0109]

number

[0110] If the previously defined selective adaptive max pooling hierarchy 113 was used to convert the question (Q) to a fixed size representation, then an adaptive max pooling hierarchy is used again (not shown in FIG. 1) to convert V t ∈R h×C To

[0111]

number

[0112]

number

[0113] In this example, U 0 is (as shown in Figure 1) question E Q , may be initialized with an adaptive max pooling layer 113 applied to the encoding of

[0114] <Response module> The response module 120 receives the encoded document E D and the final state of the controller U T may receive as input. The response module 120 may be configured to compute the response output 122 to predict, for example, 1) whether the question is a binary question (i.e., a "yes / no" question) or an elicitation question (i.e., a question that can be answered with a range of documents (D), which are the response ranges described above); 2) a start probability of the response range; and 3) a final probability of the response range (which are used to determine the response range if the question is determined to be an elicitation question); and 4) a support factor, in that a set of probabilities is output that includes the probability that the response to the question is "yes", the probability that the response is "no", and the probability that the response is a range of documents (D). In one aspect, the response module 120 has a model architecture described in Yang et al., 2018, the entirety of which is incorporated herein by reference. The response module may include a three-way classifier embodied to generate the probability of having a response that is "yes", "no", or a range of documents (D).

[0115] The bidirectional attention of the reader described above is the final controller state U in the document and response module. T (which denotes the original question (Q) transformed by the T reading blocks). In one embodiment, bidirectional attention can also be computed using the attention flow hierarchy proposed in Seo et al., 2016.

[0116] One or more (e.g., four) BiGRU layers can be optionally stacked on top of this representation. The goal of the BiGRU layers is to derive the probability of each word in a document being a support factor (layer 1), a start (layer 2), and an end (layer 3). The fourth layer can be used to implement the three-way classifier described above to predict whether the response to a given question is yes, no, or a range of documents.

[0117]

number

[0118] For any additional details regarding the role of the response module and its architecture, and in particular regarding the manner in which the response type (yes / no / document range) is determined, the manner in which the start and end of the response range is determined, or the manner in which supporting factors are determined, see Yang et al., 2018, which describes examples of response modules that may be used to implement the machine-readable system disclosed in this specification.

[0119] <Multi-head version> In one aspect, a multi-head version of the architecture of FIG. 1 may be implemented. In this architecture, K independent read heads collect (store) information before providing the information to a response module 120, which is common to all read heads, while each read head corresponds to an independent implementation of the architecture of FIG. 1 up to the response module 120. The same number of read blocks, T, may be used for each read head. Thus, each read head may be configured to receive the final state U of the controller. T The K read heads generate K independent matrices U T Finally, K such matrices are summed to compute the input of response module 120. Heads will naturally produce different outputs, or two or more heads may produce the same output.

[0120] <Weak supervision of responses> In one feature, only the first occurrence of a response can be labeled positively and the others can be labeled negatively. In another feature, called weak guidance of responses, all occurrences of a response within a support factor can be labeled positively.

[0121] <Experimental setup and results> Table 1 provides the values ​​of the hyperparameters used during training of the model. We used pre-trained Glove word embeddings in the encoder 111 to encode the words of the questions (Q) and documents (D). We also optimized the model using stochastic gradient descent with an initial training rate of 0.5. As the learning rate schedule, we divide the learning rate by 2 when the loss on the development set does not decrease for two consecutive steps. As the stopping criterion, the training process is terminated when the learning rate becomes less than 0.005.

[0122] [Table 1] Table 2 presents the performance of the aforementioned multi-phase read architecture ("our model") on the HotpotQA dataset in its destructor setting. Exact match (EM) and F1 are the two evaluation metrics. Our model outperforms the given baseline models for response prediction ("Response") and support factor prediction ("Sup Fact").

[0123] [Table 2]

[0124] <Advantages> An ablation analysis was performed to evaluate the technical impact of the different components of the model described above. To empirically study the technical impact of sequential reads (i.e., the impact of T (the number of read blocks) on the final result), the model was compared to a similar architecture that does not sequentially update the controller state (i.e., has a single read block, meaning T=1).

[0125] The sequential association of read blocks and controller update modules provides an advantage: the F1 score decreases by 8.9 points for the response prediction task and by 5.7 points for the evidence extraction task when the model does not have the ability to reformulate the question. Thus, the sequential stacking of read blocks technically improves the efficiency of the system to answer questions correctly.

[0126] The impact of K (the number of parallel heads) on the efficiency of the system is less significant than one of T, but still significant. In fact, the configuration using only a single head (K=1) is one point lower than the better model (in terms of F1 score).

[0127] With respect to weak teaching for responses as defined above (i.e., a feature where all occurrences of the response within the support factors are labeled as positive), weak teaching also improves the efficiency of the system in that it increases the joint F1 score by 0.8 points.

[0128] The impact of the self-attention layer in the reading module on the efficiency of the model is significant: removing this layer reduces the F1 score by 8.9 points in the joint metric, which accounts for the ability to propagate long-distance information between different paragraphs, not just within the local neighborhood of a token.

[0129] Also, examples where the question representation is reduced to a vector instead of a matrix (i.e., the dimensionality of the fixed representation space is reduced to C=1) reduce the joint F1 score by 13.3 points, highlighting the importance of storing the question representation as a matrix instead of a vector in order to preserve its meaning.

[0130] <Example> The embodiment will now be described in detail in connection with the above and with reference to Figure 2, which is a flow chart illustrating a computer implemented method for answering a question (Q) based on a document (D). The method may be performed by one or more processors executing code. The illustrated examples may be combined together in a final implementation of the machine-readable system disclosed herein.

[0131] Control begins at 202, when a question (Q) 102 is received from a computing device, for example, over a network (e.g., the Internet). The question includes a set of words input by a user. For example, the question may be typed by the user into an input field of a user interface, or in a voice-based system, the question may be spoken by the user and received through a microphone. In the example of voice input, a language processing module may convert the speech to text, such as using natural language processing (NLP) or other suitable types of speech-to-text conversion.

[0132] In 204, a question (Q) 102 is represented as a representation E of the question in the embedding space, which is composed of a set of embeddings, one for each word of the question. Q The resulting ensemble is then encoded by an encoder such as encoder 112 of FIG. 1, which outputs: As mentioned above, the encoding can be performed at the word level, at the character-level (each word in the query that contains one or more characters), or at both the word level and the character-level. In the word-level and character-level example, both the embeddings from the word-level encoding and the character-level encoding for single words in the query can be combined. For example, pre-trained Grove word embeddings can be used for such purpose, or other suitable word and / or character embedding models can be used.

[0133] Similarly, an existing document (D) contains a set of words, which may include correct answers to a question (Q), and is encoded by an encoder, such as encoder 111 in Fig. 1. Encoder 111 stores in the embedding space a representation E of the document (D) consisting of a set of embeddings, one for each word in the document. D Again, encoding can be performed at the word level, at the character-level (each word in the document containing one or more characters), or at both the word level and the character-level. In the word-level and character-level examples, both the embeddings from the word-level encoding and the character-level encoding for a single word in the document can be combined. In various implementations, the same encoder (i.e., an encoder that uses the same encoding algorithm) can be used for both the query and the document. In other words, the same encoder can be used by encoders 111 and 112, and the resulting embeddings belong to the same embedding space. However, in various implementations, two separate encoders 111 and 112 can be used to encode the document and the query, respectively.

[0134] Optionally, at 206, an adaptive max pooling hierarchy 113, such as the adaptive max pooling hierarchy 113 of FIG. 1, Q The encoded question E Q (which is a variable size representation of dimension (h*L), where h is the dimension of the embedding / encoding that depends on the encoding algorithm chosen, i.e., a predetermined constant size of each token, and L is the (variable) length of the question) can be converted to a fixed size representation of dimension (h*C), where C is a predetermined constant. In this alternative embodiment, yet another adaptive max pooling hierarchy (which in one embodiment is Q (the same as that applied to t) is also applied to E DThe role of this is explained in more detail below.

[0135] 208, Representation of Document E D (with or without max pooling) and the current representation of the question (initially with or without max pooling E Q ), followed by the controller state U as output by each controller update module 118 of each read block 115-t. t (t is an integer between 1 and the total number of read blocks T) Q The transformed representation of is fed iteratively into a stack of T read blocks 115-1, ..., 115-T. 0 is the encoded query E with or without max pooling according to the embodiment. Q Thus, at each stage t, the reader t of the reading block 115 takes as input the encoded document E D and the updated value of the converted question, U t-1 and, as described above, the encoded document E D Two sets of probabilities (P t start and P t end ) is output. t start and P t end As mentioned above, the corresponding attention A t Finally, the document E D Attention A t and the representation (with or without max pooling) is input to a controller update module 118, which updates the transformed query value U t (also called the controller state), which is passed to the next read block via input U t+1. The final read block is the final value of the transformed question, U T which is transmitted to the response module 120. As mentioned above, multiple parallel encoders and read blocks may be fed into the response module 120.

[0136] At 210, the response module 120 generates a representation of the document E D (with or without max pooling) and the final value of the transformed question U T as input and outputs a set of probabilities, as described above, to answer the initial question (Q), including the probability that the answer to the question is yes, the probability that the answer to the question is no, and the probability that the answer is the range of the document. For each word in the document, the answering module 120 also outputs the probability that this word is the start word of the answer range and the probability that this word is the end word of the answer range. Finally, the answering module 120 outputs the support factors, i.e., the sentences used to predict the answer. The answering module 120 transmits the answer over the network to the computing device (that transmitted the question).

[0137] At 212, the computing device outputs a response 122 to the question to the user. For example, the computing device may display the response 122 on a display, output the response 122 through one or more speakers, or output the response 122 in any other suitable manner. In one embodiment, the response 122 may be one of the responses having the highest probability among three possible responses, which are yes, no, or a range of documents. If the question is determined to be answered by a range of answers (e.g., if the probability that the response is a range of documents is higher than the respective probabilities that the response is yes or no), the range of answers is determined to be a range of documents starting from the word that has the highest probability that this word is the start word of the range of answers and ending with the word that has the highest probability that this word is the end word of the range of answers. Optionally, the supporting factors output by the answering module 120 may be provided to the user along with the response to the question. The computing device may also output the supporting factors to the user. In various embodiments, the response (and optionally the supporting factors) may be provided to the user as a sentence created in a user interface on a display of the computing device. Additionally or alternatively, the responses (and optionally the support factors) may be provided to the user audibly through one or more speakers, although the machine reading system disclosed herein is not limited to this embodiment and any different alternative means for outputting the responses and presenting them to the user are further encompassed by this disclosure.

[0138] More generally, it should be noted that the above examples and illustrative embodiments are not meant to be limiting, and all alternative embodiments that one skilled in the art would consider to embody the same general idea are included within the scope of the present disclosure. Although some specific embodiments have been described in detail above, it will be apparent to those skilled in the art that various modifications, changes and improvements of the embodiments can be made in light of the above teachings and within the scope of the appended claims without departing from the intended scope of the embodiments. In addition, areas that are believed to be familiar to those skilled in the art are not described herein in order to avoid unnecessarily obscuring the embodiments described herein. Therefore, it should be understood that the embodiments are not limited by the specific illustrative embodiments, but are limited only by the scope of the appended claims.

[0139] Although the embodiments have been described in the context of method steps, they also represent a description of corresponding components, modules or features of corresponding apparatus or systems.

[0140] Some or all of the method steps may be computer embodied in that they are performed by (or using) a processor, microprocessor, electronic circuitry or processing circuitry.

[0141] The above-described embodiments may be implemented in hardware or software. Implementation may be performed using a computer readable storage medium, such as a non-transitory storage medium, for example, a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, and a flash memory. Such computer readable media may be any available medium that can be accessed by a general-purpose or special-purpose computer system.

[0142] In general, the embodiments may be embodied in a computer program product having program code or computer executable instructions operable to perform one of the methods when the computer program product is executed on a computer. The program code or computer executable instructions may be stored, for example, on a computer readable storage medium.

[0143] In one embodiment, a storage medium (or data carrier or computer readable medium) stores a computer program or computer executable instructions which, when executed by a processor, perform one of the methods described herein. In a further embodiment, an apparatus comprises one or more processors and the storage medium mentioned above.

[0144] In an additional embodiment, an apparatus includes means, e.g., processing circuitry, e.g., a processor in communication with a memory, configured or adapted to perform one of the methods described herein.

[0145] Further embodiments include a computer having the computer program or instructions configured to perform one of the methods described herein.

[0146] The methods and embodiments described above may be embodied within an architecture such as that illustrated in Figure 3, which includes a server 300 and one or more client devices 302 that communicate over a network 304 (which may be wireless and / or wired), such as the Internet, for exchanging data. The server 300 and the computing device 302 include a data processor (processor) 312 (e.g., 312a-312e) and a memory 313 (e.g., 313a-313e), such as a hard disk. The computing device 302 may be any type of computing device that communicates with the server 300, such as an autonomous vehicle 302b, a robot 302c, a computer 302d, or a mobile phone 302e.

[0147] In one example, the question (Q) 102 is provided to the server 300 by the client device 302. In various implementations, the question (Q) 102 is received directly by the server 300. The question (Q) 102 and the document (D) 101 are processed by the server 300, which returns a response to the question (Q) 102 to the client device 302 that transmitted the question (Q) 102. The server 300 can also, for example, select a document from a number of possible documents based on the question (Q) 102. In various implementations, the question (Q) 102 and the document (D) 101 can be processed by the client device 302 that initiated the query. Training of the model can also be performed by the server 300, by a different server, or by the client device 302.

[0148] As mentioned above, multi-hop text-based question answering involves the task of sequentially integrating factors from multiple phrases to answer complex natural language questions. Multi-hop parallel attention networks have been discussed above and are configured to perform question answering tasks including inference capabilities. The network includes a read and a reformulation / update module (one read and reformulation module per hop). Each read block as mentioned above is a hop. The output of one read block is input to the next read block. The read module generates a question-aware representation (e.g., a matrix representation) of the document. From such representation, the reformulation module extracts elements to compute an updated representation of the question. The updated question is then input to the subsequent hop / read block.

[0149] This application involves a reading block that collects information sequentially and in parallel across documents to answer questions. An input length-invariant question representation can be updated by a dynamic max pooling hierarchy that compresses information from variable length text sequences (e.g., questions) into a fixed size matrix. An extractive reading-based attention mechanism computes attention vectors from the output hierarchy of the extractive machine reading model.

[0150] Multi-hop question answering in this application involves both sequential and parallel inference. Sequential inference involves reading documents, asking for information, subsequently reformulating the question, and finally extracting the correct answer. Parallel inference involves gathering evidence for a comparison or question that involves checking various attributes of the documents.

[0151] FIG. 4 includes two examples of multi-hop queries given a question (Q) and an answer A. In each case, P1 and P2 are first and second paragraphs P that provide different information relevant to answering the question (Q). The top example of FIG. 4 illustrates parallel inference that compares information in the first paragraph (P1:Lit) with information in the second paragraph (A:Adorable) to determine an answer (A:Adorable) to the question (Q). The bottom example of FIG. 4 illustrates sequential inference that uses information from the first paragraph (P1:Apratim) to identify information from the second paragraph (P2:Sarod) that was used to determine an answer (A:stringed instrument) to the question (Q).

[0152] 5 is a functional block diagram of an exemplary implementation of the multi-hop model of FIG. 1. As mentioned above, the multi-hop model includes four modules: (1) an encoding module, (2) a reading module, (3) a question reformulation module, and (4) a response module. The encoding module is an input module, and the response module is an output module. A set of the reading module and the question reformulation module forms a hop. As mentioned above, the multi-hop model includes multiple (T, where T is at least 2) hops. The response module does not require reformulation.

[0153] Given a document and a question, the Read module computes a question-aware representation of the document. The Reformulation module extracts essential elements from this document representation and uses these to update the representation of the question in the latent space. The reformulated question is propagated to subsequent hops.

[0154] Multi-hop models can have multiple heads, as described above and below. In this case, the iteration mechanism is performed many times in parallel to compute a set of independent reformulations. The final representations of documents generated by the different heads are aggregated (e.g. by an aggregation module, similar to being summed by a summation module) before being fed to the answer module, which generates answers and support factors from the aggregated or reformulated questions of the final hops.

[0155] The multi-hop model involves K independent read heads processing documents and queries in parallel. For simplicity, not all matrix subscripts by head index are provided to focus on one explanation.

[0156] A canonical representation of each token is adopted herein using a pre-trained parametric language model (BERT). Let document (D) = {p1; p2; ...; p10} be a set of input sentences of length {n1; ...; n10} each associated with a question (Q) of length L. The sentences are encoded independently through an encoder such as the pre-trained BERT model. Each token is represented by its associated BERT hidden state from the last layer of the model. The token representations are combined to generate a global representation of the set of sentences of total length 10.

[0157]

number

[0158] The representation is then passed through an additional bidirectional gated recurrent unit (BiGRU) to obtain the final representation of the document.

[0159]

number

[0160]

number

[0161]

number

[0162] First expression of the question U (0) To compute

[0163]

number

[0164]

number

[0165] <Read module> The multi-hop model involves T hops of reading that sequentially extract relevant information from documents for the current reformulation of the question. At stage t, the reformulated question representation

[0166]

number

[0167]

number

[0168] In conjunction with the Document-Question Attention module, this module analyzes the current reformulation of documents and questions.

[0169]

number

[0170]

number

[0171]

number

[0172]

number

[0173]

number

[0174]

number

[0175] The reading module interprets the document as a question-recognition representation.

[0176]

number

[0177]

number

[0178] In conjunction with document self-attention, contextualization among the 10 sentences was performed by BiGRU as described above. A limitation of the current representation of the document is that each token may have very limited knowledge of other elements of the context. To handle long-distance dependencies, we use a query-aware representation of the document, X (t)and itself, the same attention mechanism is applied to the read module output.

[0179]

number

[0180] <Question reformulation module> A functional block diagram of an exemplary reformulation module is provided in Figure 7. The reformulation module t takes the output of the previous attention module (V (t) ), the previous expression of the reformulated question U (t) and the document encoding E D The reformulation module takes as input an updated reformulation of the question U based on the input. (t+1) Generate.

[0181] In relation to reading-based attention, V (t) Given, the reformulation module uses two BiGRUs followed by a linear hierarchy and a softmax operator to

[0182]

number

[0183]

number

[0184]

number

[0185]

number

[0186] The attention value can be used to recreate each token in the document representation.

[0187]

number

[0188]

number

[0189] <Dynamic max pooling> Use a dynamic max pooling module to add the current representation of dimension (M × 2h).

[0190]

number

[0191]

number

[0192]

number

[0193] Updated wording of question

[0194]

number

[0195] <Response module> A functional block diagram of an exemplary response module is provided in Figure 8. The response module can include four BiGRUs, each followed by a fully connected (fc) hierarchy. The BiGRUs (1) support factor (p sf ) (2) the probability that each word in the document is a response start and (3) a response end (p e , p s The final BiGRU can be used as a three-way classifier to predict the probability that a response will be classified as yes, no, or a range of text.

[0196]

number

[0197]

number

[0198] To predict the support factor, the response module constructs a sentence-based representation of the documents. Each sentence is represented by Y sf The response module uses a linear hierarchy followed by a sigmoid function to calculate p, the probability that sentence (j) is a support factor for yes (I). sf i、j Calculate the following.

[0199] <Multi-head version> The multi-hop model can include multiple heads. This implementation can include multiple independent parallel heads, as discussed above. Each head can include the same number of reading and reformulation modules. Each head can store a representation of the document (V k (T) The summation module can sum the k matrices generated by the k heads to generate the input for the response module.

[0200] <Training> The multi-hop model can be trained by jointly optimizing three subtasks: searching for support factors, identifying ranges, and classifying yes, no, or ranges. The training is

[0201]

number

[0202]

number

[0203]

number

[0204] All references to a response in a support factor can be assumed to be relevant to the question. The response module can tag the start and end positions of all occurrences of a response in a given support factor as valid solutions.

[0205]

number

[0206]

number

[0207]

number

[0208]

number

[0209]

number

[0210]

number

[0211]

number

[0212] 9 and 10 include example questions, answers, and document portions before and after reformulation. FIG. 9 and FIG. 10 illustrate that the multi-hop model described herein tends to follow natural inference paths to answer bridge questions, as illustrated in the top portion of FIG. 9. Before the first reformulation module, attention tends to focus on the first stage of inference. For the question "What award did the writer of Never Let Me Go novel win in 1989?", the model tends to focus on the name of the writer in the first stage before moving to the explanation of the award in the second stage / hop. In the example of FIG. 10, for the question "What is the population according to the 2007 population census of the city in which the National Archives and Library of Ethiopia is located?", the model focuses on Addis Ababa, i.e., the name of the city in which the National Archives and Library of Ethiopia is located, in the first stage, and subsequently moves to the population of this city in the next hop.

[0213] 11 is a functional block diagram of an example question answering system 1102 configured to answer questions within questions. The question answering system 1102 is configured to receive questions for one or more bodies of text (e.g., novels, news articles, documents, etc.) from one or more user computing devices 1104 over a network 1106. The bodies of text may be received along with the question or may be stored and retrieved based on the question.

[0214] The question answering systems 1102 read the bodies of text and determine from the bodies of text respective responses to the questions. The question answering systems 1102 may transmit the responses back to the computing device 1104 that transmitted the respective questions.

[0215] The computing device 1104 may display the responses to the user. The computing device 1104 may also display other information to the user. For example, the computing device 1104 may display additional information associated with each body of text, information associated with each response, etc. The question answering system 1102 and the computing device 1104 communicate over a network 1106.

[0216] A number of different types of computing devices 1104 are illustrated in Figure 11. The computing devices 1104 include any type of computing device configured to generate and transmit questions regarding text to the question answering system 1102 over the network 1106. Examples of the computing devices 1104 are provided above and illustrated in Figure 11, including, but not limited to, smart (cellular) phones, tablet computers, laptop computers, and desktop computers. The computing devices 1104 may also include other computing devices having other form factors, such as computing devices contained in vehicles, gaming devices, televisions, consoles (e.g., smart speakers without displays, Amazon Echo, Google Home, Clova Friends mini) or other appliances (e.g., refrigerators coupled to a network, thermostats coupled to a network, etc.).

[0217] The computing device 1104 can use a variety of different operating systems. In an example where the computing device 1104 is a mobile device, the computing device 1104 can run an operating system including, but not limited to, Android, iOS developed by Apple Inc., or Windows Phone developed by Microsoft Corporation. In an example where the computing device 1104 is a laptop or desktop device, the computing device 1104 can run an operating system including, but not limited to, Microsoft Windows, Mac OS, or Linux. The computing device 1104 can also access the question answering system 1102 while running operating systems other than those mentioned above, whether currently available or developed in the future.

[0218] In some examples, the computing device 1104 can communicate with the question answering system 1102 using an application installed on the computing device 1104. In general, the computing device 1104 can communicate with the question answering system 1102 using any application capable of transmitting a question to the question answering system 1102 to be answered by the question answering system 1102. In some examples, the computing device 1104 can execute an application dedicated to interfacing with the question answering system 1102, such as an application dedicated to question answering. In some examples, the computing device 1104 can communicate with the question answering system 1102 using a more general application, such as a web browser application. An application executed by the computing device 1104 to communicate with the question answering system 1102 can display a search field on a graphical user interface (GUI) into which a user can enter a question. A user can enter a question using a touch screen or a physical keyboard, a voice-to-text program, or other form of user input.

[0219] A question may include words, numbers, letters, punctuation marks, and / or symbols. In general, a question may be a request for information retrieval (e.g., a response) from the question answering system 1102. For example, a question may be directed to providing a response from a particular body of text that responds to the question.

[0220] The computing device 1104 may receive responses from the question answering system 1102 in response to a question transmitted to the question answering system 1102. In various embodiments, the computing device 1104 may receive, and the question answering system 1102 may transmit, more than one response in response to a question. In an example of a question answering system 1102 providing multiple responses, the question answering system 1102 may determine a confidence value for each individual response (indicating the likelihood that the response is correct) and provide the confidence value along with the response to the computing device 1104. The computing device 1104 may display more than one response of the multiple responses (e.g., all responses having a confidence value greater than a predetermined value), but only the response having the highest confidence value, etc.

[0221] The computing device 1104 may execute an application including a GUI that displays the responses received from the question answering system 1102. Respective confidence values ​​may also be displayed. For example, the application used to transmit the question to the question answering system 1102 may also present (e.g., display or speak) the received responses to a user. As previously mentioned, the application that presents the received responses to a user may in some instances be dedicated to interfacing with the question answering system 1102. In other instances, the application may be a more general application, such as a web browser application.

[0222] The GUI of the application executing on the computing device 1104 can display the answers to the user in a variety of different ways, depending on what information is transmitted to the computing device 1104. In an example where the search results include a list of answers and associated confidence values, the question answering system 1102 can transmit the list of answers and their respective confidence values ​​to the computing device 1104. In this example, the GUI can display the answers and the confidence values ​​to the user as a list of possible answers.

[0223] In some examples, the question answering system 1102 or other computing system may transmit additional information to the computing device 1104, such as, but not limited to, other information associated with the application and / or body of text, the question or answer, other bodies of text associated with the body of text, etc. This additional information may be stored in a data repository and, in some examples, transmitted to the computing device 1104 by the question answering system 1102. In examples where the user device 104 receives the additional information, the GUI may display the additional information along with the response. In some examples, the GUI may display the responses as a list of applications ordered by descending context value from the top of the screen to the bottom of the screen. In some examples, the responses may be displayed below a search field where the user entered the question.

[0224] As previously discussed, the computing device 1104 may transmit the body of text along with the question (for which the body of text) for which a response is to be generated. Alternatively, the question answering system 1102 may obtain the body of text from a data source 1120. The data source 1120 may include a variety of different text providers, and the like. The data source 1120 may include digital distribution platforms, such as, but not limited to, online news sources, websites, websites including weblogs (i.e., blogs), social networking sites (e.g., Facebook, Twitter, etc.), and / or other types of data sources.

[0225] The computing device 1104, the question answering system 1102, and the data source 1120 can communicate through a network 1106. The network 1106 can include various types of networks, such as a wide area network (WAN) and / or the Internet. Although the network 1106 can represent a long-range network (e.g., the Internet or a WAN), in some embodiments, the network 1106 can include a short-range network, such as a local area network (LAN). In one embodiment, the network 1106 uses standard communication technologies and / or protocols. Thus, the network 1106 can include links using technologies such as Ethernet, Wireless Fidelity (WiFi) (e.g., 802.11), Worldwide Interoperability for Microwave Access (WIMAX), 3G, 4G, 5G, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, and the like. Similarly, networking protocols used on network 1106 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged over network 1106 may be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. Additionally, all or some of the links may be encrypted using conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), Internet Protocol Security (IPsec), etc.In other examples, the network 1106 may use custom and / or proprietary data communications technologies instead of or in addition to those previously mentioned.

[0226] 12 is a functional block diagram including an example implementation of a question answering module 1200 of the question answering system 1102. The question answering module 1200 may include elements of FIGS. 1-5. A first transceiver module 1204 receives a query including a question from the computing device 1104. The transceiver module 1204 may also receive a body of text from the computing device 1104 or obtain the body of text in different manners.

[0227] The parsing module 1208 can parse the question to obtain the question to be answered from the body of text. In various embodiments, the parsing module 1208 can perform speech-to-text conversion. For example, the parsing module 1208 can identify a string of one or more words that form the question to be answered. The parsing module 1208 can also correct misspelled words in the question and / or perform one or more other word processing functions on the question. The parsing module 1208 can include the encoding, reading, and reformulation modules discussed above.

[0228] As discussed above, the response module 1212 determines a response to the question from the body of text. The response module 1212 includes the multi-hop model discussed above. The second transceiver module 1216 transmits the determined response to the question back to the computing device 1104. In various implementations, the second transceiver module 1216 may be omitted and the first transceiver module 1204 may transmit the response back to the computing device 1104 from which the question was received.

[0229] 13 includes a flow chart illustrating an example method for receiving a question and providing a response to the question. The example of FIG.

[0230] Control begins at 1304, where the question answering module 1200 receives a question from the computing device 1104. The question includes a question to be answered. The question may also include a body of text that is used as a basis for answering the question. At 1308, the question answering module 1200 parses the question to determine the question to be answered.

[0231] As previously described, the question answering module 1200 determines 1312 an answer to the question from the body of text using a multi-hop model. The question answering module 1200 also obtains the body of text identified in the question if the body of text is not transmitted by the computing device 1104. Alternatively, a predetermined source of text (e.g., Wikipedia) may be used. At 1316, the question answering module 1200 transmits the answer to the question back to the computing device 1104.

[0232] 14 includes a functional block diagram of a training module 1400. As previously described, the training module 1400 trains a multi-hop model that is used by the question answering module 1200 to determine responses to each received question. As previously described, the training module 1400 may train the multi-hop model using a training dataset 1404.

[0233] The foregoing description is merely exemplary in nature and is not intended to limit the disclosure, its application, or uses. The broad teachings of the present disclosure may be embodied in a variety of forms. Thus, while the present disclosure includes certain examples, the true scope of the present disclosure should not be limited thereto, as other variations will become apparent upon study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be performed in different orders (or simultaneously) without altering the principles of the present disclosure. Also, although each embodiment has been described above as having certain characteristics, any one or more of these features described in connection with any embodiment of the present disclosure may be embodied / combined with the features of any different embodiment, although the combination is not expressly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other are within the scope of the present disclosure.

[0234] Spatial and functional relationships between elements (e.g., modules, circuit elements, semiconductor layers, etc.) are described using a variety of terms, including "connected," "associated," "coupled," "adjacent," "adjacent," "on top of," "above," "below," and "disposed." When a relationship between a first and second element is described within the disclosure, unless expressly referred to as "direct," the relationship may be a direct relationship in which no other intermediate element exists between the first and second elements, but may also be an indirect relationship in which one or more intervening elements exist (spatially or functionally) between the first and second elements. As used herein, the phrases A, B, and at least one of C should be construed to mean the logic (A OR B OR C) using a non-exclusive logical OR, and should not be construed to mean "at least one of A, at least one of B, and at least one of C."

[0235] In the drawings, the direction of the arrow generally indicates the flow of information (e.g., data or commands) that is of interest in the example. For example, if element A and element B exchange various information, and the information sent from element A to element B is relevant to the example, an arrow may be shown from element A to element B. Such a unidirectional arrow does not imply that no other information may be sent from element B to element A. Also, in response to the information sent from element A to element B, element B may send a request for information or an acknowledgment of receipt to element A.

[0236] In this application, including the definitions below, the term "module" or "controller" may be substituted with the term "circuitry." The term "module" may refer to, be a part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinatorial logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of any or all of the above, such as in a system-on-chip.

[0237] A module may include one or more interface circuits. In some examples, the interface circuits may include a wired or wireless interface coupled to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed across multiple modules coupled through interface circuits. For example, multiple modules may allow for load balancing. In other examples, a server (also known as a remote or cloud) module may perform some functionality on behalf of a client module.

[0238] As used herein, the term code may include software, firmware, and / or microcode and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit includes a single processor circuit that executes some or all code from multiple modules. The term group processor circuit includes a processor circuit that executes some or all code from one or more modules in combination with additional processor circuits. References to multiple processor circuits include multiple processor circuits on separate dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations thereof. The term shared memory circuit includes a single memory circuit that stores some or all code from multiple modules. The term group memory circuit includes a memory circuit that stores some or all code from one or more modules in combination with additional memory.

[0239] The term memory circuit is a subset of the term computer readable medium. As used herein, the term computer readable medium does not include a transitory electrical or electromagnetic signal propagating through a medium (e.g., on a carrier wave); thus, the term computer readable medium can be considered tangible and non-transient. Non-limiting examples of non-transient types of computer readable media are non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or masked read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital magnetic tape or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray® disks).

[0240] The apparatus and methods described in this application may be embodied in part or in whole by a special purpose computer created by configuring a general purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flow chart components and other elements described above serve as software specifications that can be converted into a computer program by the routine work of a skilled engineer or programmer.

[0241] A computer program includes processor-executable instructions stored on at least one non-transitory type of computer-readable medium. A computer program may also include or depend on stored data. A computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0242] Computer programs may include: (i) parsed descriptive text, such as HyperText Markup Language (HTML), Extensible Markup Language (XML), or JavaScript Object Notation (JSON); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compiling and execution by a just-in-time (JIT) compiler, etc. For example, the source code may be created using syntax for languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java, Fortran, Perl, Pascal, Curl, OCaml, Javascript, HTML5 (Hypertext Markup Language Fifth Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hyperte×t Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash, Visual Basic, Lua, MATLAB, SIMULINK, and Python.

Claims

1. 1. A question answering system, comprising: a first encoder module configured to receive a question from a computing device over a network, the question including a first plurality of words, and to encode the question into a first vector representation; a second encoder module configured to encode a document (D) into a second vector representation, the document including a second plurality of words; a first reading module configured to generate a third vector representation based on the first vector representation and the second vector representation; a first reformulation module configured to generate a first reformulated vector representation for the question based on the first vector representation; a second reading module configured to generate a fifth vector representation based on the second vector representation and the first reformulated vector representation; a second reformulation module configured to generate a second reformulated vector representation for the question based on the first reformulated vector representation; and a response module configured to determine a response to the question based on the second reformulated vector representation and to transmit the response over the network to the computing device; A question answering system including:

2. 2. The question answering system of claim 1, wherein each of the first encoder module and the second encoder module is configured to perform one of word level encoding, character level encoding, and both word level embeddings and character level encoding.

3. The question answering system of claim 1 , wherein the first reformulation module is configured to generate the first reformulated vector representation further based on the second vector representation.

4. The question answering system of claim 1 , wherein the second reformulation module is configured to generate the second reformulated vector representation further based on the second vector representation.

5. The question answering system of claim 1 , wherein the response module is configured to generate the response further based on the second vector representation.

6. The question answering system of claim 1 , further comprising a max pooling module configured to adjust a dimensionality of the first vector representation to a predetermined dimensionality.

7. a third reading module configured to generate a fifth vector representation based on the second vector representation and the second reformulated vector representation; and a third reformulation module configured to generate a third reformulated vector representation for the question based on the second reformulated vector representation; Further comprising: The question answering system of claim 1 , wherein the answering module is configured to determine the answer to the question based on the third reformulated vector representation.

8. 2. The question answering system of claim 1, wherein the first reading module is configured to determine (a) a first set of probabilities and (b) a second set of probabilities that respectively represent (a) a probability that each word in the document is at the beginning of an answer span and (b) a probability that each word in the document is at the end of the answer span, the answer span being a range of words in the document that correctly answer the question.

9. 9. The question answering system of claim 8, further comprising an extract attention module configured to determine an attention vector of probabilities that an entry of the second vector representation is part of the answer range based on the first set of probabilities and the second set of probabilities.

10. 10. The question answering system of claim 9, wherein the first reformulation module is configured to generate a weighted second vector representation by applying weight entries of the second vector representation based on a value of the attention vector, and to determine the first reformulated vector representation based on the weighted second vector representation.

11. a max pooling module configured to adjust a dimension of the first vector representation to a predetermined dimension; The question answering system of claim 10 , wherein the first reformulation module is configured to adjust a dimensionality of the weighted second vector representation to the predetermined dimensionality.

12. The question answering system of claim 1 , wherein the first reformulation module comprises a bidirectional gated recurrent unit (BiGRU) and the second reformulation module comprises a BiGRU.

13. The question answering system of claim 1 , wherein the second encoder module is configured to generate the second vector representation based on a combination of word-level encoding and character-level encoding.

14. 2. The question answering system of claim 1, wherein the response module is further configured to determine a first probability that the response is yes, a second probability that the response is no, and a third probability that the response is a range of the word in the second plurality of words of the document.

15. The response module setting the response to yes if the first probability is greater than the second probability and the third probability; setting the response to no if the second probability is greater than the first probability and the third probability; The question answering system of claim 14 configured to set the response to the range of the word if the third probability is greater than the first probability and the second probability.

16. The computing device further comprises: The computing device comprises: displaying the response on a display; and outputting said response through at least one speaker; The question answering system of claim 1 , configured to do at least one of the following:

17. The computing device further comprises: The question answering system of claim 1 , wherein the computing device is configured to receive the question through a microphone.

18. 2. The question answering system of claim 1, wherein the answering module includes four bidirectional gate recurrent units and four fully connected hierarchies.

19. 1. A training system comprising: The question answering system of claim 1; and a training module configured to train the question answering system by minimizing a first loss related to support factors of answers to a question, a second loss related to word ranges, and a third loss related to classification of the answers; A training system including:

20. 1. A question answering system, comprising: means for receiving a question from a computing device over a network, the question including a first plurality of words, and encoding the question with a first vector representation; means for encoding a document (D) with a second vector representation, said document including a second plurality of words; means for generating a third vector representation based on the first vector representation and the second vector representation; means for generating a first reformulated vector representation for the question based on the first vector representation; means for generating a fifth vector representation based on the second vector representation and the first reformulated vector representation; means for generating a second reformulated vector representation for the question based on the first reformulated vector representation; and means for determining a response to the question based on the second reformulated vector representation and transmitting the response over the network to the computing device; A question answering system including:

21. 1. A question answering method, comprising: receiving, by the one or more processors, a query from a computing device over a network, the query including a first plurality of words, and encoding the query in a first vector representation; encoding, by the one or more processors, a document (D) with a second vector representation, the document including a second plurality of words; generating, by the one or more processors, a third vector representation based on the first vector representation and the second vector representation; generating, by the one or more processors, a first reformulated vector representation for the question based on the first vector representation; generating, by the one or more processors, a fifth vector representation based on the second vector representation and the first reformulated vector representation; generating, by the one or more processors, a second reformulated vector representation for the query based on the first reformulated vector representation; determining, by the one or more processors, a response to the question based on the second reformulated vector representation; and transmitting, by the one or more processors, the response over the network to the computing device.

22. 1. A system for answering questions using documents, comprising: (A) a plurality of sequential hop components, a first component hop being sequentially coupled with a second component hop; (a) the first component hop includes a first read module coupled to a first reformulation module; (i) the first reading module is configured to receive the question and the document and to output a first question-recognized representation of the document to the first reformulation module; (ii) the first reformulation module is configured to extract elements from the first question recognition representation of the document and compute a first updated representation of the question; (b) the second component hop includes a second read module coupled to a second reformulation module; (i) the second reading module is configured to receive the first updated representation of the question and the document and to output a second question-recognized representation of the document to the second reformulation module; (ii) the second reformulation module is configured to extract elements from the second question recognition representation of the document and compute a second updated representation of the question, the second reformulation module comprising: (B) a plurality of parallel heads, each of the plurality of parallel heads including an independent instance of a plurality of components of (A) the sequential hops; and (C) a response module; and the response module includes receiving updated representations of the document output by each of the plurality of parallel heads; The system is configured to determine, based on the updated representation of the document, an answer to the question and a support factor within the document for the answer to the question.

Images