Method and apparatus for multi-document question answering

Abstract

A computer implementation method for multi-document query answering is performed on a server that communicates with a client device over a network. The method includes the step of receiving a runtime query from a client device and using a search engine to search for runtime documents related to the runtime query. Runtime response samples are identified within the searched runtime documents. A neural network model trained using distance maps and distance-based rank loss is used to calculate runtime scores from runtime query data representing the runtime query and from runtime response samples representing a first portion of text from a corpus of documents, each runtime score representing the probability that a response to the runtime query exists in the runtime response samples. Runtime responses are selected from the runtime response samples corresponding to the highest runtime score transmitted to the client device.

Description

Technology Field

[0001] The field of the present disclosure relates to machine reading, and more specifically to multiple document question answering. Background Technology

[0002] Machine reading comprehension consists of methods that enable computer programs to read and understand human text and provide sensible output to the user.

[0003] Question answering is a subtask of machine reading comprehension, and models designed for this purpose provide satisfactory performance on short text passages when the goal is to find a response within a given passage. A drawback of these models is that they are not suitable for finding responses in larger text corpora because they are trained using single-document question-answer datasets that include the text of the question and response, as well as their location within the document. Another disadvantage of these models is that they always provide the location of a potential response within a given passage, and thus can identify a passage even if it does not contain a response.

[0004] The standard approach to addressing these problems is to use a search engine—often called a retriever—to return phrases potentially related to the query. These phrases can be documents (e.g., web pages) or paragraphs of documents. Then, to obtain possible responses, a machine reading model (typically trained on single-phrase query responses) is applied individually to each phrase. However, this approach introduces two new problems: (i) since machine reading models are always trained on phrases that contain a response, they always provide a response even if the phrase is irrelevant, in which case some proposed responses are not discernible for the original queries; and (ii) a response must be selected from among the proposed responses, out of all responses associated with the retrieved phrases.

[0005] Another drawback of standard machine reading comprehension models is that, in the case of multi-document question-answering, it is difficult to generate large datasets containing ground truths regarding results from search engines and the locations of responses within phrases. To overcome this problem, one possibility is to use queries from single-document question-answering datasets, such as SQuAD (described by P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang in "Squad: 100,000+ questions for machine comprehension of text," published on arXiv as preprint arXiv: 1606.05250, 2016), in conjunction with search engines that retrieve data from the corpus on which the dataset was built (part of Wikipedia in the case of SQuAD), so that each query can be associated with the ground truth documents and response locations within the corpus. However, a significant portion of SQuAD queries become ambiguous in multi-document settings. For example, the query "Where was the protagonist born?" becomes ambiguous when taken out of context. This statement appears to be true for most single-document datasets. Furthermore, in the context of multi-document query-answering, adding a comment to a response at a single location is of little significance, because when search engines operate on very large corpora such as the Internet, acceptable sentences leading to the correct answer are likely to appear more than once.

[0006] Therefore, there is a continuing need for improved multi-document query methods that use machine reading comprehension models. means of solving the problem

[0007] According to one aspect of the disclosed embodiments, a computer-implemented method for multi-document query response receives a runtime query from a client device on a server and searches for documents regarding the runtime query using a search engine. Additionally, the method identifies parts of text within the searched documents and calculates runtime scores for the identified parts of text using a neural network model trained using distance supervision and distance-based ranking loss. Additionally, the method selects parts of text corresponding to the highest scores to provide a response and transmits the response from the server to the client device.

[0008] According to another aspect of the disclosed embodiments, a computer-implemented method trains a neural network model to calculate a score representing the probability that a response to a predetermined query exists in a portion of text of a corpus of documents. The method comprises: (a) calculating a first training score using a neural network model from training query data representing a predetermined query and from a first training response sample representing a first portion of text from a corpus of documents, wherein the first training response sample is labeled as containing a response to the predetermined query; (b) calculating a first set of training scores by repeating (a) for a plurality of first training response samples, each labeled as containing a response to the predetermined query; (c) calculating a second training score using a neural network model from training query data and from a second training response sample representing a second portion of text from a corpus of documents, wherein the second training response sample is labeled as not containing a response to the predetermined query; (d) a step of calculating a second set of learning scores by repeating (c) for a plurality of second learning response samples, each labeled as not containing a response to a predetermined query; (e) using the first set of learning scores and the second set of learning scores, (i) determining a minimum score among the first set of learning scores; (ii) for each learning score of the second set of learning scores, calculating a learning term which is the difference between the determined minimum score and each learning score; (iii) a step of calculating a loss by summing the learning terms—the loss is the sum of the learning terms—; and (f) a step of updating a neural network model using the loss.

[0009] According to another aspect of the disclosed embodiments, a computer-implemented method performed on a server communicating with a client device over a network receives a runtime query from the client device and searches for runtime documents regarding the runtime query using a search engine. Additionally, the method identifies runtime response samples within the searched runtime documents and calculates runtime scores from runtime query data representing the runtime query and from runtime response samples representing a first portion of text from a corpus of documents using a neural network model trained using the prior method, each runtime score representing the probability that a response to the runtime query exists in the runtime response samples. Additionally, the method selects a runtime response from the runtime response samples corresponding to the highest runtime score and transmits the runtime response to the client device. Brief explanation of the drawing

[0010] The attached drawings are merely for illustrating various embodiments and should not be interpreted as limiting. Figure 1 illustrates an example of a system architecture in which a method for multi-document query response can be performed. Figure 2 is a flowchart of a method for training a neural network model for multi-document question answering. FIG. 3 illustrates in more detail an example of a neural network model learned using the method shown in FIG. 2. Figure 4 is a flowchart of a method for using a neural network trained using the method illustrated in Figure 2. Reference numbers in drawings can be reused to identify similar and / or identical elements. Specific details for implementing the invention

[0011] According to the present disclosure, a method for performing machine reading of questions and answers for multiple documents, also known as open domains or corpora, is disclosed. To this end, a neural network model is used to evaluate the probability that a given part of text contains an answer to a posed question.

[0012] More specifically, the present disclosure presents a method for training and using a neural network model for multi-document question answering, in which the reading task in question answering is defined as a search task. This advantageously allows the search score to be used as a measure of confidence. A neural network model trained using distance maps and distance-based rank loss combines an interaction matrix, Weaver blocks, and adaptive subsampling to map to a fixed-size representation vector, which is used to emit a score. The method presented in the present disclosure generally comprises the following aspects: (i) training parameters of a neural network model (i.e., a scoring model); and (ii) using the scoring model to determine the most relevant part of the text to answer a given query.

[0013] Advantageously, the disclosed method can learn from unlabeled data for a desired task by combining multiple datasets through distant maps. This enables the use of existing datasets and eliminates the tedious task of labeling data from the training set for specific problems.

[0014] A. System Architecture

[0015] The aforementioned modes of learning and use may be implemented within a system architecture as illustrated in FIG. 1 by a first server (10a) and / or a second server (10b). In one embodiment, the first server (10a) is a learning server (implementing a learning mode) and the second server (10b) is a scoring server (implementing a scoring mode). In other embodiments, the servers (10a, 10b) may be merged or their functions may be swapped.

[0016] Servers (10a, 10b) are generally remote computer equipment connected to an extended network (15), such as the Internet, for data exchange. The first server (10a) can store or access at least one large text corpus (20), for example, some or all of the contents of a web-based encyclopedia, and at least one query and answer database (22), for example, a TriviaQA dataset.

[0017] The system architecture of FIG. 1 advantageously comprises one or more items of client equipment (11), which may be any workstation (11c), robot (11d), or mobile device (11e) that is preferably separated from the servers (10a, 10b) but can be merged with one of them and / or another (these are also connected to the network (15)). Each server (10a, 10b) and client equipment (11c, 11d, 11e) each comprises a data processing means of the processor type (12a, 12b, 12c, 12d, 12e) and optionally, a storage means (13a, 13b, 13c, 13d, 13e) such as computer memory (e.g., hard disk).

[0018] The client device (11) has one or more queries for querying a large text or corpus stored in the first server (10a) to obtain responses within a set of identified documents. The operators of the client device (11) (i.e., “users”) are generally “clients” of a service provider operating the first and / or second servers (10a, 10b) in a commercial sense.

[0019] B. Training a Neural Network Model

[0020] Referring to FIG. 2, a method for training a neural network model for query-answering on multiple document corpora includes, for example, the step of executing a search engine (24) (illustrated in FIG. 1) in memory (13a) in step (101) using a processing means (12a) of a server (10a). The search engine (24) searches for a set of documents (202) from a corpus (20) using queries (defined as queries) (22). The documents (202) searched from the corpus (20) by the search engine are related to the input queries (22).

[0021] Next, the retrieved documents (202) are divided in step (102) into a set of sentences (203) or n-grams of sentences (n is a small number between, for example, 2 and 4) using the processing means (12b) of the server (10b), because a sentence is to convey one consistent piece of information and thus becomes the smallest self-consistent unit of text.

[0022] In step (103), response samples of positive sentences (204a) or positive n-grams of sentences are extracted from a set of sentences (203). These positive sentences (204a), or positive n-grams of sentences—which are sentences—or n-grams of sentences are labeled as containing responses to a given query.

[0023] In step (103), response samples of negative sentences (204b), or negative n-grams of sentences—which are randomly selected—are also extracted from the set of sentences (203). These negative sentences, or negative n-grams of sentences—which are sentences—or n-grams of sentences are labeled as not containing a response to a given query.

[0024] Next, in step (104), each of the positive sentences (204a) and negative sentences (204b) is processed by a scoring model (205) (i.e., a neural network model) to calculate scores (206a) for the positive sentences and scores (206b) for the negative sentences using, for example, the processing means (12b) of the server (10b).

[0025] In step (105), a distance-based ranking loss (i.e., learning loss) is calculated using the scores of positive sentences (204a) and negative sentences (204b). This distance-based ranking loss aims to assign higher scores to sentences containing a response to a given query than to sentences that do not contain a response to a given query, in order to optimize the scoring model (205).

[0026] In step (107), the distance-based rank loss is used to update the scoring model (205) using a stochastic gradient descent algorithm. Examples of gradient descent algorithms include AdaGrad, RMSProp, and Adam.

[0027] Referring again to FIG. 1, in addition to the corpus (20) and queries (22), documents (202), sentences (203, 204a, 204b), models (205), and scores (206a, 206b) can be stored, for example, using a storage means (13b) of a server (10b).

[0028] C. Neural Network Model

[0029] FIG. 3 illustrates in more detail an embodiment of a scoring model (i.e., a neural network model) used to score sentences learned using the method illustrated in FIG. 2. In order to score each sentence for relevance in step (104), the input query and input sentence are tokenized using word embeddings (i.e., as a set of language modeling and feature learning techniques of natural language processing, words from a vocabulary, and possibly their syntax, are mapped to vectors of real numbers in a space of lower dimensions relative to the vocabulary size). The calculation of word embeddings may be performed using 300-dimensional fastText embeddings or any other method of calculating word embeddings, for example, using the processing means (11a or 11b) of a server (10a or 10b). The embeddings for each word of the input query are a matrix It gathers at, where n is the number of words in the input query, and is the i-th word of the input query, and the token embeddings for each word of the input sentence are a matrix It gathers at, where m is the number of words in the input sentence, and is the i-th word of the input sentence.

[0030] The scoring model in step (104) of FIG. 2 is adopted from the Weaver model for machine reading, in which response parts are removed and different pooling layers are added to reduce variable-size tensors to fixed-size tensors, followed by a fully connected neural network (FCNN), such as a multilayer perceptron (MLP). The Weaver model for machine reading was disclosed by Martin Raison, Pierre-Emmanuel Mazare, Rajarshi Das, and Antoine Bordes in "Weaver: Deep co-encoding of questions and documents for machine reading" submitted to arXiv.org on April 27, 2018 (arXiv reference arXiv:1804.10490).

[0031] The input to the neural network model (i.e., the scoring model) (205) is an interaction tensor M between the input query matrix Q and the input sentence matrix S, and It can be calculated as follows, where

[0032] 0 <i<n,

[0033] 0 <i<m,

[0034] In this case, u is q or s, and

[0035] operator represents the element-wise product, and

[0036] ; indicates concatenation for the last dimension,

[0037] Provides a tensor of the form, where e is a connected vector It is the dimension of. These interaction tensors can be grouped into mini-batches and are therefore padded with the largest query and statement sizes in the mini-batches.

[0038] Next, the interaction tensor M is input into a stack of k weaver blocks in step (301) of FIG. 3. These weaver blocks consist of two BiLSTM (Bidirectional Long Short-Term Memory) layers each along one of the first two dimensions of the interaction tensor M. The BiLSTM hidden sizes are selected so that the input and output dimensions are the same (except for the first layer of the first block). In step (302), the output of the last block (the k-th block) is Tensor of the form and, where h is the output size of the last BiLSTM layer, and m and n are variable depending on the size of the input interaction tensor.

[0039] After calculating the weaver blocks, in step (303), the tensor is along the dimensions of sentences It is pooled into a fixed-size tensor T of the form. This pooling operation is similar to dynamic pooling but is performed using an adaptive subsampling layer that performs only subsampling rather than pooling by regions. Dynamic pooling is disclosed in "Dynamic pooling and unfolding recursive autoencoders for paraphrase detection" by R. Socher, EH Huang, J. Pennin, CD Manning, and AY Ng in *Advances in neural information processing systems*, pp. 801-809, 2011. Since the input interaction tensors M are padded in the sentence and query dimensions according to the length of the query and the longest sentences in the mini-batch, Subsampling for occurs in the region corresponding to the original size, rather than the entire padded size. To this end, a grid application layer, such as the one designed for Spatial Transformer Networks, is used. Spatial Transformer Networks are disclosed in "Spatial transformer networks" by M. Jaderberg, K. Simonyan, A. Zisserman, et al. in *Advances in neural information processing systems*, pp. 2017-2025, 2015. The output elements of this grid application layer are It corresponds to the bilinearly sampled values ​​of grid points projected onto the unpadded area.

[0040] In step (304), the generated fixed-size tensor T is passed to a fully connected neural network (FCNN) containing two fully connected layers. The hidden layer has a Rectified Linear Unit (ReLU) activation function. The last layer has a sigmoid activation function with an output size of 1 and outputs an estimate of the distance between Q and S in step (305). Using a sigmoid activation function as the last layer makes the model more robust because it is a fixed-range nonlinearity, otherwise the network outputs could be scaled, causing the concept of margin for output distance to lose its meaning.

[0041] D. Distant Supervision

[0042] To overcome the problems of impossibility in constructing large-scale multi-document datasets, the method for multi-document query answering of the present disclosure uses a distant map.

[0043] In the method of the present disclosure, a distant map implies using two different datasets: one is a question-answer dataset containing queries and their associated responses, and the other is corpus information required to respond to the queries. Subsequently, every sentence in the corpus is labeled positive if it contains the exact text of the response, and negative otherwise. Of course, the fact that a sentence contains the text of the response does not guarantee that the sentence is satisfactory for responding, but this enables a first choice. Additionally, the use of a distant map can enable existing datasets to be combined to train a model to perform a task for which the dataset does not exist beforehand.

[0044] E. Distance Based Ranking Loss

[0045] Scoring models used for the purpose of scoring sentences are essentially distance estimation models. Therefore, the task of estimating the confidence that a response pertains to a given part of a text can be regarded as a ranking problem in information retrieval, and thus, models can be trained using retrieval losses. To train models for ranking information retrieval, two common losses that can be used are contrastive loss and triplet loss. However, these losses are not suitable for handling the presence of noisy labels, as is the case with distance maps. In fact, if a response text appears more than once within documents retrieved by a search engine, it may be because some of the sentences containing text similar to the response do not actually accept the response. To account for this possibility, a distance-based ranking loss, also known as n-uplet loss, can be used, which takes all positive sentences and an equal number of sampled negative sentences as input. This distance-based ranking loss uses a scoring model as a distance function and aims to associate smaller distances (i.e., scores) for input data containing positive sentences than for input data containing negative sentences.

[0046] n-uplet loss allows for the occurrence of some positives that are unrelated to the query. However, the requirement is that the best positive sentences obtain a higher score than all negative sentences; that is, n-uplet loss penalizes only when there exists a negative sentence that is closer to the query than all positives. Therefore, vector representations of query Q, a list of positive sentences , and a list of negative sentences Given this, and i is the index of the sentence, the n-uplet loss is calculated as follows:

[0047]

[0048] Here is a scoring model, and represents the margin parameter.

[0049] F. Runtime Usage of Neural Network Models

[0050] FIG. 4 is a flowchart of a runtime (or test) method that can be performed on the system exemplified in FIG. 1 using a scoring model (205) trained according to the method of FIG. 2. Generally, FIG. 4 describes a method for using a reader network, the method comprising: receiving a runtime query(s) (23) from a client device (11) at a server (10) which may or may not be about text in a corpus (20); searching for documents (202) regarding the runtime query using a search engine (24); identifying sentences (102), or more generally parts of text, within the searched documents; calculating a runtime score (402) for the identified sentences using a neural network model (205) trained using a distance map and distance-based rank loss; selecting a sentence corresponding to the highest score to provide a response (404); and transmitting the response to the client device (11).

[0051] More specifically, during runtime (or testing), some of the elements are the same as in training, but some are different. Similar to the training method illustrated in FIG. 2, during runtime, queries or queries in 22 are used by the search engine (24) in step (101) of FIG. 4 to query the text or corpus (20) to obtain a set of documents (202), and similarly in step (102), the set of documents (202) is divided into a set of sentences (203). However, unlike during training time when scores (206) are calculated, in step (106) during runtime, scores (402) are calculated for the set of sentences (203) using a scoring model (205) (trained according to the method of FIG. 2). More specifically, at training time, a distance model is applied using one selected positive sentence (206a) and one selected negative sentence (206b) to obtain estimated distances, while at runtime, the distance model (205) is applied to all candidate sentences (203) to obtain estimated distances used as scores (402). After calculating a runtime response in step (404) based on the sentences associated with the highest runtime scores calculated in step (402), the predicted response, which includes the identified sentences (203) associated with the documents (202) from the corpus (20), is returned in response to a runtime query (e.g., from the server (10a) to the client device (11).

[0052] G. General

[0053] Accordingly, the present disclosure describes a computer implementation method for multi-document query-answering performed on a server (11) communicating with a client device (12) via a network (15). The method comprises: (A) receiving a runtime query (23) from a client device (12); (B) searching for runtime documents (202) regarding the runtime query using a search engine (24); (C) identifying runtime response samples (203) within the searched runtime documents (202); and (D) calculating runtime scores (402) using a neural network model (205) from runtime query data representing the runtime query (23) and from runtime response samples (203) representing a first portion of text from a corpus (20) of documents—each runtime score (402) represents the probability that a response to the runtime query (23) exists in the runtime response samples (203)—; (E) a step of selecting a runtime response (404) from runtime response samples (203) corresponding to the highest runtime score (402); and (F) a step of transmitting the runtime response (404) to a client device.

[0054] Also, the method calculates a first learning score (206a) using a neural network model (205) from a first learning response sample (204a) representing a first portion of text from a corpus (20) of documents, from learning query data representing a predetermined query (22), wherein the first learning response sample (204a) is labeled as containing a response to the predetermined query; and (b) calculates a first set of learning scores (206a) by repeating (a) for a plurality of first learning response samples (204a), each labeled as containing a response to the predetermined query (22); (c) Calculate a second learning score (206b) using a neural network model (205) from a second learning response sample (204b) representing a second portion of text from the learning query data and from a corpus of documents, wherein the second learning response sample (204b) is labeled as not containing a response to a predetermined query; (d) Calculate a second set of learning scores (206b) by repeating (c) for a plurality of second learning response samples (204b), each labeled as not containing a response to a predetermined query (22); (e) Using the first set of learning scores and the second set of learning scores, (i) determine a minimum score among the first set of learning scores (206a); (ii) For each learning score of the second set of learning scores (206b), calculate a learning term which is (or depends thereon) the difference between the determined minimum score and each learning score (206b); (iii) calculating the loss by summing the learning terms - the loss is the sum of the learning terms (or depends thereon) -; (f) training the neural network model by updating the neural network model (205) using the loss.

[0055] Additionally, the present disclosure describes a computer implementation method for finding entries in a Q&A database (e.g., an encyclopedia database) that associate a predetermined query with a predetermined response. A predetermined response within a corpus of documents is retrieved using a search engine. Response samples from the corpus of documents to be input into a neural network model are labeled, and (i) whenever a predetermined response is found in a part of text by the search engine, they are labeled as containing a response to the predetermined query, and (ii) whenever a predetermined response is not found in a part of text by the search engine, they are labeled as not containing a response to the predetermined query. Each part of text may be a paragraph or shorter than a paragraph, and may be, for example, a sentence.

[0056] The devices and methods described herein may be partially or fully implemented by a special-purpose computer created by configuring a general-purpose computer to execute one or more specific functions implemented by computer programs. The aforementioned function blocks, flowchart components, and other elements serve as software specifications that can be converted into computer programs by the routine work of a skilled technician or programmer.

[0057] Computer programs include processor-executable instructions stored on at least one non-transient, tangible computer-readable medium. Computer programs may also include or rely on stored data. Computer programs may include a Basic Input / Output System (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0058] It will be understood that variations of the embodiments disclosed above and other features and functions, or alternatives thereof, may preferably be combined into many different systems or applications. Furthermore, various currently unexpected or unintended alternatives, modifications, variations, or improvements therein may subsequently be made by those skilled in the art, and these are also intended to be included by the foregoing description and the following claims.

Claims

Claim 1 A computer implementation method for training a neural network model to calculate a score representing the probability that a response to a predetermined query exists in a portion of text from a corpus of documents, comprising: (a) calculating a first training score using the neural network model from training query data representing a predetermined query and from a first training response sample representing a first portion of text from the corpus of documents, wherein the first training response sample is labeled as containing a response to the predetermined query; (b) calculating a first set of training scores by repeating (a) for a plurality of first training response samples, each labeled as containing a response to the predetermined query; (c) calculating a second training score using the neural network model from training query data and from a second training response sample representing a second portion of text from the corpus of documents, wherein the second training response sample is labeled as not containing a response to the predetermined query; (d) a plurality of second A computer-implemented method comprising: (c) repeating for training response samples to calculate a second set of training scores; (e) using the first set of training scores and the second set of training scores, (i) determining a minimum score among the first set of training scores; (ii) for each training score of the second set of training scores, calculating a training term which is the difference between the determined minimum score and each training score; (iii) calculating a loss by summing the training terms, wherein the loss is the sum of the training terms; and (f) updating the neural network model using the loss. Claim 2 A computer-implemented method according to claim 1, wherein the step of calculating the loss further comprises the step of summing the difference between a predetermined margin and the determined minimum score and the second learning score. Claim 3 A computer implementation method according to claim 1, wherein the neural network model is trained using an adaptive subsampling layer configured to convert a first matrix of variable size into a second matrix of fixed size of n×m, the size of the first matrix is ​​larger than the size of the second matrix, and the training using the adaptive subsampling layer further comprises: partitioning the rows and columns of the first matrix into n and m parts, respectively; and subsampling one value from each partition to generate the n×m second matrix. Claim 4 A computer-implemented method according to claim 3, wherein the neural network model comprises: a first block comprising at least one layer; a second block comprising the adaptive subsampling layer arranged at the output of the first block; and a fully connected neural network (FCNN) arranged at the output of the second block. Claim 5 A computer implementation method according to claim 4, wherein the neural network model is trained using a BiLSTM layer for at least one layer of the first block. Claim 6 A computer implementation method according to claim 1, wherein the learning is distant supervision learning. Claim 7 A computer-implemented method according to claim 1, further comprising: a step of finding an entry in a Q&A database that associates the predetermined query with the predetermined response; a step of searching for the predetermined response in a corpus of documents using a search engine; and a sample of responses from the corpus of documents to be input to a neural network model, wherein the sample is labeled as containing the response to the predetermined query whenever the predetermined response is found in a part of text by the search engine, and the sample is labeled as not containing the response to the predetermined query whenever the predetermined response is not found in a part of text by the search engine. Claim 8 A computer implementation method in paragraph 7, wherein each part of the text is a paragraph or a sentence shorter than a paragraph. Claim 9 In paragraph 8, a computer implementation method in which the corpus of the above documents is an encyclopedia database. Claim 10 A computer-implemented method performed on a server communicating with a client device over a network, comprising: (A) receiving a runtime query from a client device; (B) searching for runtime documents regarding the runtime query using a search engine; (C) identifying runtime response samples within the searched runtime documents; (D) calculating runtime scores using a neural network model from runtime query data representing the runtime query and from runtime response samples representing a first portion of text from a corpus of documents, wherein each runtime score represents the probability that a response to the runtime query exists in the runtime response samples; (E) selecting a runtime response from the runtime response samples corresponding to the highest runtime score; and (F) the step of transmitting the runtime response to the client device, wherein the neural network model comprises: (a) calculating a first learning score using the neural network model from learning query data representing a predetermined query and from a first learning response sample representing a first portion of text from the corpus of documents, wherein the first learning response sample is labeled as containing a response to the predetermined query; (b) calculating a first set of learning scores by repeating (a) for a plurality of first learning response samples, each labeled as containing a response to the predetermined query; (c) calculating a second learning score using the neural network model from a second learning response sample representing a second portion of text from the learning query data and from the corpus of documents, wherein the second learning response sample is labeled as not containing a response to the predetermined query; (d) calculating a second set of learning scores by repeating (c) for a plurality of second learning response samples, each labeled as not containing a response to the predetermined query;(e) a computer-implemented method, wherein, using the first set of learning scores and the second set of learning scores, (i) determine a minimum score among the first set of learning scores; (ii) for each learning score of the second set of learning scores, calculate a learning term which is the difference between the determined minimum score and each learning score; (iii) calculate a loss by summing the learning terms—the loss being the sum of the learning terms—; (f) learn by updating the neural network model using the loss. Claim 11 A computer implementation method according to claim 10, wherein calculating the loss further comprises summing the difference between a predetermined margin and the determined minimum score and the second learning score. Claim 12 A computer implementation method according to claim 10, wherein the neural network model is trained using an adaptive subsampling layer configured to convert a first matrix of variable size into a second matrix of fixed size of n×m, the size of the first matrix is ​​larger than the size of the second matrix, and the training using the adaptive subsampling layer further comprises: partitioning the rows and columns of the first matrix into n and m parts, respectively; and subsampling one value from each partition to generate the n×m second matrix. Claim 13 A computer-implemented method according to claim 12, wherein the neural network model comprises: a first block comprising at least one layer; a second block comprising the adaptive subsampling layer arranged at the output of the first block; and a fully connected neural network (FCNN) arranged at the output of the second block. Claim 14 A computer implementation method according to claim 13, wherein the neural network model is trained using a BiLSTM layer for at least one layer of the first block. Claim 15 In paragraph 10, the above learning is a computer implementation method in which the learning is distant supervised learning. Claim 16 A computer-implemented method according to claim 10, further comprising: a step of finding an entry in a Q&A database that associates the predetermined query with the predetermined response; a step of searching for the predetermined response in a corpus of the documents using a search engine; and a sample of the response from the corpus of the documents to be input to the neural network model, wherein the sample is labeled as containing the response to the predetermined query whenever the predetermined response is found in a part of the text by the search engine, and the sample is labeled as not containing the response to the predetermined query whenever the predetermined response is not found in a part of the text by the search engine. Claim 17 In paragraph 16, a computer implementation method in which each part of the text is a paragraph or a sentence shorter than a paragraph. Claim 18 In paragraph 17, a computer implementation method in which the corpus of the above documents is an encyclopedia database. Claim 19 A computer-implemented method for multi-document query response, comprising: receiving a runtime query from a client device at a server; searching for documents regarding the runtime query using a search engine; identifying parts of text within the searched documents; calculating a runtime score for the identified parts of text using a neural network model trained using a distance map and distance-based ranking loss; selecting the parts of text corresponding to the highest score to provide a response; and transmitting the response from the server to the client device. Claim 20 A computer-implemented method according to claim 19, wherein the step of calculating the runtime score for the parts of the identified text further comprises the step of mapping to a fixed-size representation vector by combining an interaction matrix, Weaver blocks, and adaptive subsampling using the neural network model, and wherein the fixed-size representation vector is used to output the score.

Images