Augmenting machine learning language models with search engine results

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for augmenting a machine learning language model using search engine results. One of the methods includes obtaining question data representing a question, generating a search engine query for a search engine from the question data, obtaining a plurality of documents identified by the search engine in response to processing the search engine query, generating a plurality of conditioning inputs from the plurality of documents, each of the conditioning inputs representing at least a portion of one or more of the obtained documents, using a neural network for each of the plurality of generated conditioning inputs to process network inputs generated from (i) the question data and (ii) the conditioning inputs to generate network outputs representing candidate answers to the question, and generating answer data representing a final answer to the question from the network outputs representing the respective candidate answers.

Description

[Technical field]

[0001] This specification relates to processing inputs using neural networks. [Background technology]

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al., "Training compute-optimal large language models", arXiv preprint arXiv:2203.15556, 2022 [Non-Patent Document 2] JW Rae、S. Borgeaud、T. Cai、K. Millican、J. Hoffmann、HF Song、J. Aslanides、S. Henderson、R. In、S. Young、E. Rutherford、T. Hennigan、J. Menick、A. Cassirer、R. Powell、G. van den Driessche、LA Hendricks、M. Rauh、P. Huang、A. Glaese、J. Welbl、S. Dathathri、S. Huang、J. Uesato、J. Mellor、I. Higgins、A. Creswell、N. McAleese、A.Wu、E. Elsen、SM Jayakumar、E. Buchatskaya、D. Budden、E. Sutherland、K. Simonyan、M. Paganini、L. Cipher、L. Martens、XL Li、A. Kuncoro、A. Nematzadeh、E. Gribovskaya、D. Donato、A. Lazaridou、A. Mensch、J. Lespiau、M. Tsimpoukelli、N. Grigorev、D. Fritz、T. Sottiaux、M. Pajarskas、T. Pohlen、Z. Gong、D. Toyama、C. of Masson d' Autume、Y. Li、T. Terzi、V. Mikulik、I. Babuschkin、A. Clark、D. of the Houses、A. Guy、C. Jones、J. Bradbury、M. Johnson、BA Hechtman、L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG.Irving, “Scaling language models: Methods, analysis & insights from training gopher,” CoRR, abs / 2112.11446, 2021. [Non-Patent Document 3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer", arXiv preprint arXiv:1910.10683, 2019 [Non-Patent Document 4] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020 [Non-Patent Document 5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners", arXiv preprint arXiv:2005.14165, 2020 [Non-Patent Document 6] Kudo et al., arXiv:1808.06226 [Non-Patent Document 7] Devlin et al., arXiv:1808.04805 Summary of the Invention [Means for solving the problem]

[0004] This specification describes a system implemented as a computer program on one or more computers at one or more locations that executes a neural network configured to process network inputs representing input text and to generate network outputs representing predictions for the input text.

[0005] To generate the network output, the system is configured to generate a search engine query for the search engine from the input text, and to obtain a set of results from the search engine in response to the query. The system can then incorporate the results of the search engine query into the network input before processing the network input using the neural network. In this manner, the system can perform language processing tasks using the neural network, such as, for example, question answering, with significantly fewer computing resources than would otherwise be required. The network output can also incorporate information embedded in the search engine results, including up-to-date information that was not available during training of the neural network, thereby improving predictions about the input text.

[0006] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0007] Large-scale language models implemented as neural networks can produce impressive results in a variety of natural language processing tasks, including question answering. However, some of these models, particularly transformer-based model implementations, can have over a billion parameters and can require significant computing resources, power, and time to process the network inputs and generate the network outputs. Such models can have over 10 billion or even over 100 billion parameters. There can also be environmental considerations when such models are used at scale, such as in a search engine.

[0008] When the neural network is implemented on a digital assistant device, for example, a mobile device, implemented in a computing system that includes back-end components, particularly data servers, communicating with the digital assistant device via a data communication network such as the Internet, additional considerations arise. It is then necessary to optimize the computing load between the digital assistant device and the back-end components. This need can be particularly acute for large language models due to their significant memory and computing requirements compared to those typically found on mobile devices.

[0009] The techniques described herein address these problems. In some implementations, the techniques described allow for reduced computational load and improved load balancing, especially when large language models are implemented as neural networks in multitasking and parallel processing computer systems distributed across multiple sites and interconnected by a data communications network.

[0010] In some implementations, the described techniques enable beneficial distribution of computing load between local mobile computing devices and back-end servers in a network. More particularly, in implementations, conditioning a language model neural network on multiple conditioning inputs representing documents retrieved from an Internet search based on a question, as well as on question data from the question, enables the use of a smaller language model neural network, thereby enabling the implementation of the neural network on a mobile device with limited memory and computing resources.

[0011] Furthermore, using the techniques described herein, the system can leverage search engine results to generate predictions for input text using the most current information contained therein. Some existing systems use pre-trained neural networks without access to such search engine results to generate predictions, and therefore the predictions may be less reliable because the neural network can only encode information that was available to the neural network during training, i.e., the predictions rely on outdated information and therefore may be inaccurate or at least out of date. Thus, using the techniques described herein, the system can generate predictions that are more accurate and timely.

[0012] Furthermore, some existing systems must repeatedly retrain their neural networks to ensure that they encode the most current information. Because the system described herein can repeatedly access new search engine results, the system is not required to retrain the neural network, thus saving significant computational resources.

[0013] Using the techniques described herein, a system can generate predictions for an input text in response to processing a search engine query using information encoded in multiple different documents provided by the search engine. The multiple different documents may each include different respective information relevant to the prediction. Thus, predictions generated by the system may be more accurate than predictions generated using a single document.

[0014] Moreover, by generating multiple answers using multiple retrieved evidences, i.e., multiple different conditioning inputs, followed by a re-ranking phase using scores generated by the same language model neural network that generated the answers, the described system improves the quality of the generated answers without requiring larger, harder training of the neural network. That is, by extending the generation process as described above, the system can generate answers that exceed the quality of answers generated by larger neural networks that do not access search engines or that generate only a single answer in response to inputs that include a single conditioning input. Thus, these extension techniques can alleviate the problem of lower performance of smaller pre-trained neural networks, and may be particularly suitable for deployment on devices with constrained memory space, e.g., mobile devices, smart speakers, or other edge devices, that prevent efficient storage of models with a very large computational footprint, leading to a very large number of parameters.

[0015] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram of an example neural network system. [Diagram 2] FIG. 2 is a flow diagram of an exemplary process for generating a final answer to a received question. [Diagram 3] FIG. 4 is a flow diagram of an example process for determining relevant conditioning inputs. [Figure 4] FIG. 4 is a flow diagram of an example process for re-ranking candidate answers. [Diagram 5] FIG. 1 illustrates performance improvements achievable by an example neural network system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference numbers and designations in the various drawings indicate like elements.

[0018] 1 is a diagram of an exemplary neural network system 100. Neural network system 100 is an example of a system, implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0019] The system 100 receives an input text 102 and generates a network output 112 that represents a prediction about the input text 102 .

[0020] In particular, the system 100 employs a neural network 110 that is configured to process network inputs 104 that represent input text 102 and to generate network outputs 112 that represent predictions for the input text 102. In an implementation, the neural network 110 may be a pre-trained neural network.

[0021] To generate the network output 112, the system 100 is configured to generate a search engine query 114 for a search engine 120 from the input text 102, and to obtain a set of search results 122 from the search engine 120 in response to the query 114. Each search result 122 identifies a respective document.

[0022] Search engine 120 may be any suitable search engine accessible by system 100 and that searches any suitable corpus of documents, e.g., web pages, books, or other documents. For example, search engine 120 may be an Internet search engine that searches through the Internet and returns results 122 that reference documents available on the Internet. As another example, search engine 120 may be a different search engine that searches a private corpus of documents, e.g., documents available on an internal network or documents stored in a collection of one or more databases.

[0023] The system 100 can then use the neural network 110 to incorporate results 122 of the search engine query 114 into the network input 104 before processing the network input 104. Thus, the network output 112 incorporates information embedded in the search engine results 122, including up-to-date information that was not available during training of the neural network 110, and can therefore improve predictions about the input text 102.

[0024] Neural network 110 may have any suitable neural network architecture that enables the model to map an input sequence of text tokens from a vocabulary to an output sequence of text tokens from the vocabulary.

[0025] For example, the neural network 110 may have an encoder-decoder transformer-based architecture.

[0026] As another example, the neural network 110 may have a decoder-only transformer-based architecture, where an input sequence is provided to the neural network 110 as a "prompt."

[0027] In general, a transformer-based architecture may be characterized by having a succession of self-attention neural network layers that have an attention layer input for each element of the input and are configured to apply an attention mechanism across the attention layer inputs to generate an attention layer output for each element of the input. There are a number of different attention mechanisms that may be used.

[0028] In particular, the neural network 110 may be an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each particular text token in the output sequence conditional on a current input sequence, which includes (i) the input sequence followed by (ii) any text tokens that precede the particular text token in the output sequence.

[0029] More specifically, to generate the particular text token, the neural network 110 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of text tokens. The neural network 110 may then use the score distribution to select a text token from the vocabulary as the particular text token. For example, the neural network 110 may greedily select the highest scoring token or may sample the token from the distribution, e.g., using top-k sampling, nucleus sampling, or another sampling technique.

[0030] As a particular example, neural network 110 may be an autoregressive transformer-based neural network that includes multiple layers that each apply a self-attention operation. Neural network 110 may have any of a variety of transformer-based neural network architectures. Examples of such architectures include J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al., "Training compute-optimal large language models," arXiv preprint arXiv:2203.15556, 2022, and J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, XL Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d' Autume, Y.Li、T. Terzi、V. Mikulik、I. Babuschkin, A. Clark、D. of The Houses、A. Guy、C. Jones, J.S. Bradbury、M. Johnson, BA Hechtman, L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG. Irving, CoRR, abs / 2112.11446, 2021, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, and Yanqi Zhou, Wei Li, and Peter J Liu, arXiv, arXiv:1910.10683, 2019, Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Roman Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le chatbot」、CoRR、abs / 2001.09977, 2020, and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners", arXiv preprint arXiv:2005.14165, 2020.

[0031] The tokens in the vocabulary may be any suitable text tokens, e.g., words, word pieces, punctuation marks, characters, bytes, etc., that represent elements of text in one or more natural languages, and possibly numbers and other text symbols found in a corpus of text. For example, system 100 may tokenize a given sequence of words by applying a tokenizer, e.g., the SentencePiece tokenizer (Kudo et al., arXiv:1808.06226) or another tokenizer, to split the sequence into tokens from the vocabulary.

[0032] Before neural network 110 is used to generate network output 112, neural network 110 is pre-trained, for example, by system 100 or by one or more other systems.

[0033] In particular, system 100 or other systems pre-train neural network 110 on a language modeling task, e.g., a task that requires, given a current sequence of text tokens, predicting the next token that will follow the current sequence in the training data. That is, the language modeling task may require, for each given unlabeled text sequence in the training dataset, predicting the text sequence that followed the given unlabeled text sequence in a corresponding document. As a particular example, language model neural network 110 may be pre-trained on a maximum-likelihood objective on a large dataset of text, e.g., text that is publicly available from the Internet or another text corpus.

[0034] After training, the system 100 may be configured to use the search engine results 122 and the neural network 110 to perform any suitable machine learning task.

[0035] For example, the system 100 may be configured to perform question answering, where the input text 102 identifies a question and the network output 112 represents an answer to the question.

[0036] For example, a question may be provided by a user of the system 100, e.g., by providing the input text 102 directly to the system, or by providing the system 100 with audio data representing a verbalization of the input text 102. In these implementations, the search engine results 122 may include a set of documents that are relevant to the question, and thus the neural network 110 may leverage the search engine results to answer the question using information that was not available at the time the neural network 110 was trained. In these implementations, the data representing the input text 102 may be referred to as "question data."

[0037] As another example, the system 100 may be configured to perform fact checking, where the input text 102 represents a statement and the network output 112 represents a prediction about whether the statement is factually accurate. Similarly, in these implementations, the search engine results 122 may include a set of documents related to the statement.

[0038] In some implementations, system 100 performs one of these downstream tasks, such as question answering, using neural network 110 without further training neural network 110. For example, system 100 can perform a question answering task using neural network 110 trained only on language modeling tasks.

[0039] Although the following description refers to an implementation in which the system 100 is configured to perform question answering, it should be understood that in general, the neural network 110 may be configured to perform any suitable task using the input text.

[0040] Once the system 100 generates the network output 112, the system 100 may provide the network output 112 to a user.

[0041] For example, system 100 may be implemented as part of or in communication with a digital assistant device, e.g., a mobile device, a smart watch or other wearable device, or a smart speaker device, which can provide network output 112 to a user, e.g., by generating audio representing network output 112 and playing the audio to the user via a speaker.

[0042] As another example, the system 100 can provide the network output 112 for presentation in a user interface of a user device, eg, a user device through which a user submitted the input text 102 .

[0043] 2 is a flow diagram of an exemplary process 200 for generating a network output. For convenience, the process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, can perform the process 200.

[0044] The system obtains question data representing a question (step 202).

[0045] From the question data, the system generates a search engine query for a search engine (step 204).

[0046] The system can generate the search engine query from the input text, i.e., from the question data, in any suitable manner. For example, the search engine query can be (or can include) the input text itself, e.g., the question that the neural network is to answer, in implementations in which the neural network is configured to perform question answering. That is, the search engine query can be equivalent to the input text as is. As another example, the system can process the input text to generate updated text to act as the search engine query, e.g., by processing the input text using one or more pre-determined templates. As another example, the system can process the input text using a machine learning model, e.g., by processing a sequence of tokens representing the input text using another neural network, to generate a network output representing the text of the search engine query.

[0047] The system retrieves a plurality of documents identified by the search engine in response to processing the search engine query, step 206. For example, the system may receive a set of search results from the search engine, each of which identifies a respective document from a corpus of documents to be searched by the search engine.

[0048] The system can submit a search engine query to a search engine and receive a set of returned documents D. For example, the system can submit the search engine query using an application programming interface (API) provided by the search engine. The system can be configured to retrieve a predetermined number p of documents D, i.e., the p documents indexed by the search engine that are highest ranked by the search engine in response to the submitted search engine query. In some implementations, the system receives each document in HTML format and processes the HTML data to extract clean text for the document.

[0049] The system generates a plurality of conditioning inputs from the plurality of documents, each of which represents at least a portion of one or more of the retrieved documents (step 208).

[0050] Each conditioning input represents some or all of the text of the corresponding document, e.g., the conditioning inputs may each include a different subsequence of the sequence of tokens representing the corresponding text. Because at least some of the documents D may be represented by sequences that are longer than the maximum sequence length that can be processed by the neural network, the system may generate multiple different conditioning inputs for a single document, where each conditioning input represents a different subset, e.g., disjoint subsets, of the text of the document.

[0051] In some implementations, the system can generate conditioning inputs from multiple different documents, for example, by concatenating respective subsets of the text of each document.

[0052] As a particular example, for each retrieved document, the system may segment the document into fixed-sized segments. Such segments may be referred to as "paragraphs," although such segments do not necessarily align with the original paragraphs of the document. For example, each segment may contain a predetermined number of sentences from the original document (e.g., a predetermined number of text sequences separated by periods or other specified punctuation marks). The system may then generate a respective conditioning input from each segment, for example, by generating a sequence of tokens that represent the text of the segment.

[0053] For each of the multiple generated conditioning inputs, the system uses a neural network to process (i) the question data and (ii) network inputs generated from the conditioning inputs to generate network outputs representing candidate answers to the question (step 210).

[0054] In some implementations, rather than generating a network input for each conditioning input, the system can determine which of the generated conditioning inputs are most relevant to the question expressed by the input text, and generate respective network inputs for only the most relevant conditioning inputs.

[0055] Determining the relevant conditioning inputs is described below with reference to FIG.

[0056] The network input for a given conditioning input may include or be generated from both the input text and the conditioning input. For example, the system may concatenate the input text and the conditioning input to generate a single sequence, e.g., with one or more delimiter tokens separating subsequences representing the input text and the subsequences representing the conditioning input.

[0057] In some implementations, the system further uses k prompt tuples to generate network inputs corresponding to each respective conditioning input, where k≧1. Each prompt tuple may correspond to a respective second question that is different from the question to be answered by the neural network. Each prompt tuple may include (i) second question data (e.g., a sequence of tokens representing the second question) representing the respective second question, (ii) second answer data (e.g., a sequence of tokens representing the answer) representing an answer to the second question, and (iii) a second conditioning input (e.g., a sequence of tokens representing the document) representing a document related to the second question. For example, the system may obtain the prompt tuples from a predetermined training dataset. Including prompt tuples in the network inputs may be referred to as “k-shot prompting.”

[0058] In some implementations, the neural network is a pre-trained neural network that has been trained on a different machine learning task than the task for which the system is using the neural network, e.g., a different task than question answering. Thus, in these implementations, the neural network has not been trained to generate a network output having a suitable format for the machine learning task for which the system is using the neural network, e.g., has not been trained to generate a network output having a format required for question answering. Thus, prompt tuples may be included in the network inputs so that the neural network has access to examples of suitable formats for the neural network's output.

[0059] For example, for each conditioning input, the system can generate a network input including k+1 subsequences, where the k subsequences correspond to respective second questions and the (k+1)th subsequence corresponds to a question to be answered by the neural network. The k subsequences corresponding to the second questions may each include corresponding second question data, second answer data, and second conditioning inputs, while the subsequences corresponding to the questions to be answered by the neural network may only include input text representing the question and conditioning inputs (i.e., do not have a corresponding answer to be generated by the neural network). Thus, the k subsequences corresponding to the second questions may represent an appropriate format of the network output of the neural network.

[0060] As a specific example, each subsequence corresponding to each second question may have the following format: [one or more "evidence" tokens] + [second conditioning input] + [one or more "question" tokens] + [second question data] + [one or more "answer" tokens] + [second answer data] where the "evidence", "question", and "answer" tokens are any suitable tokens that are predetermined. The subsequence corresponding to the question to be answered by the neural network may then have the following format: [one or more "evidence" tokens] + [conditioning input] + [one or more "question" tokens] + [text input] + [one or more "answer" tokens]

[0061] In some implementations, the system generates multiple candidate answers for each of the network inputs. That is, the system can process each network input using a neural network a fixed (greater than one) number of times to generate multiple different candidate answers. As explained above, because the system samples from a probability distribution to generate each token of a given output sequence, processing the same network input multiple different times will generally result in different candidate answers being generated.

[0062] The system generates answer data representing a final answer to the question from the network outputs representing each candidate answer (step 212).

[0063] In particular, for each candidate answer, the system can generate a respective likelihood value for the candidate answer that represents the likelihood that the candidate answer is a correct answer to the question. The system can then select the candidate answer with the highest corresponding likelihood value as the final answer. In other words, the system "re-ranks" the candidate answers and then selects the highest ranked candidate answer as the final answer.

[0064] Re-ranking candidate answers is described in more detail below with reference to FIG.

[0065] The system can then provide the answer data as output, for example by providing the answer data as audio that is played to the user or as text that is to be presented to the user in a user interface.

[0066] Alternatively, after determining the final answer to the question, the system can provide the final answer to one or more external systems for storage or further processing. For example, the system can provide the final answer to a user system implemented on the user's device for providing the final answer to the user, for example, by displaying the final answer on a display of the device, or by generating audio data representing a verbalization of the final answer and playing the audio data for the user. In some implementations, the neural network is implemented on the user's device, and in some other implementations, the neural network is implemented on a different device, for example, the neural network may be deployed on the cloud.

[0067] 3 is a flow diagram of an exemplary process 300 for determining relevant conditioning inputs. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, can perform the process 300.

[0068] The system obtains a set of candidate conditioning inputs, step 302. For example, the system may have generated each conditioning input using one or more documents identified in search results obtained from a search engine such as those described above.

[0069] The system generates an embedding for the query data (step 304) and generates an embedding for each of the conditioning inputs (step 306).

[0070] As used herein, an embedding is an ordered set of numbers that represent an input within a particular embedding space. For example, an embedding can be a vector of floating-point or other numbers with a fixed number of dimensions.

[0071] The system can generate embeddings of sequences of text, for example of conditioning inputs or of question inputs, in an appropriate manner.

[0072] For example, the system can use "term frequency - inverse document frequency" (tf-idf) embeddings to generate the embeddings, e.g., by combining the tf-idf embeddings of each of the tokens in the text sequence.

[0073] As another example, the system can map each token to a word embedding, e.g., a word2vec or Glove embedding, and then combine, e.g., average or sum, the word embeddings.

[0074] As another example, the system may use an embedding neural network, e.g., a self-attention based neural network, to process network inputs representing the input text to generate an embedding for the input text. For example, the embedding neural network may have been trained through unsupervised learning to minimize, e.g., the BERT loss (Devlin et al., arXiv:1808.04805).

[0075] For each conditioning input, the system determines the similarity between the embedding of the conditioning input and the embedding of the query data, step 308. For example, the similarity may be measured according to any suitable similarity measure between the two vectors, e.g., cosine similarity, Euclidean distance, etc.

[0076] The system selects the conditioning inputs with the highest similarity as the most relevant conditioning inputs, step 310. For example, the system may select each conditioning input that has at least a threshold similarity, or may select a threshold number of the most similar conditioning inputs.

[0077] 4 is a flow diagram of an exemplary process 400 for reranking candidate answers. For convenience, the process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as the neural network system 100 shown in FIG. 1, can perform the process 400.

[0078] The system receives a set of candidate answers (step 402).

[0079] For each candidate answer, the system generates a respective likelihood value for the candidate answer that represents the likelihood that the candidate answer is a correct answer to the question (step 404).

[0080] For example, given the N conditioning inputs p from which the network inputs are generated, i Each candidate answer a to question q is generated using one of i For p(a i |q) or p(a i ,q|p i ) and using the predictions to generate a likelihood score. As a specific example, the likelihood score can be i |q) or p(ai ,q|p i ), or a combination of both probabilities using a Product-of-Experts (PoE). For example, PoE can calculate a product or weighted sum of probabilities. In some implementations, PoE also calculates p(p i |q) is taken into account, e.g., by including

[0081] For example, the system may generate the following predictions:

[0082]

number

[0083] Alternatively, or in addition, the system may generate a prediction of:

[0084]

number

[0085] However, p i is a i are the conditioning inputs used to generate

[0086] In some implementations, the system calculates the probability p(p j |q) may be the similarity (or a normalized version of the similarity) calculated as described above with reference to Figure 3. In some other implementations, the system calculates this probability using a neural network, as described below.

[0087] For probability p(x|y,z), element x is sometimes called the "posterior element" and elements y and z are sometimes called the "prior elements."

[0088] The system can predict the respective probabilities using that or another neural network, e.g., these likelihoods may be obtained from a smaller, pre-trained language model. For example, if the neural network is an autoregressive neural network, to generate a prediction for a particular probability, the system can generate a network input that includes one or more prior elements of the particular probability and use the trained parameter values ​​of the neural network to determine the likelihood that the neural network will generate a network output that represents the posterior element of the particular probability.

[0089] As a specific example, p(a i |q,p i To generate a prediction for q and p i Generate network inputs containing i For example, this likelihood can be determined according to the model: i can be determined from the probability of the last token of

[0090] In some such implementations, the system uses k-shot prompting to generate a network input that includes (i) one or more prior elements of particular probability and (ii) one or more prompt tuples. The system can then determine the probability that the neural network will generate a network output that identifies a posterior element of particular probability in response to processing the network input by inferring the likelihood that such a network output will be generated autoregressively. For example, p(p i When predicting |q, the system predicts (q',p i '), and given q, p i Similarly, we can determine the likelihood of generating p(a i |q,p i ), the system predicts (q',p i ',ai '), and q and p i Given a, i Similarly, we can determine the likelihood of generating p(q|a i ,p i ), the system predicts (q',p i ',a i '), and a prompt tuple of the form i and p i Given p(a, we can determine the likelihood of generating q. Similarly, i |p i ), the system predicts (p i ',a i '), and p i Given a, i Similarly, we can determine the likelihood of generating p(q|p i ), the system predicts (p i ',q) and p i Given q, we can determine the likelihood of generating q.

[0091] The system may then select the candidate answer with the highest corresponding likelihood value as the final answer (step 406).

[0092] As previously described, in some implementations, the system is implemented on a digital assistant device such as a mobile device. In such implementations, the neural network may be implemented (fully) as a neural network on the mobile device. Search engine queries may be processed using a back-end data server, which may be implemented using a parallel and distributed computing system. The mobile device generally has less working memory than the back-end data server, less computing power than the back-end data server, or both. The computing power may be measured in computing operations per second, e.g., FLOPS (floating point operations per second). In this way, the computational load may be optimized, especially when the neural network comprises an autoregressive, transformer-based neural network, and when the search engine comprises an Internet search engine. This, in turn, may enable the use of smaller language models, e.g., language models with fewer than 10 billion parameters, e.g., 7 billion parameters, 1 billion parameters, or fewer parameters.

[0093] FIG. 5 shows exact match question answering performance (on the y-axis) versus FLOPS (on the x-axis) comparing the performance of an exemplary neural network system of the type shown in FIG. 1 (sloped line) with the performance of a pre-trained language model that does not use a search engine (dashed horizontal line and stars). The exemplary neural network system is either a 1 or 7 billion parameter version of the pre-trained Gopher language model (Rae et al., supra), and uses a variety of different techniques for determining the likelihood that a candidate answer is the correct answer. The stars indicate the performance of the pre-trained Gopher language model with 280 billion parameters. In some cases, implementations of the techniques described are worth over 273 billion parameters.

[0094] In some implementations, the system may be used as part of a real-world control system where the questions and answers each relate to the operation of an item of machinery or tools in the real world. The method may then include obtaining a natural language question regarding the operation of the item of machinery, processing the question using a neural network system as described above to generate a final answer to the question, and then using the answer to control the item of machinery or tools. For example, the item of machinery or tools may be a robot and the question may be "What is the maximum safe load for a Model 123.45 robot?" or the item of machinery or tools may be an oven and the question may be "At what temperature should I bake the cake?".

[0095] This specification uses the term "configured" in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. A computer program or programs configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0096] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or in a combination of one or more of them, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0097] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In some cases, an apparatus may include, in addition to hardware, code that creates an execution environment for a computer program, e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0098] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program may be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., a file that stores one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0099] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0100] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or computers.

[0101] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0102] A computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. In general, the central processing unit will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer will also include one or more mass storage devices, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data therefrom, or transfer data thereto, or both. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, to name just a few examples.

[0103] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0104] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser in response to a request received from a web browser on the user's device. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0105] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of, e.g., machine learning training or production, e.g., inference, workloads.

[0106] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0107] An embodiment of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user may interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0108] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., results of user interaction, may be received at the server from the device.

[0109] Although the present specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the present specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in some combinations, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of a subcombination.

[0110] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence illustrated, or that all of the illustrated operations be performed to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0111] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes illustrated in the accompanying figures do not necessarily require the particular order illustrated, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0112] 100 Neural network system, system 102 Input text, text input 104 Network Input 110 Neural Networks, Language Models Neural Networks 112 Network Output 114 Search Engine Queries, Queries 120 Search Engines 122 search results, results, search engine results

Claims

1. obtaining question data representing a question; generating a search engine query for a search engine from the query data; retrieving a plurality of documents identified by the search engine in response to processing the search engine query; generating a plurality of conditioning inputs from the plurality of documents, each conditioning input representing at least a portion of one or more of the retrieved documents; for each of a plurality of said generated conditioning inputs, processing (i) said question data and (ii) network inputs generated from said conditioning inputs using a neural network to generate network outputs representing candidate answers to said question; generating answer data representing a final answer to the question from the network outputs representing each candidate answer; The method includes:

2. The step of generating a plurality of conditioning inputs from the plurality of documents includes, for each of the plurality of documents: extracting a text sequence from each retrieved document; determining a respective conditioning input from each of one or more subsequences of the extracted text sequence; 2. The method of claim 1, comprising:

3. generating, for each conditioning input representing at least a portion of one or more respective documents, a relevance score representing a predicted relevance of said portion of said one or more respective documents to said query; determining a ranking of the conditioning inputs according to the corresponding relevance scores; Further comprising:

3. The method of claim 1 or 2, wherein the subset of conditioning inputs comprises the n highest ranked conditioning inputs in the ranking, where n>1.

4. generating the relevance scores for the conditioning inputs, generating an embedding of said query data; generating a respective embedding for each conditioning input; for each conditioning input, determining a similarity between the embedding of the conditioning input and the embedding of the query data; 4. The method of claim 3, comprising:

5. 5. The method of claim 1, wherein for each network input generated from a respective conditioning input, the network input is further generated from one or more prompt tuples each including: (i) second question data representing a different second question; (ii) second answer data representing an answer to the respective second question; and (iii) second conditioning inputs representing documents related to the respective second questions.

6. 6. The method of claim 5, wherein the neural network is pre-trained on a language modeling task and has not been trained on a question answering task.

7. generating answer data representing a final answer to the question from the network outputs representing each candidate answer, generating, for each candidate answer, a respective likelihood value representing the likelihood that said candidate answer is a correct answer to said question; determining the final answer to be the candidate answer corresponding to the highest likelihood value; 7. The method of claim 1 , comprising:

8. for each candidate answer a' to the question q corresponding to a respective conditioning input p, generating a respective likelihood value for the candidate answer a' generating one or more initial likelihood values, said one or more initial likelihood values ​​being: a first initial likelihood value representing an estimate for the probability p(a'|p,q) of the candidate answer given the conditioning input and the question; a second initial likelihood value representing an estimate for the probability p(a'|q) of the candidate answer given the question; a third initial likelihood value representing an estimate for the probability p(p|q) of the conditioning input given the question; a fourth initial likelihood value representing an estimate for the probability p(q|p) of the question given the conditioning inputs; or a fifth initial likelihood value representing an estimate for the probability p(q|a',p) of the question given the candidate answers and the conditioning inputs; and generating said likelihood value from said one or more initial likelihood values; 8. The method of claim 7, comprising:

9. 9. The method of claim 8, wherein for at least one of the initial likelihood values ​​representing an estimate for a probability of a respective posterior element given one or more respective prior elements, the initial likelihood value is generated using the neural network.

10. using the neural network to generate specific initial likelihood values ​​representing estimates for the probability of a specific posterior element given one or more specific prior elements; determining a likelihood, using the set of trained parameter values ​​of the neural network, that the neural network will generate a second network output representative of the particular posterior element in response to processing a second network input generated from the one or more particular prior elements.

10. The method of claim 9, comprising:

11. 11. The method of claim 10, wherein the second network input is further generated from one or more prompt tuples having the same particular posterior element and one or more particular prior elements as the particular initial likelihood value.

12. the neural network comprises an autoregressive, transformer-based neural network; The method of claim 1 , wherein the search engine comprises an Internet search engine.

13. the neural network is implemented on a mobile device; Processing the search engine query using a backend data server. Further comprising:

13. The method of claim 1, wherein the mobile device has one or both of: less working memory than the backend data server; and less computing power than the backend data server.

14. 14. A system comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of a respective method according to any one of claims 1 to 13.

15. One or more computer storage media having instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations of a respective method as recited in any one of claims 1 to 13.

Images