Extension of machine learning language models using search engine results

Abstract

System comprising one or more computers and one or more storage devices that store instructions which, when executed by the one or more computers, cause the one or more computers to perform operations that include: Obtaining query data that constitutes a question; Generating a search engine query for a search engine from the query data; Obtaining a large number of documents identified by the search engine in response to processing the search engine query; Generating a multitude of conditioning inputs, each representing at least one section of one or more of the obtained documents, from the multitude of documents; for each conditioning input generated from a multitude of inputs, processing a network input generated from (i) the question data and (ii) the conditioning input using a neural network to produce a network output that represents a candidate response to the question; and Generating response data that represents a final answer to the question from the network outputs representing the respective candidate responses.

Description

GENERAL STATE OF THE ART

[0001] This description concerns the processing of inputs using neural networks.

[0002] Neural networks are machine learning models that use one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input for the next layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from a given input according to the current values ​​of its respective set of parameters. SUMMARY

[0003] This specification describes a system implemented as computer programs on one or more computers at one or more locations, executing a neural network configured to process a network input representing an input text and to produce a network output representing a prediction about the input text.

[0004] To generate the network output, the system is configured to create a search engine query from the input text and retrieve a set of results from the search engine in response. The system can then include the search engine query results in the network input before processing the network input using the neural network. In this way, the system can perform tasks such as natural language processing, like answering questions, using the neural network with significantly fewer computational resources than would otherwise be required. Because the network output can also include information embedded in the search engine results, such as up-to-date information that was unavailable during the neural network's training, the prediction of the input text can be improved.

[0005] The item described in this specification can be implemented in certain embodiments such that one or more of the following advantages are achieved.

[0006] Large-scale language models implemented as neural networks can achieve impressive results in a range of natural language processing tasks, including answering questions. However, implementations of some of these models, particularly transformer-based models, can have more than a billion parameters and require significant computing resources, energy, and time to process network input and generate network output. Sometimes, such models can have more than 10 billion or even more than 100 billion parameters. If such models were used on a large scale, for example, in the way search engines are used, environmental factors could also come into play.

[0007] An additional consideration arises when the neural network is implemented on a digital assistant device, such as a mobile device, which is itself implemented in a computing system that includes a backend component, particularly a data server, communicating with the digital assistant device via a data communication network such as the internet. Therefore, there is a need to optimize the computational load between the digital assistant device and the backend component. This need can be particularly acute for a large-scale speech model, as it has significantly higher memory and computing requirements compared to typical models on a mobile device.

[0008] The techniques described here solve these problems. In some implementations, the described techniques enable a reduced computational load and improved load balancing, especially when the large-scale language model is implemented as a neural network in a multitasking and parallel processing computer system that is distributed across multiple locations and interconnected by a data communication network.

[0009] In some implementations, the described techniques enable an advantageous distribution of the computational load between a local, mobile computing device and a backend server in a network. Specifically, in some implementations, conditioning the neural network to a variety of conditioning inputs—documents obtained from an internet search based on a question, as well as question data—allows the use of a smaller neural network for the language model. This facilitates the implementation of the neural network on a mobile device with limited memory and computing resources.

[0010] Furthermore, a system can use the techniques described in this specification to leverage search engine results to generate a prediction about an input text, utilizing current information contained within those results. Some existing systems use pre-trained neural networks without access to such search engine results to generate predictions, and thus the predictions can be less reliable because the neural network can only encode information available to it during training; that is, these predictions may be based on outdated information and therefore be erroneous or at least outdated. Using the techniques described in this specification, a system can generate predictions that are more accurate and timely.

[0011] Furthermore, some existing systems need to retrain neural networks repeatedly to ensure that the neural networks encode the latest information. Since the systems described in this specification can repeatedly access new search engine results, the system does not need to retrain the neural network, thus saving significant computing resources.

[0012] Using the techniques described in this specification, a system can generate predictions for an input text using information encoded in several different documents provided by a search engine in response to a search query. These many different documents can each contain different information relevant to the prediction. Thus, the predictions generated by the system can be more accurate than predictions generated using a single document.

[0013] Furthermore, the described system improves the quality of the generated responses by generating multiple responses using several retrieved pieces of evidence—that is, multiple different conditioning inputs—followed by a re-evaluation phase that uses evaluations generated by the same neural language model network that produced the responses, without requiring a larger, more difficult-to-train neural network. That is, by extending the generation process as described above, the system can produce responses of a higher quality than those generated by a larger neural network that does not access a search engine or that produces only a single response in response to an input containing a single conditioning input.Therefore, these extension techniques can mitigate minor performance problems of smaller pre-trained neural networks and may be particularly suitable for use on devices with limited storage space, such as mobile devices, smart speakers, or other edge devices that prevent them from efficiently storing models with extremely high computational demands, such as an extremely large number of parameters.

[0014] The details of one or more embodiments of the subject matter are shown in the accompanying figures and the following description. Further features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram of an example of a neural network system. Fig. Figure 2 is a flowchart of an example process for generating a final answer to a received question. Fig. Figure 3 is a flowchart of an example process for determining relevant conditioning inputs. Fig. Figure 4 is a flowchart of an example process for re-evaluating candidate responses. Fig. Figure 5 illustrates the performance improvements that can be achieved through examples of the neural network system.

[0015] Identical reference symbols in different drawings indicate identical elements. DETAILED DESCRIPTION

[0016] Fig. Figure 1 is a diagram of an example of a neural network system 100. The neural network system 100 is an example of a system that is implemented as computer programs on one or more computers at one or more locations, where the systems, components, and techniques described below may be implemented.

[0017] System 100 receives an input text 102 and generates a network output 112, which represents a prediction about the input text 102.

[0018] In particular, the system 100 uses a neural network 110 configured to process a network input 104 representing an input text 102 and to generate a network output 112 representing a prediction about the input text 102. In implementations, the neural network 110 can be a pre-trained neural network.

[0019] To generate network output 112, system 100 is configured to generate a search engine query 114 for a search engine 120 from the input text 102 and to retrieve a set of search results 122 from the search engine 120 as a response to the query 114. Each search result 122 identifies a corresponding document.

[0020] Search engine 120 can be any suitable search engine that system 100 can access and that searches any suitable corpus of documents, such as websites, books, or other documents. For example, search engine 120 could be an internet search engine that searches documents available on the internet and delivers results 122. Alternatively, search engine 120 could be a different search engine that searches a private corpus of documents, such as documents available on an internal network or stored in a collection of one or more databases.

[0021] System 100 can then include the results 122 of the search engine query 114 in the network input 104 before processing the network input 102 using the neural network 110. This allows the network output 112 to include information embedded in the search engine results 122, including current information that was not available during the training of the neural network 110, thus improving the prediction of the input text 102.

[0022] The neural network 110 can have any suitable neural network architecture that allows the model to map an input sequence of text tokens from a vocabulary to an output sequence of text tokens from the vocabulary.

[0023] The neural network 110, for example, can have an encoder-decoder-transformer-based architecture.

[0024] As another example, neural network 110 can have a decoder-transformer-based architecture where the input sequence is provided as an "input prompt" for neural network 110.

[0025] In general, a transformer-based architecture can be characterized by having a sequence of self-aware neural network layers. A self-aware neural network layer has a network input for each element of the input and is configured to apply an attention mechanism to the attention layer's input to generate an attention layer network output for each element of the input. Many different attention mechanisms can be used.

[0026] In particular, neural network 110 can be an autoregressive neural network that autoregressively generates the output sequence of text tokens by conditioned on a current input sequence that includes (i) the input sequence followed by (ii) all text tokens preceding the specific text token in the output sequence.

[0027] More precisely, to generate a specific text token, neural network 110 can process the current input sequence to create a scoring distribution, such as a probability distribution, that assigns a corresponding score, such as a corresponding probability, to each token in the text token vocabulary. Based on this scoring distribution, neural network 110 can then select a text token from the vocabulary as the specific text token. For example, neural network 110 might greedily select the token with the highest score, or it might sample a token from the distribution using top-k sampling, core sampling, or another sampling technique.

[0028] As a specific example, neural network 110 can be an autoregressive transformer-based neural network containing a multitude of layers, each applying a self-attention process. Neural network 110 can have any transformer-based neural network architecture. Examples of such architectures are described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; JW Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D.Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu und G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-totext transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R.So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu and Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv-Vorabdruck arXiv:2005.14165, 2020.

[0029] The vocabulary tokens can be any suitable text tokens, such as words, word parts, punctuation marks, characters, bytes, etc., representing text elements in one or more natural languages, as well as optionally numbers and other text symbols found in a text corpus. System 100 can, for example, tokenize a given sequence of words by applying a tokenizer, such as the sentence part tokenizer (Kudo et al., arXiv:1808.06226) or another tokenizer, to divide the sequence into tokens from the vocabulary.

[0030] Before using neural network 110 to generate network outputs 112, neural network 110 is pre-trained, e.g. by system 100 or by one or more other systems.

[0031] In particular, System 100 or the other system(s) trains the neural network 110 on a language modeling task, e.g., a task where, based on a current sequence of text tokens, the goal is to predict the next token that follows the current sequence in the training data. Equivalently, the language modeling task may require that, for each given unlabeled sequence in a training dataset, a text sequence is predicted that follows the given unlabeled sequence in a corresponding document. As a specific example, the neural network 110 can be trained with a maximum-probability goal on a large text dataset, e.g., text that is publicly available on the internet or another text corpus.

[0032] After training, the system 100 can be configured to use the search engine results 122 and the neural network 110 to perform any machine learning task.

[0033] For example, the system 100 can be configured to perform a question answering function, where the input text 102 identifies a question and the network output 112 represents an answer to the question.

[0034] The question can be provided, for example, by a user of System 100, either by directly providing the input text 102 to the system or by providing audio data that represents a verbalization of the input text 102 to System 100. In these implementations, the search engine results 122 can include a set of documents relevant to the question; in this way, the neural network 110 can use the search engine results to answer questions using information that was not available at the time the neural network 110 was trained. In these implementations, the data representing the input text 102 is sometimes referred to as "question data".

[0035] As another example, the system 100 can be configured to perform a fact check where the input text 102 represents a statement and the network output 112 makes a prediction as to whether the statement is factually true. Similarly, the search engine results 122 can include a set of documents relevant to the statement.

[0036] In some implementations, System 100 uses Neural Network 110 to perform one of these downstream tasks, such as answering questions, without further training Neural Network 110. For example, System 100 can use Neural Network 110 that has only been trained on a language modeling task to perform the question-answering task.

[0037] Although the following description refers to implementations in which System 100 is configured to perform answering questions, Neural Network 100 can generally be configured to perform any task using the input text.

[0038] Once System 100 has generated network output 112, System 100 can provide network output 112 to the user.

[0039] For example, the System 100 can be implemented as part of or communicate with a digital assistant device, such as a mobile device, smartwatch or other wearable device, or smart speaker device, and the digital assistant device can provide the network output 112 to the user, for example by generating speech that represents the network output 112 and playing the speech back to the user through a speaker.

[0040] As another example, the system 100 can provide the network output 112 for presentation in a user interface of a user device, e.g. the user device through which the user made the text input 102.

[0041] Fig. Figure 2 is a flowchart of an example process 200 for generating a network output. For simplicity, process 200 is described as being performed by a system of one or more computers located at one or more sites. For example, a neural network system, such as the one described in Fig. 1. The neural network system 100 shown, which is programmed according to this description, performs the process 200.

[0042] The system obtains query data that constitutes a question (step 202).

[0043] The system generates a search engine query from the question data for a search engine (step 204).

[0044] The system can generate the search engine query from the input text, i.e., from the question data, in any suitable way. For example, the search engine query can be (or include) the input text itself, such as the question the neural network is to answer in implementations where the neural network is configured to perform question-answering. That is, the search engine query can be identical to the input text in its current form. As another example, the system can process the input text to generate an updated text that functions as the search engine query, for example, by processing the input text using one or more predefined templates. As yet another example, the system can process the input text using a machine learning model, for example...by processing a sequence of text tokens representing the input text, using another neural network to generate a network output representing the text of the search engine query.

[0045] The system obtains a multitude of documents identified by the search engine in response to processing the search engine query (step 206). For example, the system may receive from the search engine a set of results, each identifying a corresponding document from the corpus of documents being searched by the search engine.

[0046] The system can submit the search engine query to the search engine and receives a set of multiple documents D in return. The system can submit the search engine query, for example, using an application programming interface (API) provided by the search engine. The system can be configured to retrieve a predetermined number p of documents D, i.e., the p documents indexed by the search engine that were ranked highest by the search engine in response to the query. In some implementations, the system receives each document in HTML format and processes the HTML data to extract the document's plain text.

[0047] The system generates a multitude of conditioning inputs, each representing at least one section of one or more of the obtained documents, from the multitude of documents (step 208).

[0048] Each conditioning input represents part or all of the text of the corresponding document; for example, the conditioning inputs can each contain a different subsequence of the sequence of tokens representing the corresponding text. Since at least some of the documents D can be represented by sequences longer than the maximum sequence length that can be processed by the neural network, the system can generate multiple different conditioning inputs for a single document, with each conditioning input representing a different subset of the document's text, e.g., disjoint subsets.

[0049] In some implementations, the system can generate a conditioning input from several different documents, e.g., by concatenating the respective subsets of the text from each document.

[0050] For example, the system can divide each retrieved document into fixed-size segments. Such segments are sometimes referred to as "paragraphs," although they do not necessarily correspond to the original paragraphs of the document. Each segment can, for instance, contain a predetermined number of sentences from the original document (e.g., a predetermined number of text sequences separated by periods or other punctuation marks). The system can then generate a corresponding conditioning input from each segment, for example, by creating a sequence of tokens representing the text of that segment.

[0051] For each conditioning input generated from a multitude of possibilities, the system processes a network input generated from (i) the question data and (ii) the conditioning input using a neural network to produce a network output that represents a candidate response to the question (step 210).

[0052] In some implementations, instead of generating a network input for each conditioning input, the system can determine which of the generated conditioning inputs are most relevant to the question represented by the input text, and generate a corresponding network input only for the most relevant conditioning inputs.

[0053] The determination of the relevant conditioning inputs is described below with reference to Fig. 3 described.

[0054] The network input for a given conditioning input can include or be generated from both the input text and the conditioning input. For example, the system can concatenate the input text and the conditioning input to create a single sequence, perhaps with one or more delimiting tokens that separate the subsequence representing the input text from the subsequence representing the conditioning input.

[0055] In some implementations, the system further uses k prompts to generate the network input corresponding to the respective conditioning input, k ≥ 1. Each prompt can correspond to a second question that differs from the question the neural network is to answer. Each prompt tuple can include (i) second question data representing the respective second question (e.g., a sequence of tokens representing the second question), (ii) second answer data representing an answer to the second question (e.g., a sequence of tokens representing the answer), and (iii) a second conditioning input representing a document relevant to the second question (e.g., a sequence of tokens representing the document). For example, the system can obtain the prompts from a predefined training dataset.Including prompts in network input is sometimes referred to as "k-shot prompting".

[0056] In some cases, the neural network is a pre-trained neural network that was trained for a different machine learning task than the one the system is using it for, for example, a task other than question answering. Therefore, in these implementations, the neural network was not trained to produce network output that has the appropriate format for the machine learning task the system is using it for; for example, it was not trained to produce network output that has the required format for answering questions. Thus, the prompts can be included in the network input, giving the neural network access to examples of the correct format for its output.

[0057] For example, the system can generate a network input for each conditioning input that contains k + 1 subsequences, where k subsequences correspond to a respective second question and the (k + 1) te The subsequence corresponds to the question that the neural network is to answer. The k subsequences corresponding to the second questions can each contain the corresponding second question data, the second answer data, and the second conditioning input, while the subsequence corresponding to the question to be answered by the neural network can only contain the input text representing the question and the conditioning input (i.e., without the corresponding answer that the neural network is to generate). Thus, the k subsequences corresponding to the second question can represent the correct format of the neural network's output.

[0058] As a specific example, each subsequence corresponding to a given second question can have the following format: [One or more "proof" tokens] + [second conditioning input] + [one or more "question" tokens] + [second question data] + [one or more "answer" tokens] + [second answer data] where the "proof" tokens, "question" tokens, and "answer" tokens are all suitable tokens that have been previously defined. Then the subsequence corresponding to the question to be answered by the neural network can have the following format: [One or more "proof" tokens] + [conditioning input] + [one or more "question" tokens] + [text input] + [one or more "answer" tokens]

[0059] In some implementations, the system generates multiple candidate responses for each network input. That is, the system can process each network input a fixed number of times (greater than one) using the neural network to generate several distinct candidate responses. Because the system, as described above, draws samples from probability distributions to generate each token of a given sequence, processing the same network input multiple times generally results in different candidate responses.

[0060] The system generates response data, which represents a final answer to the question, from the network outputs representing the respective candidate responses (step 212).

[0061] In particular, the system can generate a corresponding probability value for each candidate answer, representing the probability that the candidate answer is a correct answer to the question. The system can then select the candidate answer with the highest corresponding probability value as the final answer. In other words, the system "re-ranks" the candidate answers and then selects the candidate answer with the highest ranking as the final answer.

[0062] The reclassification of the candidates' answers will be described below with reference to Fig. 4 described in more detail.

[0063] The system can then provide the response data as output, for example by providing the response data as speech played to a user or as text presented to the user on a user interface.

[0064] Alternatively, after determining the final answer to the question, the system can provide the final answer to one or more external systems for storage or further processing. For example, the system can provide the final answer to a user system implemented on a user's device to deliver the final answer to the user, such as by displaying the final answer on a device display or by generating audio that verbalizes the final answer and playing it back to the user. In some implementations, the neural network is implemented on the user's device; in others, the neural network is implemented on a different device; for example, the neural network may be deployed in the cloud.

[0065] Fig. Figure 3 is a flowchart of an example process 300 for determining relevant conditioning inputs. For simplicity, process 300 is described as being performed by a system of one or more computers located at one or more sites. For example, a neural network system, such as the one described in Fig. 1. The depicted system 100 of the neural network, which is programmed according to this description, performs the process 300.

[0066] The system obtains a set of possible conditioning inputs (step 302). For example, the system can specify any conditioning input using one or more documents identified in the search results obtained from a search engine, as described above.

[0067] The system creates an embedding of the question data (step 304) and a corresponding embedding for each conditioning input (step 306).

[0068] In this description, an embedding is an ordered collection of numerical values ​​that represents an input into a specific embedding space. An embedding can, for example, be a vector of floating-point or other numerical values ​​that has a fixed dimensionality.

[0069] The system can generate the embedding of a sequence of text, e.g., a conditioning input or the question input, in any suitable way.

[0070] For example, the system can generate the embedding using “[term frequency - inverse document frequency]” (tf-idf) embeddings, e.g., by combining the tf-idf embedding of each of the tokens in the text sequence.

[0071] As another example, the system can assign each token to a word embedding, e.g., a word2vec or glove embedding, and then combine the word embeddings, e.g., averaging or summing.

[0072] As another example, the system can process a network input representing the input text using a neural network, e.g., a self-attention-based neural network, to generate the embedding for the input text. The embedding neural network may, for example, have been trained using unsupervised learning, e.g., to minimize BERT loss (Devlin et al. arXiv:1808.04805).

[0073] For each conditioning input, the system determines the similarity between the embedding of the conditioning input and the embedding of the question data (step 308). For example, the similarity between two vectors can be measured according to any suitable similarity measure, e.g., cosine similarity, Euclidean distance, and so on.

[0074] The system selects the conditioning inputs that exhibit the greatest similarity as the most relevant conditioning inputs (step 310). For example, the system can select any conditioning input that meets at least one similarity threshold, or it can select a threshold of the most similar conditioning inputs.

[0075] Fig. Figure 4 is a flowchart of an example Process 400 for re-ranking candidate responses. For simplicity, Process 400 is described as being performed by a system of one or more computers located at one or more sites. For example, a neural network system, such as the one described in Fig. 1. The neural network system 100 shown, which is programmed according to this description, performs the process 400.

[0076] The system receives a set of candidate responses (step 402).

[0077] The system generates a probability value representing the probability that the candidate's answer is a correct answer to the question for each candidate's answer (step 404).

[0078] For example, the system can a for each candidate answer i to the question q, which uses one of the N conditioning inputs p i, from which network inputs were generated, generate a prediction of one or more of the following: p(a i |q) or p(a i ,q|p i ) and use the prediction(s) to generate the probability rating. As a specific example, the probability rating can be equal to p(a). i |q), p(a i ,q|p i ) or a combination of both probabilities using the Product of Experts (PoE). For example, PoE can calculate a product or a weighted sum of the probabilities. In some implementations, PoE also considers, e.g., p(p) i |q) in the product or weighted sum.

[0079] The system can, for example, generate a prediction of: p(ai|q)=∑j=1Np(pj|q)⋅p(a|q,pj) Instead or additionally, the system can generate a prediction of: p(ai,q|pi)=p(q|ai,pi)⋅p(ai|pi)p(q|pi), where pi The conditioning input is used to generate a i was used.

[0080] In some implementations, the system can determine the similarity (or a normalized version of the similarity) as described above with reference to Fig. 3 was calculated as the probability p(p j |q). In some other implementations, the system calculates this probability using the neural network, as described below.

[0081] For a probability p(x|y,z) the element x is sometimes called the "posterior element" and the elements y and z are sometimes called the "prior element".

[0082] The system can use this or another neural network to predict the relevant probabilities; for example, these probabilities can be derived from a smaller, pre-trained language model. For instance, if the neural network is an autoregressive neural network, the system can generate a network input containing one or more of the preceding elements of that probability to generate a prediction for a given probability. It can then use the trained parameter values ​​of the neural network to determine the probability that the neural network will produce a network output representing the posterior element of that probability.

[0083] As a special example, the system can be used to make a prediction for p(a i |q,p i ) to generate a network input containing q and p iincludes and determines the probability that the neural network will produce a network output that a i identified. This probability can be determined, for example, according to the model, from the probability of the last token of a i be determined.

[0084] In some such implementations, the system uses k-shot prompting to generate a network input that includes (i) one or more previous elements of the specified probability and (ii) one or more prompts. The system can then determine the probability that the neural network will produce a network output that identifies the posterior element of the specified probability by deriving the probability of autoregressively generating such a network output in response to processing the network input. In predicting p(p i|q) the system can, for example, display prompts of the form (q', p i ') use and the probability of generating p i Determine for a given q. Similarly for predicting p(a i |q,p i ) the system can display prompts of the form (q',pi',ai') use and the probability of generating a i given q and p i determine. Similarly, in the prediction p(q|a) i ,p i ) the system can display prompts of the form (q',pi',ai') use and calculate the probability of generating q for a given a i and p i determine. Similarly, in the prediction p(a) i |p i ) the system can, for example, display prompts of the form (pi',ai') use and the probability of generating a i given p idetermine. Similarly, in the prediction p(qlp) i ) the system can, for example, display prompts of the form (pi',q) use and calculate the probability of generating q for a given p i determine.

[0085] The system can then select the candidate answer with the highest corresponding probability value as the final answer (step 406).

[0086] As previously described, in some implementations the system is deployed on a digital assistant device, such as a mobile device. In such implementations, the neural network can be (fully) deployed on the mobile device. Processing the search engine query can be performed using a backend data server, which can be implemented with a parallel and distributed computing system. The mobile device typically has less memory than the backend data server, less processing power than the backend data server, or both. Processing power can be measured in operations per second, such as FLOPS (floating-point operations per second). This allows for optimization of the computational load, especially if the neural network is an autoregressive transformer-based neural network and the search engine is an internet search engine.This in turn can facilitate the use of a smaller language model, e.g., one with fewer than 10 billion parameters, such as 7 billion parameters, 1 billion parameters, or less.

[0087] Fig. Figure 5 shows the performance in answering questions with exact agreement (on the y-axis) compared to FLOPS (on the x-axis) by comparing the performance of exemplary neural network systems from the in Fig.The example neural network systems (diagonal lines) are shown in Figure 1, compared to a pre-trained language model that does not also use a search engine (dashed horizontal line and asterisk). The example neural network systems are either 1bn or 7bn parameter versions of a pre-trained Gopher language model (Rae et al. ibid) and they use various different techniques to determine the probability that a candidate answer is correct. The asterisk illustrates the performance of a pre-trained Gopher language model with 280bn parameters. In some cases, the implementations of the described techniques are worth more than 273bn parameters.

[0088] In some implementations, the system can be used as part of a real-world control system, where the question and answer each relate to the operation of a machine part or tool. The process might then involve retrieving a natural language question about the machine part's operation, using a neural network system as described above to process the question to generate the final answer, and then controlling the machine part or tool using that answer. For example, the machine part or tool could be a robot, and the question might be, "What is the maximum safe load for a model 123.45 robot?"; or the machine part or tool could be an oven, and the question might be, "At what temperature should I bake a cake?"

[0089] In this description, the term "configured" is used in connection with systems and computer program components. For a system consisting of one or more computers to be configured to perform certain operations or actions, this means that software, firmware, hardware, or a combination thereof is installed on the system which, when operating, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform certain operations or actions, this means that the one or more programs contain instructions which, when executed by a data processing device, cause the device to perform those operations or actions.

[0090] Embodiments of the subject matter and the functional operations described herein may be implemented in digital electronic circuits or in physically embodied computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more thereof. Embodiments of the subject matter specified herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a physical non-transitory storage medium for execution by, or control of, a data processing device.The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device for direct or serial access, or a combination of one or more of these. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is produced to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0091] The term "data processing device" refers to data processing hardware and encompasses all types of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also consist of or include specialized logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may optionally include code that creates an execution environment for the computer program, such as code representing processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0092] A computer program, which may also be called or described as a program, software, software application, app, module, software module, script, or code, may be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be provided in any form, including as a standalone program or module, component, subroutine, or any other unit suitable for use in a computer environment. A program may correspond to a file in a file system, but it need not. A program may be stored in a portion of a file that contains other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in several coordinated files, such as...Files that store one or more modules, subprograms, or code sections. A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and connected by a data communication network.

[0093] In this description, the term "database" is used in its broadest sense to refer to any collection of data: The data need not be structured in a particular way, or at all, and can be stored on storage devices in one or more locations. Thus, for example, the index database can contain several data collections, each organized differently and accessible in different ways.

[0094] In this description, the term "machine" is similarly used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, a machine is implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular machine; in other cases, multiple machines may be installed and run on the same computer or computers.

[0095] The processes and logic flows described in this document can be executed by one or more programmable computers running one or more computer programs to perform functions by processing input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry, such as an FPGA or an ASIC, or by a combination of dedicated logic circuitry and one or more programmed computers.

[0096] Computers suitable for running a computer program may be based on microprocessors of a general or special type, or on both, or on any other type of central processing unit. Generally, a central processing unit receives instructions and data from read-only memory, random-access memory, or both. The essential elements of a computer are a central processing unit for carrying out or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and memory may be augmented by or incorporated into special logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as hard drives, hard drives, or flash drives.magnetic, magneto-optical, or optical data carriers, or is functionally coupled to one or more of these devices to receive data from them, transmit data to them, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game console, a GPS (Global Positioning System) receiver, or a portable storage device, such as a USB (Universal Serial Bus) flash drive, to name just a few.

[0097] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks and CD-ROM and DVD-ROM disks.

[0098] To provide interaction with a user, the embodiments of the item described in this description can be implemented on a computer that has a display device, e.g., a monitor with a CRT (cathode ray tube) or LCD (liquid crystal display) for showing information to the user, and a keyboard and a pointing device, e.g., a mouse or a trackball, with which the user can provide input to the computer. Other types of devices can also be used for interaction with a user; for example, the feedback provided to the user can be any form of sensory feedback, e.g., visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input.Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received by the web browser. A computer can also interact with a user by sending text messages or other messages to a personal device, such as a smartphone running a messaging application, and receiving reply messages from the user.

[0099] Data processing devices for implementing machine learning models may, for example, also include special hardware accelerator units for processing general and computationally intensive parts of machine learning training or production, e.g., inference workloads.

[0100] Machine learning models can be implemented and deployed using a machine learning framework, such as a TensorFlow framework.

[0101] Implementations of the object described herein may be found in a computer system that includes a back-end component, such as a data server; a middleware component, such as an application server; a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein; or any combination of such back-end, middleware, or front-end components. The system components may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), such as the internet.

[0102] The computing system can include clients and servers. A client and a server are generally located remotely and typically interact via a communication network. The client-server relationship arises from computer programs running on the respective computers, which have a client-server relationship with each other. In some embodiments, a server transmits data, such as an HTML page, to a user device, for example, for the purpose of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, for example, as a result of user interaction, can be received by the device at the server.

[0103] Although this description contains many specific implementation details, these should not be interpreted as limitations on the scope of protection of an invention or on the scope of what can be claimed, but rather as descriptions of features that may be specific to certain embodiments of certain inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in several embodiments or in any suitable subcombination.Furthermore, although features described above may be effective in certain combinations and may even be initially claimed as such, in some cases one or more features from a claimed combination may be extracted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0104] Similarly, while operations are depicted in the drawings and listed in the claims in a specific order, this should not be interpreted as requiring that such operations be performed in the specific order shown or in sequential order, or that all depicted operations must be performed to achieve desirable results. Multitasking and parallel processing may be advantageous under certain circumstances. Furthermore, the separation of different system modules and components in the embodiments described above should not be interpreted as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0105] Certain embodiments of the object have been described. Other embodiments are within the scope of the following claims. For example, the actions mentioned in the claims can be performed in a different order and still achieve desirable results. As an example, the processes shown in the accompanying figures do not necessarily require the specific order shown or a sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; JW Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Glag, J. P. Huang, J. Huang, R. Ring, S. Young, E. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Marten, A. Li, A. Kunco, A. Kunco. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autu, Y. Milik, T. V. Mili, V. Y. Babuschkin, A. Clark, D. de Las Casas, A. Guy,J.Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu und G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-totext transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu und Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs / 2001.09977, 2020; und Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.Language models are few-shot learners. arXiv-Vorabdruck arXiv:2005.14165, 2020

[0028] . Devlin et al. arXiv:1808.04805

[0072]

Claims

[1] System comprising one or more computers and one or more storage devices which store instructions which, when executed by the one or more computers, cause the one or more computers to perform operations which include: Obtaining query data that constitutes a question; Generating a search engine query for a search engine from the query data; Obtaining a large number of documents identified by the search engine in response to processing the search engine query; Generating a multitude of conditioning inputs, each representing at least one section of one or more of the obtained documents, from the multitude of documents; for each conditioning input generated from a multitude of inputs, processing a network input generated from (i) the question data and (ii) the conditioning input using a neural network to produce a network output that represents a candidate response to the question; and Generating response data that represents a final answer to the question from the network outputs representing the respective candidate responses. [2] System according to claim 1, wherein generating a plurality of conditioning inputs from the plurality of documents for each of the plurality of documents comprises: Extracting a text sequence from each obtained document; and Determining a suitable conditioning input from each of one or more subsequences of the extracted text sequence. [3] System according to one of claims 1 or 2, wherein the processes further comprise: For each conditional input that represents at least one section of one or more relevant documents, generate a relevance score that represents a predicted relevance of the section of the one or more relevant documents to the question; and Determining a classification of the conditioning inputs according to the corresponding relevance ratings, where the subset of conditioning inputs includes the n highest-ranked conditioning inputs in the ranking, n ≥ 1. [4] System according to claim 3, wherein generating the relevance ratings for the conditioning inputs comprises: Creating an embed of the question data; Generating a separate embedding for each conditioning input; and For each conditioning input, determine a similarity between the embedding of the conditioning input and the embedding of the question data. [5] System according to any one of claims 1-4, wherein for each network input generated from a respective conditioning input, the network input is further generated from one or more prompt tuples, each comprising (i) second question data representing a different second question, (ii) second response data representing a response to the respective second question, and (iii) a second conditioning input representing a document relevant to the respective second question. [6] System according to claim 5, wherein the neural network was pre-trained on a language modeling task and not on a question-answering task. [7] System according to any one of claims 1-6, wherein generating response data representing a final answer to the question from the network outputs representing the respective candidate responses comprises: Generating a probability value representing the probability that the candidate's answer is a correct answer to the question for each candidate's answer; and Determine the final answer as the candidate answer with the highest probability value. [8] System according to claim 7, wherein for each candidate response a' to question q corresponding to a respective conditioning input p, generating the respective probability value for the candidate response a' comprises: Generating one or more initial probability values, wherein the one or more initial probability values ​​comprise one or more of the following: a first initial probability value, which represents an estimate of the probability p(a'|p,q) of the candidate's response given the conditioning input and the question, a second initial probability value, which represents an estimate of the probability p(a'|q) of the candidate's answer given the question, a third initial probability value, which represents an estimate of the probability p(p|q) of the conditioning input given the question, a fourth initial probability value, which represents an estimate of the probability p(q|p) of the question given the conditioning input, or a fifth initial probability value, which represents an estimate of the probability p(q|a',p) of the question given the candidate's response and the conditioning input; and Generating the probability value from one or more initial probability values. [9] System according to claim 8, wherein for at least one of the initial probability values, which represents an estimate for a probability of a respective posterior element in view of one or more respective prior elements, the initial probability value is generated using the neural network. [10] System according to claim 9, wherein generating a certain initial probability value, which represents an estimate of a probability of a certain posterior element given one or more certain prior elements, using the neural network comprises: Determining a probability, using a set of trained parameter values ​​of the neural network, that the neural network would produce a second network output representing the specified posterior element in response to the processing of a second network input generated by one or more specified prior elements. [11] System according to claim 10, wherein the second network input was further generated from one or more input prompts having the same specific posterior element and one or more specific prior elements as the specific initial probability value. [12] System according to any of the preceding claims, wherein the neural network comprises an autoregressive transformer-based neural network and wherein the search engine comprises an Internet search engine. [13] System according to any of the preceding claims, wherein the processes further comprise: Implementing the neural network on a mobile device; Processing the search engine query using a backend data server; and where the mobile device has less memory and less computing power than the backend data server, or both. [14] One or more computer storage media that store instructions which, when executed by one or more computers, cause the one or more computers to perform operations that include: Obtaining query data that constitutes a question; Generating a search engine query for a search engine from the query data; Obtaining a large number of documents identified by the search engine in response to processing the search engine query; Generating a multitude of conditioning inputs, each representing at least one section of one or more of the obtained documents, from the multitude of documents; for each conditioning input generated from a multitude of inputs, processing a network input generated from (i) the question data and (ii) the conditioning input using a neural network to produce a network output that represents a candidate response to the question; and Generating response data that represents a final answer to the question from the network outputs representing the respective candidate responses. [15] One or more computer storage media according to claim 14, wherein generating a plurality of conditioning inputs from the plurality of documents for each of the plurality of documents comprises: Extracting a text sequence from each obtained document; and Determining a suitable conditioning input from each of one or more subsequences of the extracted text sequence. [16] One or more computer storage media according to one of claims 14 or 15, wherein the operations further comprise: For each conditional input that represents at least one section of one or more relevant documents, generate a relevance score that represents a predicted relevance of the section of the one or more relevant documents to the question; and Determining a classification of the conditioning inputs according to the corresponding relevance ratings, where the subset of conditioning inputs includes the n highest-ranked conditioning inputs in the ranking, n ≥ 1. [17] One or more computer storage media according to claim 16, wherein generating the relevance ratings for the conditioning inputs comprises: Creating an embed of the question data; Generating a separate embedding for each conditioning input; and For each conditioning input, determine a similarity between the embedding of the conditioning input and the embedding of the question data. [18] One or more computer storage media according to one of claims 15-17, wherein for each network input generated from a respective conditioning input, the network input is further generated from one or more prompt tuples, each comprising (i) second question data representing a different second question, (ii) second answer data representing an answer to the respective second question, and (iii) a second conditioning input representing a document relevant to the respective second question. [19] One or more computer storage media according to claim 18, wherein the neural network was pre-trained on a language modeling task and not on a question-answering task. [20] One or more computer storage media according to any one of claims 14-19, wherein generating response data representing a final answer to the question from the network outputs representing the respective candidate responses comprises: Generating a probability value representing the probability that the candidate's answer is a correct answer to the question for each candidate's answer; and Determine the final answer as the candidate answer with the highest probability value. [21] One or more computer storage media according to claim 20, wherein for each candidate response a' to question q corresponding to a respective conditioning input p, generating the respective probability value for the candidate response a' comprises: Generating one or more initial probability values, wherein the one or more initial probability values ​​comprise one or more of the following: a first initial probability value, which represents an estimate of the probability p(a'|p,q) of the candidate's response given the conditioning input and the question, a second initial probability value, which represents an estimate of the probability p(a'|q) of the candidate's answer given the question, a third initial probability value, which represents an estimate of the probability p(p|q) of the conditioning input given the question, a fourth initial probability value, which represents an estimate of the probability p(q|p) of the question given the conditioning input, or a fifth initial probability value, which represents an estimate of the probability p(q|a',p) of the question given the candidate's response and the conditioning input; and Generating the probability value from one or more initial probability values. [22] One or more computer storage media according to claim 21, wherein for at least one of the initial probability values, which represents an estimate for a probability of a respective posterior element in view of one or more respective prior elements, the initial probability value is generated using the neural network. [23] One or more computer storage media according to claim 22, wherein generating a certain initial probability value, which represents an estimate of a probability of a certain posterior element given one or more certain prior elements, using the neural network comprises: Determining a probability, using a set of trained parameter values ​​of the neural network, that the neural network would produce a second network output representing the specified posterior element in response to the processing of a second network input generated by one or more specified prior elements. [24] One or more computer storage media according to claim 23, wherein the second network input was further generated from one or more prompts having the same specific posterior element and one or more specific prior elements as the specific initial probability value. [25] One or more computer storage media according to one of claims 14-24, wherein the neural network comprises an autoregressive transformer-based neural network and wherein the search engine comprises an Internet search engine. [26] One or more computer storage media according to any one of claims 14-25, wherein the operations further comprise: Implementing the neural network on a mobile device; Processing the search engine query using a backend data server; and where the mobile device has less memory and less computing power than the backend data server, or both.

Images