Extending machine learning language models using search engine results

Abstract

The present invention provides a method, system, and apparatus, including a computer program encoded on a computer storage medium, for extending machine learning language models using search engine results. [Solution] One method includes the steps of: obtaining question data representing a question; generating a search engine query for a search engine from the question data; obtaining a plurality of documents identified by the search engine in response to processing the search engine query; generating a plurality of conditioned inputs from the plurality of documents, each representing one or more parts of the obtained documents; for each of the plurality of generated conditioned inputs, using a neural network to process (i) the question data and (ii) the network input generated from the conditioned input to generate a network output representing a candidate answer to the question; and generating answer data representing a final answer to the question from the network outputs representing each candidate answer.

Description

[Technical Field]

[0001] This specification relates to processing input using neural networks. [Background technology]

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for an incoming input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, for example, the next hidden layer or output layer. Each layer of the network generates an output from an incoming input according to the current values ​​of its respective set of parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al., "Training compute-optimal large language models", arXiv preprint arXiv:2203.15556, 2022 [Non-Patent Document 2] JW Rae、S. Borgeaud、T. Cai、K. Millican、J. Hoffmann、HF Song、J. Aslanides、S. Henderson、R. In、S. Young、E. Rutherford、T. Hennigan、J. Menick、A. Cassirer、R. Powell、G. van den Driessche、LA Hendricks、M. Rauh、P. Huang、A. Glaese、J. Welbl、S. Dathathri、S. Huang、J. Uesato、J. Mellor、I. Higgins、A. Creswell、N. McAleese、A.Wu、E. Elsen、SM Jayakumar、E. Buchatskaya、D. Budden、E. Sutherland、K. Simonyan、M. Paganini、L. Cipher、L. Martens、XL Li、A. Kuncoro、A. Nematzadeh、E. Gribovskaya、D. Donato、A. Lazaridou、A. Mensch、J. Lespiau、M. Tsimpoukelli、N. Grigorev、D. Fritz、T. Sottiaux、M. Pajarskas、T. Pohlen、Z. Gong、D. Toyama、C. of Masson d' Autume、Y. Li、T. Terzi、V. Mikulik、I. Babuschkin、A. Clark、D. of the Houses、A. Guy、C. Jones、J. Bradbury、M. Johnson、BA Hechtman、L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG.Irving, “Scaling language models: Methods, analysis & insights from training gopher,” CoRR, abs / 2112.11446, 2021. [Non-Patent Document 3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer", arXiv preprint arXiv:1910.10683, 2019 [Non-Patent Document 4] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020 [Non-Patent Document 5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners", arXiv preprint arXiv:2005.14165, 2020 [Non-Patent Document 6] Kudo et al., arXiv:1808.06226 [Non-Patent Document 7] Devlin et al., arXiv:1808.04805 [Overview of the project] [Means for solving the problem]

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that runs a neural network configured to process a network input representing an input text and to generate a network output representing a prediction about the input text.

[0005] To generate network output, the system is configured to generate a search engine query from the input text and, in response to the query, retrieve a set of results from the search engine. The system can then use a neural network to incorporate the results of the search engine query into the network input before processing the network input. In this way, the system can perform neural network-based language processing tasks, such as question answering, using significantly fewer computing resources than would otherwise be required. Furthermore, the network output can incorporate information embedded in the search engine results, including more recent information that was not available during the neural network training, thus improving predictions about the input text.

[0006] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages:

[0007] Large language models implemented as neural networks can produce remarkable results in various natural language processing tasks, including question answering. However, the implementation forms of some of these models, particularly transformer-based models, may have over a billion parameters and can require significant computing resources, power, and time to process network inputs and generate network outputs. Such models may have over 10 billion or over 100 billion parameters. If such models were used at scale, for example, in a search engine, environmental considerations could also arise.

[0008] Additional considerations arise when a neural network is implemented in a computing system that includes a digital assistant device, particularly a data server, which is a backend component communicating with the digital assistant device via a data communication network such as the Internet, for example, when implemented on a mobile device. Then, it is necessary to optimize the computing load between the digital assistant device and the backend component. This need can be particularly critical in the case of large language models due to their significant memory and computing requirements compared to the requirements typically seen on mobile devices.

[0009] The techniques described herein address these problems. In some implementations, the techniques enable, particularly when a large language model is implemented as a neural network in a multi-tasking and parallel processing computer system that is distributed across multiple sites and interconnected by a data communication network, a reduction in computational load and an improvement in load distribution.

[0010] In some implementations, the techniques described enable a beneficial distribution of computing load between local mobile computing devices and backend servers within a network. More specifically, in an implementation, a language model neural network is conditioned on a plurality of conditioning inputs representing documents retrieved from an Internet search based on a question, as well as on question data from the question, enabling the use of a smaller language model neural network, thereby enabling the implementation of a neural network on a mobile device with limited memory and computing resources.

[0011] Further, using the techniques described herein, a system can utilize search engine results and use the latest information included in the search engine results to generate predictions about input text. Some existing systems use pre-trained neural networks to generate predictions without access to such search engine results, and thus the reliability of the predictions may be low because the neural network can only encode information that was available to the neural network during training, i.e., these predictions rely on outdated information and can thus be inaccurate or at least old. Accordingly, using the techniques described herein, a system can generate more accurate and timely predictions.

[0012] Furthermore, some existing systems must repeatedly re-train a neural network to ensure that the neural network encodes the latest information. The systems described herein can repeatedly access new search engine results, so the systems do not need to re-train the neural network, thus saving significant computing resources.

[0013] Using the techniques described herein, a system can, in response to processing a search engine query, generate a prediction for input text using encoded information from multiple different documents provided by the search engine. Each of these different documents may contain different information relevant to the prediction. Therefore, the prediction generated by the system may be more accurate than one generated using a single document.

[0014] Furthermore, the system described by generating multiple answers using multiple extracted pieces of evidence, i.e., multiple different conditioned inputs, followed by a re-ranking stage using scores generated by the same language model neural network that generated the answers, improves the quality of the generated answers without requiring larger, harder training of the neural network. That is, by extending the generation process as described above, the system can generate answers that surpass the quality of answers generated by larger neural networks that do not access a search engine or generate only a single answer in response to inputs containing a single conditioned input. Thus, these extension techniques can mitigate the lower performance problem of smaller pre-trained neural networks and may be particularly suitable for deployments on devices with constrained memory space, such as mobile devices, smart speakers, or other edge devices, where it is difficult to efficiently store models with extremely large computational footprints, for example, leading to a very large number of parameters.

[0015] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0016] [Figure 1] This is a diagram illustrating an exemplary neural network system. [Figure 2] This is a flowchart illustrating an exemplary process for generating the final answer to a received question. [Figure 3] This is a flowchart illustrating an exemplary process for determining the relevant conditioning inputs. [Figure 4] This is a flowchart illustrating an exemplary process for re-ranking candidate answers. [Figure 5] This figure shows the performance improvements that can be achieved through the example of a neural network system. [Modes for carrying out the invention]

[0017] Similar reference numbers and names in various drawings refer to the same elements.

[0018] Figure 1 shows an exemplary neural network system 100. The neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below may be implemented.

[0019] System 100 receives input text 102 and generates network output 112 representing a prediction about input text 102.

[0020] In detail, system 100 uses a neural network 110 configured to process a network input 104 representing an input text 102 and to generate a network output 112 representing a prediction about the input text 102. In implementations, the neural network 110 may be a pre-trained neural network.

[0021] To generate network output 112, the system 100 is configured to generate a search engine query 114 for the search engine 120 from the input text 102, and to retrieve a set of search results 122 from the search engine 120 in response to the query 114. Each search result 122 identifies its own document.

[0022] The search engine 120 can be any suitable search engine that is accessible by the system 100 and searches any suitable corpus of documents, such as web pages, books, or other documents. For example, the search engine 120 could be an internet search engine that searches through the internet and returns results 122 that refer to documents available on the internet. As another example, the search engine 120 could be a different search engine that searches a private corpus of documents, such as documents available on an internal network, or documents stored in a collection of one or more databases.

[0023] Next, system 100 can use the neural network 110 to incorporate the results 122 of the search engine query 114 into the network input 104 before processing the network input 104. Thus, the network output 112 incorporates the information embedded in the search engine results 122, including the latest information that was not available during the training of the neural network 110, and can therefore improve predictions about the input text 102.

[0024] The neural network 110 may have any suitable neural network architecture that enables the model to map input sequences of text tokens from the vocabulary to output sequences of text tokens from the vocabulary.

[0025] For example, the neural network 110 may have an encoder-decoder transformer-based architecture.

[0026] As another example, the neural network 110 may have a transformer-based architecture consisting only of decoders, where the input sequence is provided to the neural network 110 as a "prompt".

[0027] Generally, transformer-based architectures can be characterized by having a sequence of self-attention neural network layers. Each self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism across the attention layer inputs to generate an attention layer output for each element of the input. There are numerous different attention mechanisms that can be used.

[0028] More specifically, the neural network 110 may be an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each specific text token in the output sequence, given the current input sequence which includes (i) an input sequence and (ii) any text tokens that precede a particular text token in the output sequence that follows.

[0029] More specifically, to generate a particular text token, the neural network 110 can process the current input sequence to generate a score distribution, e.g., a probability distribution, which assigns a score, e.g., a probability, to each token in the vocabulary of text tokens. The neural network 110 can then use the score distribution to select a text token from the vocabulary as the particular text token. For example, the neural network 110 can greedily select the token with the highest score, or it can sample tokens from the distribution using, for example, top-k sampling, nucleus sampling, or another sampling technique.

[0030] As a specific example, neural network 110 may be an autoregressive transformer-based neural network that includes multiple layers, each applying a self-attention operation. Neural network 110 may have any of the various transformer-based neural network architectures. Examples of such architectures include J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al., "Training compute-optimal large language models," arXiv preprint arXiv:2203.15556, 2022, and J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, LA Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, XL Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d' Autume, Y.Li、T. Terzi、V. Mikulik、I. Babuschkin, A. Clark、D. of The Houses、A. Guy、C. Jones, J.S. Bradbury、M. Johnson, BA Hechtman, L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG. Irving, CoRR, abs / 2112.11446, 2021, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, and Yanqi Zhou, Wei Li, and Peter J Liu, arXiv, arXiv:1910.10683, 2019, Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Roman Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le chatbot」、CoRR、abs / 2001.This includes works described in 09977, 2020, and in the arXiv preprint arXiv:2005.14165, 2020, by Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., titled "Language models are few-shot learners."

[0031] Tokens in a vocabulary can be any suitable text token, such as a word, wordpiece, punctuation mark, character, byte, etc., that represents one or more elements of text in a natural language, and possibly a number and other textual symbols found in a corpus of text. For example, system 100 can tokenize a given sequence of words by applying a tokenizer, such as the SentencePiece tokenizer (Kudo et al., arXiv:1808.06226) or another tokenizer, to divide the sequence into tokens from the vocabulary.

[0032] Before using the neural network 110 to generate the network output 112, the neural network 110 is pre-trained, for example, by system 100 or by one or more other systems.

[0033] More specifically, system 100 or another system pre-trains a neural network 110 in language modeling tasks, such as tasks that require predicting the next token following a given sequence of text tokens in training data, given a current sequence of text tokens. That is, a language modeling task may require predicting the text sequence following a given unlabeled text sequence in a corresponding document for each given unlabeled text sequence in the training dataset. In a particular example, the language modeling neural network 110 may be pre-trained for a maximum-likelihood objective on a large dataset of text, for example, publicly available text from the internet or another text corpus.

[0034] After training, system 100 can be configured to perform any suitable machine learning task using search engine results 122 and the neural network 110.

[0035] For example, system 100 may be configured to perform question answering, where input text 102 identifies a question and network output 112 represents the answer to the question.

[0036] For example, a question may be provided by a user of system 100, for example, by providing input text 102 directly to the system, or by providing system 100 with audio data representing a verbalization of the input text 102. In these implementations, the search engine results 122 may include a set of documents related to the question, and thus the neural network 110 can leverage the search engine results to answer the question using information that was not available at the time the neural network 110 was trained. In these implementations, the data representing the input text 102 may be called “question data”.

[0037] As another example, system 100 may be configured to perform fact-checking, where input text 102 represents a description and network output 112 represents a prediction of whether the description is factually accurate. Similarly, in these implementations, search engine results 122 may include a set of documents related to the description.

[0038] In some implementations, system 100 uses the neural network 110 to perform one of these downstream tasks, such as question answering, without further training the neural network 110. For example, system 100 can perform a question answering task using a neural network 110 that has only been trained on a language modeling task.

[0039] The following description refers to an implementation in which system 100 is configured to perform question answering, but it should be understood that, in general, the neural network 110 can be configured to perform any appropriate task using input text.

[0040] When system 100 generates network output 112, system 100 can provide network output 112 to the user.

[0041] For example, system 100 may be implemented as part of a digital assistant device, such as a mobile device, smartwatch or other wearable device, or smart speaker device, or may be communicating with such a device, and the digital assistant device may provide the network output 112 to the user by, for example, generating sound representing the network output 112 and playing the sound to the user via a speaker.

[0042] As another example, system 100 can provide a network output 112 for presentation in the user interface of a user device, for example, through which the user has submitted input text 102.

[0043] Figure 2 is a flowchart of an exemplary process 200 for generating network output. For convenience, process 200 is described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, e.g., the neural network system 100 shown in Figure 1, can perform process 200.

[0044] The system retrieves question data representing the question (step 202).

[0045] The system generates search engine queries for the search engine from the question data (step 204).

[0046] The system can generate a search engine query from input text, i.e., from question data, in any suitable manner. For example, in an implementation where a neural network is configured to perform question answering, the search engine query may be (or may contain) the input text itself, e.g., the question that the neural network will answer. In other words, the search engine query may be equivalent to the input text as is. As another example, the system may process the input text to generate updated text that acts as a search engine query, for example, by processing the input text using one or more predetermined templates. As yet another example, the system may process the input text using a machine learning model, for example, by processing a sequence of tokens representing the input text using another neural network, to generate a network output representing the text of the search engine query.

[0047] The system retrieves multiple documents identified by the search engine in response to processing the search engine query (step 206). For example, the system may receive a set of search results from the search engine, each identifying a document from the corpus of documents searched by the search engine.

[0048] The system can submit a search engine query to a search engine and receive a set of multiple documents D returned. For example, the system can submit a search engine query using an application programming interface (API) provided by the search engine. The system may be configured to retrieve a predetermined number of documents D, i.e., p documents indexed by the search engine and ranked highest by the search engine in response to a submitted search engine query. In some implementations, the system receives each document in HTML format and processes the HTML data to extract the clean text of the document.

[0049] The system generates multiple conditional inputs from multiple documents, each representing one or more portions of the retrieved documents (step 208).

[0050] Each conditioned input represents part or all of the text of the corresponding document, for example, each conditioned input may contain different subsequences of the sequence of tokens that represent the corresponding text. Since at least some of the documents D may be represented by sequences longer than the maximum sequence length that can be processed by the neural network, the system can generate multiple different conditioned inputs for a single document, where each conditioned input represents different subsets of the document's text, for example, disparate subsets.

[0051] In some implementations, the system can generate conditional input from multiple different documents, for example, by concatenating each subset of the text from each document.

[0052] As a specific example, for each retrieved document, the system can segment the document into segments of a fixed size. Such segments may be called “paragraphs,” but they do not necessarily correspond to the original paragraphs of the document. For example, each segment may contain a predetermined number of sentences from the original document (e.g., a predetermined number of text sequences separated by periods or other specified punctuation marks). The system can then generate a corresponding conditional input from each segment, for example, by generating a sequence of tokens representing the text of the segment.

[0053] For each of the multiple generated conditioned inputs, the system uses a neural network to process (i) the question data and (ii) the network input generated from the conditioned input to generate a network output representing a candidate answer to the question (step 210).

[0054] In some implementations, instead of generating a network input for each conditioned input, the system can determine which of the generated conditioned inputs is most relevant to the question expressed by the input text, and then generate a network input only for the most relevant conditioned input.

[0055] Determining the relevant conditioned inputs is explained below with reference to Figure 3.

[0056] A network input for a given conditional input may include both the input text and the conditional input, or may be generated from both. For example, the system may concatenate the input text and the conditional input to generate a single sequence, where, for example, one or more delimiter tokens separating subsequences represent the input text and the subsequences represent the conditional input.

[0057] In some implementations, the system uses k prompt tuples to generate a network input corresponding to each conditioned input, where k ≥ 1. Each prompt tuple may correspond to a different second question, which the neural network will answer. Each prompt tuple may contain (i) second question data representing each second question (e.g., a sequence of tokens representing the second question), (ii) second answer data representing the answer to the second question (e.g., a sequence of tokens representing the answer), and (iii) a second conditioned input representing a document related to the second question (e.g., a sequence of tokens representing the document). For example, the system can obtain prompt tuples from a predetermined training dataset. Including prompt tuples in the network input is sometimes called "k-shot prompting."

[0058] In some implementations, the neural network is a pre-trained neural network that has been trained on a machine learning task different from the task on which the system is using the neural network, for example, a task different from the question-answering task. Therefore, in these implementations, the neural network is not trained to produce a network output that has the appropriate format for the machine learning task on which the system is using the neural network, for example, not trained to produce a network output that has the format required for question-answering. Thus, prompt tuples may be included in the network input so that the neural network has access to examples of the appropriate format for the neural network output.

[0059] For example, for each conditioning input, the system can generate a network input containing k+1 subsequences, where k subsequences correspond to each second question, and the (k+1)th subsequence corresponds to the question that the neural network will answer. The k subsequences corresponding to the second question may each contain the corresponding second question data, second answer data, and second conditioning input, respectively, while the subsequence corresponding to the question that the neural network will answer may contain only the input text representing the question and conditioning input (i.e., it does not contain the corresponding answer that will be generated by the neural network). Therefore, the k subsequences corresponding to the second question can represent the appropriate format of the neural network's network output.

[0060] As a specific example, each subsequence corresponding to each second question may have the following format: [One or more "evidence" tokens] + [Second conditional input] + [One or more "question" tokens] + [Second question data] + [One or more "answer" tokens] + [Second answer data] However, the “evidence” token, the “question” token, and the “answer” token are arbitrary, predetermined tokens. The subsequence corresponding to the question to be answered by the neural network may have the following format: [One or more "evidence" tokens] + [Conditional input] + [One or more "question" tokens] + [Text input] + [One or more "answer" tokens]

[0061] In some implementations, the system generates multiple candidate answers for each network input. That is, the system can process each network input using the neural network a fixed (greater than 1) number of times to generate multiple different candidate answers. As explained above, the system samples from a probability distribution to generate each token in a given output sequence, so processing the same network input multiple different times generally results in the generation of different candidate answers.

[0062] The system generates answer data representing the final answer to the question from the network output representing each candidate answer (step 212).

[0063] In detail, for each candidate answer, the system can generate a likelihood value for that answer, representing the likelihood that the answer is the correct answer to the question. The system can then select the candidate answer with the highest corresponding likelihood value as the final answer. In other words, the system "re-ranks" the candidate answers and then selects the highest-ranked candidate answer as the final answer.

[0064] The process of re-ranking the candidate answers will be explained in more detail below, with reference to Figure 4.

[0065] The system can then provide the response data as output, for example, by providing the response data as audio to be played back to the user, or as text to be presented to the user in the user interface.

[0066] Alternatively, after determining the final answer to a question, the system may provide the final answer to one or more external systems for storage or further processing. For example, the system may provide the final answer to a user system implemented on the user's device for providing the final answer to the user, for example, by displaying the final answer on the device's display, or by generating audio data representing a verbalization of the final answer and playing the audio data for the user. In some implementations, the neural network is implemented on the user's device, while in some other implementations, the neural network is implemented on a different device, for example, the neural network may be deployed on the cloud.

[0067] Figure 3 is a flowchart of an exemplary process 300 for determining the relevant conditioning inputs. For convenience, the process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, e.g., the neural network system 100 shown in Figure 1, can perform the process 300.

[0068] The system obtains a set of candidate conditional inputs (step 302). For example, the system may generate each conditional input using one or more documents identified in the search results obtained from a search engine, such as those described above.

[0069] The system generates embeddings for the question data (step 304) and generates embeddings for each conditional input (step 306).

[0070] In this specification, an embedding is an ordered set of numbers representing inputs within a particular embedding space. For example, an embedding may be a vector of floating-point or other numerical numbers having a fixed number of dimensions.

[0071] The system can generate embedded sequences of text, such as conditional inputs or question inputs, in an appropriate manner.

[0072] For example, a system can use "term frequency - inverse document frequency" (tf-idf) embeddings to generate embeddings, for instance, by combining the tf-idf embeddings of each token in a text sequence.

[0073] As another example, the system could map each token to a word embedding, such as a word2vec or Glove embedding, and then combine the word embeddings, for example, by averaging or summing them.

[0074] As another example, the system could use an embedded neural network, such as a self-attention-based neural network, to process a network input representing the input text and generate an embedding for the input text. For example, the embedded neural network might be trained through unsupervised learning to minimize, for example, the BERT loss (Devlin et al., arXiv:1808.04805).

[0075] For each conditioned input, the system determines the similarity between the embedding of the conditioned input and the embedding of the question data (step 308). For example, the similarity may be measured according to any suitable similarity measure between the two vectors, such as cosine similarity, Euclidean distance, etc.

[0076] The system selects the conditional input with the highest similarity as the most relevant conditional input (step 310). For example, the system may select each conditional input having at least a threshold similarity, or it may select the conditional input with the most similar threshold number.

[0077] Figure 4 is a flowchart of an exemplary process 400 for re-ranking candidate answers. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a neural network system appropriately programmed according to this specification, e.g., the neural network system 100 shown in Figure 1, can perform process 400.

[0078] The system receives a set of candidate answers (step 402).

[0079] For each candidate answer, the system generates a likelihood value for that answer, which represents the likelihood that the candidate answer is the correct answer to the question (step 404).

[0080] For example, N conditional inputs p from which network inputs are generated. i Each candidate answer a for question q generated using one of the following i Regarding this, the system is p(a i |q) or p(a i ,q|p i One or more predictions can be generated from p(a), and these predictions can be used to generate a likelihood score. As a specific example, the likelihood score is p(a). i |q) or p(ai , q|p i ) or a combination of both probabilities using the product-of-experts (PoE). For example, PoE can calculate the product or weighted sum of probabilities. In some implementations, PoE also considers p(p i |q) in the product or weighted sum, for example, includes it.

[0081] For example, the system can generate the following predictions.

[0082]

Number

[0083] Instead or in addition, the system can generate the following predictions,

[0084]

Number

[0085] where p i is the conditional input used to generate a i .

[0086] In some implementations, the system can use the similarity (or a normalized version of the similarity) calculated as described above while referring to FIG. 3 as the above probability p(p j |q). In some other implementations, the system calculates this probability using a neural network as described below.

[0087] For the probability p(x|y,z), the element x may be called the "posterior element", and the elements y and z may be called the "prior elements".

[0088] The system can predict each probability using its own or another neural network, and for example, these likelihoods may be obtained from a smaller, pre-trained language model. For example, if the neural network is an autoregressive neural network, to generate a prediction for a particular probability, the system can generate a network input containing one or more prior elements of the particular probability, and use the trained parameter values ​​of the neural network to determine the likelihood that the neural network will produce a network output representing the posterior elements of the particular probability.

[0089] As a specific example, p(a i |q,p i To generate predictions for ), the system uses q and p i Generate a network input that includes a i We can determine the likelihood that will generate a network output that identifies a. For example, this likelihood can be calculated according to the model, a i This can be determined from the probability of the last token.

[0090] In some such implementations, the system uses k-shot prompting to generate a network input that includes (i) one or more prior elements of a particular probability and (ii) one or more prompt tuples. The system can then determine the probability that generates such a network output by inferring the likelihood that the neural network autoregressively generates a network output that identifies the posterior elements of a particular probability in response to processing the network input. For example, p(p i When predicting |q), the system is (q',p i Using a prompt tuple of the form '), given q, p i The likelihood of generating can be determined. Similarly, p(a i |q,p i When predicting (q',p i ',ai Use a prompt tuple of the form ') and q and p i Given, a i The likelihood of generating can be determined. Similarly, p(q|a i ,p i When predicting (q',p i ',a i Using a prompt tuple of the form ')', i and p i Given a, we can determine the likelihood of generating q. Similarly, p(a i |p i When predicting ), the system predicts (p i ',a i Using a prompt tuple of the form '), p i Given, a i The likelihood of generating can be determined. Similarly, p(q|p i When predicting ), the system predicts (p i Using a prompt tuple of the form ',q), p i Given a value, we can determine the likelihood of generating q.

[0091] Next, the system can select the candidate answer with the highest corresponding likelihood value as the final answer (step 406).

[0092] As previously described, in some implementations, the system is implemented on a digital assistant device such as a mobile device. In such implementations, the neural network can be implemented (completely) as a neural network on the mobile device. Search engine queries may be processed using a backend data server, which may be implemented using parallel and distributed computing systems. Mobile devices generally have less working memory, less computing power, or both, than a backend data server. Computing power can be measured in computing operations per second, e.g., FLOPS (floating point operations per second). In this way, the computational load can be optimized, especially if the neural network is an autoregressive, transformer-based neural network, and if the search engine is an internet search engine. This then makes it possible to use smaller language models, e.g., fewer than 10 billion parameters, e.g., 7 billion parameters, 1 billion parameters, or fewer.

[0093] Figure 5 shows the exact match question answering performance (on the y-axis) against FLOPS (on the x-axis), comparing the performance (slant line) of an exemplary neural network system of the type shown in Figure 1 with the performance (horizontal dashed line and asterisks) of a pre-trained language model that does not use a search engine. The exemplary neural network system is either a 1 billion or 7 billion parameter version of the pre-trained Gopher language model (Rae et al., ibid.), using various different techniques to determine the likelihood that a candidate answer is the correct answer. The asterisks represent the performance of the pre-trained Gopher language model with 280 billion parameters. In some cases, implementations of the techniques described are worth more than 273 billion parameters.

[0094] In some implementations, the system may be used as part of a real-world control system, where the questions and answers relate, respectively, to the operation of machine or tool items in the real world. The method may then include obtaining a natural language question about the operation of the machine item, processing the question using a neural network system as described above to generate a final answer to the question, and then using the answer to control the machine or tool item. For example, the machine or tool item may be a robot and the question may be "What is the maximum safe load for the Model 123.45 robot?", or the machine or tool item may be an oven and the question may be "At what temperature should the cake be baked?".

[0095] This specification uses the term “configured” in relation to system and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action while in operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action.

[0096] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or sequential access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0097] The term “data processing device” refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In some cases, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code that constitutes one or more of these.

[0098] Computer programs, sometimes called, or sometimes described as, programs, software, software applications, apps, modules, software modules, scripts, or code, may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program may be stored in a single file dedicated to the program in question, in part of a file holding one or more scripts stored in other programs or data, such as a markup language document, or in a set of coordinated files, such as a file storing one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or on multiple computers located in one site, or distributed across multiple sites and interconnected by a data communication network.

[0099] In this specification, the term “database” is used broadly to refer to any collection of data that may be stored on a storage device in one or more locations, and which does not need to be structured in any particular way, or does not need to be structured at all. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0100] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same one or more computers.

[0101] The processes and logic flows described herein may be executed by one or more programmable computers running one or more computer programs to perform their functions by operating on input data and generating outputs. The processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0102] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are the central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into dedicated logic circuits. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operablely coupled to them for receiving data from them, transferring data to them, or both. However, a computer is not required to have such devices. Moreover, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0103] Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0104] To provide user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used similarly to provide user interaction, for example, the feedback given to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser in response to a request received from a web browser on the user's device. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0105] Data processing devices for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling machine learning training or fabrication, such as inference, common and computationally intensive parts of the workload.

[0106] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0107] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or a middleware component, for example, an application server, or a client computer having a frontend component, for example, a graphical user interface, a web browser, or an application through which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communications in any form or medium, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0108] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device to display data to a user interacting with a device acting as a client and to receive user input from the user. Data generated on the user device, such as the results of user interaction, may be received from the device to the server.

[0109] This specification includes numerous specific implementation details, which should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in relation to separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in relation to a single embodiment may also be implemented separately in multiple embodiments or in any preferred partial combination. Furthermore, features may be described above as acting in several combinations, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0110] Similarly, while operations are illustrated in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific illustrated order or sequence, or that all illustrated operations be performed to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0111] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the processes illustrated in the appended drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of Symbols]

[0112] 100 Neural Network Systems, Systems 102 Input text, text input 104 Network Input 110 Neural Networks, Language Model Neural Networks 112 Network Output 114 Search engine queries, queries 120 search engines 122 search results, results, search engine results

Claims

1. Steps to obtain question data representing the question, The steps include generating search engine queries for a search engine from the aforementioned question data, The steps include obtaining a plurality of documents identified by the search engine in response to the processing of the search engine query, The steps include generating a plurality of conditional inputs from the plurality of documents, each representing one or more parts of the acquired documents, For each of the multiple generated conditioned inputs, the step of using a neural network to process (i) the question data and (ii) the network input generated from the conditioned input to generate a network output representing a candidate answer to the question, The steps include generating answer data representing the final answer to the question from the network output representing each candidate answer, and A method that includes this.

2. The step of generating multiple conditional inputs from the multiple documents is performed for each of the multiple documents, The steps involve extracting a text sequence from each acquired document, The steps include determining each conditional input from each of one or more subsequences of the extracted text sequence, and The method according to claim 1, including the method described in claim 1.

3. For each conditional input representing at least a portion of one or more documents, the step of generating a relevance score representing the predicted relevance of the portion of one or more documents to the question; A step of determining the ranking of the conditional input according to the corresponding relevance score, and It further includes, The method according to claim 1 or 2, wherein the subset of the conditional inputs comprises n highest-ranked conditional inputs in the ranking, where n ≥ 1.

4. The step of generating the relevance score for the conditional input is: The steps include generating the embedded question data, The steps include generating an embedding for each conditional input, For each conditional input, the steps include determining the similarity between the embedding of the conditional input and the embedding of the question data. The method according to claim 3, including the method described in claim 3.

5. The method according to any one of claims 1 to 4, wherein for each network input generated from each conditioned input, the network input is further generated from one or more prompt tuples, each comprising (i) second question data representing a different second question, (ii) second answer data representing an answer to each second question, and (iii) a second conditioned input representing a document related to each second question.

6. The method according to claim 5, wherein the neural network is pre-trained on a language modeling task but not on a question answering task.

7. The step of generating answer data representing the final answer to the question from the network output representing each candidate answer is: For each candidate answer, the process involves generating a likelihood value that represents the likelihood that the candidate answer is the correct answer to the question, and The step of determining the final answer to be the candidate answer corresponding to the highest likelihood value, The method according to any one of claims 1 to 6, including the method described in any one of claims 1 to 6.

8. For each candidate answer a' to the question q corresponding to each conditional input p, the step of generating the respective likelihood value for the candidate answer a' is: A step of generating one or more initial likelihood values, wherein the one or more initial likelihood values Given the conditional input and the question, a first initial likelihood value represents an estimate for the probability p(a'|p,q) of the candidate answer, The second initial likelihood value represents an estimate for the probability p(a'|q) of the given candidate answer to the aforementioned question. The above question represents an estimate for the probability p(p|q) of the given conditional input, a third initial likelihood value, A fourth initial likelihood value representing an estimate for the probability p(q|p) of the question given the conditional input, or A fifth initial likelihood value representing an estimate for the probability p(q|a',p) of the question, given the candidate answers and the conditional inputs. A step comprising one or more of the following, A step of generating the likelihood value from the one or more initial likelihood values. The method according to claim 7, including the method described in claim 7.

9. The method of claim 8, wherein for at least one of the initial likelihood values, each representing an estimate for the probability of each posterior element given one or more prior elements, the initial likelihood value is generated using the neural network.

10. The step of using the aforementioned neural network to generate a specific initial likelihood value that represents an estimate for the probability of a particular posterior element, given one or more specific prior elements, is: A step of determining the likelihood that the neural network will produce a second network output representing a particular posterior element in response to processing a second network input generated from one or more particular pre-elements, using the set of trained parameter values ​​of the neural network. The method according to claim 9, including the method described in claim 9.

11. The method according to claim 10, wherein the second network input is further generated from one or more prompt tuples having the same specific posterior element and one or more specific prior elements as the specific initial likelihood value.

12. The aforementioned neural network comprises an autoregressive, transformer-based neural network, The method according to any one of claims 1 to 11, wherein the search engine comprises an internet search engine.

13. The aforementioned neural network is implemented on a mobile device, The step of processing the search engine query using a backend data server. It further includes, The method according to any one of claims 1 to 12, wherein the mobile device has less working memory than the backend data server and / or less computing power than the backend data server.

14. A system comprising one or more computers and one or more storage devices, wherein the one or more storage devices store instructions, and when an instruction is executed by the one or more computers, the system causes the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 13.

15. One or more computer storage media storing instructions, wherein when the instructions are executed by one or more computers, the one or more computers cause the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 13.

Images