Generating Output Sequences with Inline Evidence Using Language Model Neural Networks

Abstract

A method, system, and apparatus for generating an output sequence using a language model neural network, the method including a computer program encoded on a computer storage medium, wherein, in particular, the output sequence includes a response to an input query and in-line evidence including citations from context documents that support the response.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application No. 63 / 320,633, filed March 16, 2022, which is incorporated by reference in its entirety.

[0002] This specification relates to the use of neural networks to process inputs to generate output sequences. [Background technology]

[0003] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, either another hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Training compute-optimal large language models" by J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, LA Hendricks, J. Welbl, A. Clark, et al., arXiv preprint arXiv:2203.15556, 2022. [Non-Patent Document 2] JW Rae、S. Borgeaud、T. Cai、K. Millican、J. Hoffmann、HF Song、J. Aslanides、S. Henderson、R. In、S. Young、E. Rutherford、T. Hennigan、J. Menick、A. Cassirer、R. Powell、G. van den Driessche、LA Hendricks、M. Rauh、P. Huang、A. Glaese、J. Welbl、S. Dathathri、S. Huang、J. Uesato、J. Mellor、I. Higgins、A. Creswell、N. McAleese、A.Wu、E. Elsen、SM Jayakumar、E. Buchatskaya、D. Budden、E. Sutherland、K. Simonyan、M. Paganini、L. Cipher、L. Martens、XL Li、A. Kuncoro、A. Nematzadeh、E. Gribovskaya、D. Donato、A. Lazaridou、A. Mensch、J. Lespiau、M. Tsimpoukelli、N. Grigorev、D. Fritz、T. Sottiaux、M. Pajarskas、T. Pohlen、Z. Gong、D. Toyama、C. of Masson d'Autume、Y. Li、T. Terzi、V. Mikulik、I. Babuschkin、A. Clark、D. of the Houses、A. Guy、C. Jones、J. Bradbury、M. Johnson、BA Hechtman、L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG.Irving, “Scaling language models: Methods, analysis & insights from training gopher,” CoRR, abs / 2112.11446, 2021. [Non-Patent Document 3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," arXiv preprint arXiv:1910.10683, 2019. [Non-Patent Document 4] “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020 by Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. [Non-Patent Document 5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners," arXiv preprint arXiv:2005.14165, 2020. [Non-Patent Document 6] Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. Summary of the Invention [Means for solving the problem]

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations that uses a language model neural network to generate a response to a received request. In particular, the response generated by the system includes (i) a response to the request and (ii) "evidence" from one or more context text documents that supports the response. The evidence includes a direct quote from one of the context text documents.

[0006] For example, the system may provide an interface between a user and an information retrieval system that has access to a corpus of documents, allowing the system to leverage the information retrieval system to provide more reliable information, particularly verifiable and accurate information.

[0007] In one aspect, a method includes receiving an input text query; obtaining one or more first context text sequences and a respective natural language identifier for each of the first context text sequences; generating a first input sequence including the input text query, the one or more first context text sequences, and a respective natural language identifier for each of the one or more first context text sequences; processing the first input text sequence using an autoregressive language model neural network to generate a first output text sequence comprising: (i) a first output text subsequence that is a response to the input text query; (ii) a second output text subsequence that is one of the respective natural language identifiers for the first context text sequences; and (iii) a third output text subsequence that is text from the first context text sequence identified by the natural language identifier in the second output text subsequence; and providing at least the first output text subsequence and the third output text subsequence in response to the input text query.

[0008] In some implementations, providing at least a first output text subsequence and a first context text sequence in response to the input text query comprises providing a first output text subsequence, a second output text subsequence, and a third output text subsequence in response to the query.

[0009] In some implementations, the method further includes determining from the second output text subsequence a source of the first context text sequence identified by a natural language identifier in the second output text subsequence, and providing a reference to the source of the first context text sequence in response to a query.

[0010] In some implementations, the method includes obtaining one or more second context text sequences and a respective natural language identifier for each of the second context text sequences; generating a second input sequence including an input text query, the one or more second context text sequences, and a respective natural language identifier for each of the one or more second context text sequences; and generating (i) a fourth output text subsequence that is a response to the input text query; (ii) a fifth output text subsequence that is one of the respective natural language identifiers for the second context text sequences; and (iii) a fourth output text subsequence that is identified by the natural language identifier in the fifth output text subsequence. and a sixth output text subsequence, the sixth output text subsequence being text from the second context text sequence; generating a respective score for each output text sequence in a set including the first and second output text sequences; determining that the first output text sequence has a higher score than any output text sequence in the set; and in response to determining that the first output text sequence has the highest score, providing at least the first output text subsequence and a third output text subsequence in response to the input text query.

[0011] In some implementations, generating a respective score for each output text sequence in the set that includes the first and second output text sequences comprises scoring each of the output text sequences using the learned reward model.

[0012] In some implementations, the first output sequence includes a respective token from a vocabulary of tokens at each of a plurality of time steps, and the autoregressive neural network is configured to generate, for each time step in the first output sequence (e.g., for each of the plurality of time steps, a token corresponding to a current time step may be conveniently referred to as a “current token”), a respective score for each token in the vocabulary conditional on the first input text sequence and any token in the output sequence at any time step prior to the time step of the first output sequence (any token in the output sequence preceding the current token), and generating the first output sequence comprises, at each time step, selecting a token at the time step (the current token) using the respective scores of the tokens in the vocabulary generated by the neural network for the time step.

[0013] In some implementations, the tokens of the second output text subsequence also correspond to corresponding tokens of the (second) plurality of time steps. The step of generating the first output sequence comprises, at each time step (each of the second plurality of time steps) in the second output text subsequence after the first time step in the second output text subsequence, receiving respective scores generated by the neural network at the time step, generating a constrained score distribution in one of the natural language identifiers that assigns non-zero scores only to tokens that immediately follow a token already generated in the second output text subsequence, and sampling tokens at the time step from the constrained score distribution.

[0014] In some implementations, the second output text subsequence is preceded by one or more first predetermined syntactic tokens in the first output text sequence, and generating the first output sequence comprises determining that at a particular time step one or more first predetermined syntactic tokens were selected in one or more time steps immediately preceding the particular time step and, accordingly, determining that the particular time step is the first time step in the second output text subsequence; receiving respective scores generated by the neural network at the particular time step; in response to determining that the particular time step is the first time step in the second output text subsequence, generating a constrained score distribution that assigns a non-zero score only to tokens that are the first token in one of the natural language identifiers; and sampling tokens at the time step from the constrained score distribution.

[0015] In some implementations, the tokens of the third output text subsequence also correspond to corresponding tokens of the (third) plurality of time steps. The step of generating the first output sequence comprises, at each time step in the third output text subsequence after the first time step in the third output text subsequence (i.e., each of the third plurality of time steps), receiving respective scores generated by the neural network at the time step, generating a constrained score distribution that assigns non-zero scores only to tokens that immediately follow a token already generated in the third output text subsequence in the first context text sequence identified by the natural language identifier in the second output text subsequence, and sampling tokens at the time step from the constrained score distribution.

[0016] In some implementations, the third output text subsequence is preceded by one or more second predetermined syntactic tokens in the first output text sequence, and generating the first output sequence comprises determining that at a second particular time step, the one or more second predetermined syntactic tokens were selected at one or more time steps immediately preceding the second particular time step, and accordingly determining that the particular time step is a first time step in the third output text subsequence; receiving respective scores generated by the neural network at the particular time step; in response to determining that the particular time step is the first time step in the third output text subsequence, generating a constrained score distribution that assigns non-zero scores only to tokens that appear in a first context text sequence identified by the natural language identifier in the second output text subsequence; and sampling tokens at the time step from the constrained score distribution.

[0017] In some implementations, obtaining one or more first context text sequences and a respective natural language identifier for each of the first context text sequences comprises submitting a query derived from the input text query to a search engine, obtaining one or more context documents from the search engine in response to the query, and selecting one or more first context sequences from the one or more context documents.

[0018] In some implementations, the respective natural language identifier for each of the first context text sequences is the title of the context document from which the first context text sequence was selected.

[0019] In some implementations, the neural network is pre-trained through unsupervised learning based on the language modeling objective.

[0020] In some implementations, the neural network is fine-tuned through supervised learning, reinforcement learning, or both.

[0021] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0022] The system described herein provides a user interface to access a generative language model neural network that generates responses to received requests. In particular, generative language models (LMs) are increasingly useful for answering questions about the world. However, by default, LMs generate unsubstantiated claims, which users must choose to either blindly accept or verify for themselves.

[0023] Described herein are techniques that help users evaluate responses generated by LM by generating claims and supporting evidence. In particular, this evidence takes the form of verbatim citations extracted from longer context documents retrieved from one or more text databases. The documents may be retrieved by an Internet search engine or any other suitable information retrieval system. Thus, the system provides a user interface between the user and the information retrieval system, enhancing the reliability and verifiability of information retrieved using the information retrieval system.

[0024] To use generative techniques to ensure that citations are "verbatim," we describe a special syntax that the language model uses when citing documents, and in some cases, constrains its output to be exact citations from the retrieved documents. This ensures that the language model quotes accurately from the context documents, even if the model was previously trained for purposes that do not require citations from the input.

[0025] Furthermore, large-scale language models implemented as neural networks can produce excellent results in a variety of natural language processing tasks, including question answering. However, some of these models, particularly implementations of transformer-based models, can have more than a billion parameters and may require significant computing resources, power, and time to process the network inputs to generate the network outputs. In some cases, such models can have more than 10 or 100 billion parameters. When such models are used at scale to serve a large number of user requests, they consume large amounts of energy.

[0026] Additional considerations arise when the neural network is implemented in a digital assistant device, such as a mobile device implemented in a computing system that includes back-end components, particularly data servers, that communicate with the digital assistant device over a data communications network such as the Internet. Therefore, it is necessary to optimize the computing load between the digital assistant device and the back-end components. This need can be particularly acute for large language models, as their memory and computing requirements are significantly larger than those typically found on mobile devices.

[0027] The techniques described herein address these problems, and in some implementations, the described techniques facilitate reduced computational load and improved load balancing, especially when large language models are implemented as neural networks in multitasking and parallel processing computer systems distributed across multiple sites and interconnected by a data communications network.

[0028] In some implementations, the described techniques allow for efficient distribution of computing load between local mobile computing devices and back-end servers in a network. More specifically, in implementations, conditioning a language model neural network on a context representing documents retrieved from an internet search based on a question allows for the use of smaller language model neural networks, facilitating the implementation of neural networks on mobile devices with limited memory and computing resources.

[0029] Furthermore, using the techniques described herein, the system can leverage search engine results to generate predictions for input text using up-to-date information contained in the search engine results. Some existing systems use pre-trained neural networks without access to such search engine results to generate predictions, and because the neural networks can only encode information that was available to the neural networks during training, the predictions can be unreliable, i.e., they rely on old information and therefore may be inaccurate or at least out of date. Thus, using the techniques described herein, the system can generate more accurate and timely predictions.

[0030] Furthermore, some existing systems must repeatedly retrain their neural networks to ensure that they encode the most current information. Because the system described herein can repeatedly access new search engine results, the system does not need to retrain the neural network, thus saving significant computational resources.

[0031] Using the techniques described herein, a system can generate predictions for an input text using information encoded in multiple different documents provided by a search engine in response to processing the search engine query. The multiple different documents can each contain different information relevant to the prediction. Thus, predictions generated by the system can be more accurate than predictions generated using a single document.

[0032] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0033] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. [Brief description of the drawings]

[0034] [Figure 1] FIG. 1 illustrates an exemplary sequence generation system. [Diagram 2] FIG. 2 is a flow diagram of an example process for generating an output sequence. [Diagram 3] FIG. 4 is a flow diagram of an example process for selecting candidate output sequences. [Figure 4] FIG. 2 illustrates an exemplary user interface for presenting an output sequence to a user. [Diagram 5] FIG. 1 illustrates an example of training a language model neural network. [Figure 6] FIG. 13 illustrates an exemplary user interface for evaluating generated samples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Like reference numbers and designations in the various drawings indicate like elements.

[0036] 1 illustrates an exemplary sequence generation system 100. The sequence generation system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0037] The sequence generation system 100 serves as a user interface to an information retrieval system that accesses one or more text databases (not shown) or provides functionality to a user interface implemented on a user computer that is separate from but in communication with the sequence generation system 100. The text databases collectively form a corpus of documents. The corpus of documents may be, for example, web pages and other documents accessible through the Internet. Alternatively, the corpus of documents may be part of a proprietary text database of, for example, a scientific publisher or other organization. The sequence generation system 100 processes an input text query 102 from a user using the context sequence generation system 104, the input sequence generation system 110, and the language model neural network 114 to generate an output sequence 116.

[0038] The input text query 102 may be a query submitted to the system 100 by a user through a user computer, a question submitted to the system 100 through a user computer, or another request requiring a response from the system 100. In some cases, the system receives the query as text from the user computer. In other cases, the system receives a natural language voice query from the user and converts the voice into the input text query 102 by applying a voice recognition engine to the voice. The input text query 102 is received in the form of a sound (audio) signal captured by a microphone of the user computer and converted by the voice recognition engine, i.e., a voice-to-text converter, to form the input text query 102. Alternatively, it may be typed in using a data entry device of the user computer.

[0039] When the system 100 receives an input text query 102 , the context sequence generation system 104 obtains one or more first context text sequences 106 and a respective natural language identifier 108 for each of the first context text sequences 106 .

[0040] For example, each context text sequence 106 may be extracted from a respective context document and the identifier 108 may be the title of the context document. As another example, some or all of the context text sequences 106 may be extracted from the same context document and the identifier 108 may be a section header or other identifier of the portion of the document from which the context text sequences are extracted.

[0041] Obtaining the context sequence is described in more detail below with reference to FIG.

[0042] The input sequence generation system 110 then generates a first input sequence 112 that includes the input text query 102, one or more first context text sequences 106, and a respective natural language identifier 108 for each of the one or more first context text sequences.

[0043] For example, the first input sequence 112 may include the query 102, the context text sequence 106, and the identifier 108 arranged according to a predefined input syntax. In some cases, the first input sequence 112 may also include other text, such as one or more natural language "prompts," one or more separator tokens that separate various elements of the input sequence, or both. The natural language prompts are examples of input and output pairs, where the inputs are examples of inputs that may be provided and the outputs are examples of outputs that will be generated. Prompts are described in more detail below.

[0044] The sequence generation system 100 then processes the first input sequence 112 using the autoregressive language model neural network 114 to generate a first output text sequence 116 .

[0045] The output sequence 116 includes (i) a first output text subsequence that is a response to the input text query 102, (ii) a second output text subsequence that is one of the respective natural language identifiers 108 for the first context text sequence 106, and (iii) a third output text subsequence that is text from the first context text sequence identified by the natural language identifier in the second output text subsequence.

[0046] In particular, (i), (ii), and (iii) are arranged in the output sequence according to a predefined output syntax, one example of which is described in more detail below with reference to FIG.

[0047] The sequence generation system 100 then provides at least a first output text subsequence and a third output text subsequence in response to the input text query 102. Thus, the system 100 provides a text response to the input text query 102 and text from one of the context text sequences 106 as supporting evidence for the text response.

[0048] In some implementations, the sequence generation system 100 generates multiple candidate output sequences 116 in response to the input query 102 .

[0049] In these implementations, the system 100 also generates a respective score for each candidate output sequence and provides, in response to a user query, only text from the candidate output sequences with the highest scores.

[0050] In some of these implementations, if none of the candidates have a score above the threshold, the system 100 instead outputs a default text response to the user's query, such as "I don't know" or "I am not sure."

[0051] Scoring of candidate output sequences is described below with reference to FIG.

[0052] The language model neural network 114 may be any suitable language model neural network that receives an input sequence consisting of text tokens selected from a vocabulary and autoregressively generates an output sequence consisting of text tokens from the vocabulary. For example, the language model neural network 114 may be a transformer-based language model neural network or a recurrent neural network-based language model.

[0053] The tokens in the vocabulary may be any suitable text tokens that represent text elements in one or more natural languages, such as words, parts of words, punctuation marks, and optionally numbers or other text symbols contained in a corpus of text. In general, the input text query 102, the natural language identifier 108, and / or the context text sequence 106 are also sequences of tokens selected from the vocabulary.

[0054] The language model neural network 114 is referred to as an autoregressive neural network because the neural network 114 autoregressively generates the output sequence of tokens by generating each particular token in the output sequence conditional on the current input sequence, which includes any tokens that precede the particular text token in the output sequence, i.e., tokens already generated for any previous positions preceding the particular token's position in the output sequence, and context inputs that provide the context for the output sequence.

[0055] For example, the current input sequence when generating a token at any given position in the output sequence may include the input sequence and a token of the output sequence at any preceding position preceding the given position in the output sequence. As a specific example, the current input sequence may include the input sequence followed by a token at any preceding position preceding the given position in the output sequence. Optionally, in the current input sequence, tokens from the input sequence and the output sequence may be separated by one or more predefined tokens, i.e., a specified set of one or more tokens from the vocabulary in the current input sequence. That is, there may be one or more predefined tokens between the tokens from the input sequence and the output sequence.

[0056] More specifically, to generate a particular token at a particular position in the output sequence, the neural network 114 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in a vocabulary of tokens. The neural network 114 may then use the score distribution to select a token from the vocabulary as the particular token. For example, the neural network 114 may greedily select the highest scoring token or may sample the token from the distribution, e.g., using kernel sampling or another sampling technique.

[0057] As a specific example, the language model neural network 114 may be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation, and (ii) an output sub-network that processes the output of the last attention block to generate a score distribution.

[0058] Lieut The snake squirrel of the squid The snowflakes are worth it These are the results of J.S. Hoffmann, S. Borgeaud、A. Mensch、E. Buchatskaya、T. Cai、E. D. Rutherford. d. L. Houses LA Hendricks. Welbl、A. Clark, “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022, JW Rae, and S. Clark. Borgeaud、T. Cai、K. Millican, J. (1999). Hoffmann, HF Song, J. Aslanides、S. R. Henderson. Ring、S. Young、E. Rutherford;T. Hennigan、J. Menick、A. Cassirer、R. Powell、G. of the Driessche、LA Hendricks、M. Rauh、P. Huang、A. Glaese、J. Welbl、S. Dathathri、S. Huang、J. Uesato、J. Mellor、I. Higgins、A. Creswell、N. McAleese、A.Wu、E. Elsen, SM Jayakumar, E.K. Buchatskaya、D. Budden、E. Sutherland、K. Simonyan、M. Paganini、L. Sifre、L. Martens, XL Li, A. Kuncoro、A. Nematzadeh、E. Gribovskaya、D. Donato、A. Lazaridou、A. Mensch, J. Chem. Lespiau、M. Tsimpoukelli、N. Grigorev、D. Fritz、T. Sottiaux、Mr. Pajarskas、T. Pohlen、Z. Gong、D. Toyama、C. of Masson of Autumn、Y. Li、T. Terzi、V. Mikulik、I. Babuschkin, A. Clark、D. of The Houses、A. Guy、C. Jones, J.S.Bradbury、M. Johnson, BA Hechtman, L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG. Irving, “Scaling Language Models: Methods, Analysis, and Insights from Training Gopher” (CoRR, abs / 2112.11446, 2021), Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, and Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu transformer arXiv arXiv:1910.10683 arXiv:1910.10683 2019 Daniel Adiwardana Minh-Thang Luong David R. So Jamie Hall Noah Fiedel Roma Thoppilan Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le, and Towards a Human-Like Open-Domain chatbot CoRR, abs / 2001.09977, 2020, Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam、Girish Sastry、Amanda Askellらによる「Language models are few-shot learners can be downloaded from arXiv arXiv:2005.14165.

[0059] In general, however, a Transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then at least partially updates each of the hidden states by applying self-attention to generate a respective output hidden state for each of the input tokens. The input hidden state of the first attention block is an embedding of the input token in the input sequence, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0060] In this example, the output sub-network processes the output hidden state produced by the last attention block in the sequence of the last input token in the input sequence to generate a score distribution.

[0061] In general, because the neural network 114 is autoregressive, the system 100 can use the same neural network 114 to generate multiple different candidate output sequences in response to the same request, for example, by using beam search decoding from the score distribution generated by the neural network 114, by using a sample-and-rank decoding strategy, by using different random seeds for the pseudo-random number generator used in sampling different runs through the neural network 114, or by using another decoding strategy that exploits the autoregressive properties of the neural network 114.

[0062] In some implementations, the language model 114 is pre-trained, i.e., trained in language modeling tasks that do not require providing evidence in response to a user question, and the system 100 causes the neural network 114 to generate output sequences according to a predetermined syntax through natural language prompts in the input sequence.

[0063] For example, system 100 or another training system pre-trains language model neural network 114 for a language modeling task, e.g., a task in which, given a current sequence of text tokens, one must predict the next token that follows the current sequence in training data. As a specific example, language model neural network 114 can be pre-trained based on a maximum likelihood objective on a large dataset of text, e.g., publicly available text from the Internet or another text corpus.

[0064] In some other implementations, after pre-training, the system 100 fine-tunes the language model 114, for example through supervised learning, reinforcement learning, or both, based on the objective of generating output sequences according to syntax, as described in more detail below with reference to FIG.

[0065] In some of these implementations, the system 100 still includes one or more natural language prompts in the input to the language model 114 at inference time, i.e., after training.

[0066] As described above, the natural language prompts are examples of input and output pairs, where the inputs are examples of inputs that can be provided and the outputs are examples of outputs that will be generated. Thus, each prompt includes an example input sequence that is an example query, an example set of one or more context sequences, and respective identifiers of the one or more context sequences arranged according to a predefined input syntax. Each prompt also includes an example first output text subsequence that is a response to the example query, an example second output text subsequence that is one of the respective natural language identifiers for one of the example context text sequences, and an example third output text subsequence that is text from the example context text sequence identified by the natural language identifier in the example second output text subsequence, arranged according to an output syntax. Optionally, the input sequence may also include one or more tokens from a vocabulary that delimit each prompt and one or more tokens that delimit the final prompt and the user query.

[0067] Furthermore, in some implementations, the system 100 performs "constrained sampling" when selecting tokens to include in the output sequence, ensuring that the output of the neural network 114 is syntactic and that the sequence is internally consistent, i.e., the evidence is a direct quote from the context text sequence 106 identified by the natural language identifier 108 in the output sequence.

[0068] In cases where system 100 generates multiple candidate output sequences, constrained sampling eliminates the need for the system to score invalid or inconsistent output sequences, significantly reducing the number of candidates that need to be generated to ensure high quality output, and significantly improving the computational efficiency of system 100, i.e., reducing the amount of computational resources consumed by system 100.

[0069] An example of constrained sampling is described in more detail below with reference to FIG.

[0070] 2 is a flow diagram of an exemplary process 200 for generating an output sequence given an input query. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a sequence generation system suitably programmed in accordance with this specification, such as sequence generation system 100 shown in FIG. 1, can perform process 200.

[0071] The system receives an input text query from a user, for example, using a user interface (step 202).

[0072] The system obtains one or more first context text sequences and a respective natural language identifier for each of the first context text sequences (step 204).

[0073] For example, the system may obtain one or more context sequences and a respective natural language identifier for each of the first context sequences by submitting a search query derived from the input text query to a search engine. The search engine is configured to access a corpus of documents and search the corpus of documents based on the search query. For example, the search query may be the same text as the input text query and may be modified by the system to add synonyms or correct typos or spelling errors.

[0074] The system can then retrieve one or more documents from the search engine in response to the search query. The one or more documents can be ranked by the search engine according to, for example, their quality and relevance to the received search query.

[0075] The system may then select one or more first context sequences from the one or more context documents, for example by selecting one or more highest ranked search results. The system also associates a respective natural language identifier with each first context sequence.

[0076] In some implementations, the search engine also provides snippets from the corresponding context documents as part of the search results that identify the corresponding context documents. In some of these implementations, the system can generate a context sequence for a given document by extracting snippets and text surrounding the snippets from the corresponding context documents. For example, the system can use snippets to extract snippet text to account for the fact that documents vary in length and often exceed the maximum context window size of a language model (described below).

[0077] Thus, especially for few-shot prompts when presenting multiple documents at once, the system may need to limit the number of tokens spent on document content in a given input sequence. Thus, the system may truncate documents by using snippets as described above. For example, the system may use snippets to truncate a given document into a maximum token-length fragment that contains relevant search snippets.

[0078] In some implementations, the system can ensure that a truncated fragment begins at the beginning of a sentence or paragraph.

[0079] As a specific example, when a train is running, the system can randomly select such a starting position to increase the variety of inputs. During inference, the system can allow a maximum number of characters before the start of a snippet fragment, e.g., 250, 500, or 1000 characters, identify the first sentence that begins within that range, and use that first sentence as the beginning of the truncated fragment.

[0080] The search engine may be any suitable search engine accessible by the system that searches any suitable corpus of documents, such as web pages, books, or other documents. For example, the search engine may be an Internet search engine that searches the Internet and returns results that reference documents available on the Internet. As another example, the search engine may be another search engine that searches a private corpus of documents, such as documents available on an internal network or stored in a collection of one or more databases.

[0081] For example, the respective natural language identifier for each of the first context text sequences may be the title of the context document from which the first context text sequence is selected.

[0082] The system generates a first input sequence (step 206) including an input text query, one or more first context text sequences, and a respective natural language identifier for each of the one or more first context text sequences.

[0083] The system processes the first input text sequence using an autoregressive language model neural network to generate a first output text sequence (step 208).

[0084] The first output text sequence includes a first output text subsequence that is a response to the input text query, a second output text subsequence that is one of the respective natural language identifiers for the first context text sequence, and a third output text subsequence that is text from the first context text sequence identified by the natural language identifier in the second output text subsequence.

[0085] The system provides (eg, to a user) at least a first output text subsequence and a third output text subsequence in response to an input text query (step 210).

[0086] The system may provide a first output text subsequence, a third output text subsequence, and optionally a second output text subsequence in response to a query.

[0087] Further, in some implementations, the system can determine from the second output text subsequence a source of the first contextual text sequence identified by a natural language identifier in the second output text subsequence and provide a reference to the source of the first contextual text sequence in response to the query. For example, the system can provide the reference as a hyperlink linking to the source of the first contextual text sequence, e.g., a web page.

[0088] An exemplary presentation of an output sequence generated by the system is described below with reference to FIG.

[0089] As described above, in some implementations, the system generates a set of multiple candidate output sequences (including the first output text sequence) and a respective score for each candidate output sequence, and provides only the first output sequence in response to a determination that the first output text sequence has the highest score among any of the candidate output sequences.

[0090] For example, when processing a first input text sequence, the system may generate at least some of the candidate output sequences in the set by sampling different candidate output sequences from outputs generated by the language model neural network.

[0091] Furthermore, in some implementations, the system can generate more context sequences than fit into the "context window" of the language model neural network. That is, the language model neural network may only be able to process input sequences that do not exceed a maximum number of characters, e.g., due to memory constraints or due to the framework in which the neural network was trained. In some implementations, including the natural language identifier and tokens of all context sequences may exceed this maximum. In these implementations, the system generates multiple different input sequences, each of which includes a respective subset of the context sequences.

[0092] In other words, the system can also obtain one or more second context text sequences in addition to the first context text sequence and a respective natural language identifier for each of the second context text sequences, and generate a second input sequence including the input text query, the one or more second context text sequences, and a respective natural language identifier for each of the one or more second context text sequences. The system can then process the second input text sequence using an autoregressive language model neural network to generate a second output text sequence comprising: (i) a fourth output text subsequence that is a response to the input text query, (ii) a fifth output text subsequence that is one of the respective natural language identifiers for the second context text sequences, and (iii) a sixth output text subsequence that is text from the second context text sequence identified by the natural language identifier in the fifth output text subsequence.

[0093] The system then generates a respective score for each candidate output text sequence in a set, e.g., a set including the first and second output text sequences, and determines that the first output text sequence has a higher score than any output text sequence in the set. In some cases, this can be done by using the learned reward model to score each of the output text sequences. The use of the learned reward model to score the candidate output sequences is described below with reference to FIG. 3.

[0094] The system may then provide at least a first output text subsequence and a third output text subsequence in response to the input text query in response to determining that the first output text sequence has the highest score.

[0095] FIG. 3 illustrates an example of the operation of a sequence generation system where the system generates multiple candidate output sequences in response to a given text query.

[0096] As shown in the example of FIG. 3, the system 100 receives a question 302, for example, from a user computer.

[0097] The system 100 performs an Internet search 304 to identify the top k documents most relevant to the question 302. Typically, k is an integer greater than 1, such as 5, 10, 20, or 100. For example, the system 100 may submit the question 302 or a query derived from the question 302 to an Internet search engine and obtain search results from the Internet search engine that identify the top k documents.

[0098] The system then uses a generator 306 to generate one or more input sequences to the language model neural network 114 and sample 308 N candidate output sequences using the language model neural network 114. In some implementations, the number N of candidate output sequences is greater than the number k of documents.

[0099] For example, the generator 306 may generate a single input sequence that includes context from all k documents and then process the single input sequence multiple times using the language model neural network 114 to sample N candidate output sequences.

[0100] As another example, the generator 306 may generate multiple input sequences, each including context from a respective subset of the k documents, and then process each of the multiple input sequences multiple times using the language model neural network 114 to sample N candidate output sequences.

[0101] As another example, the generator 306 can generate multiple input sequences, each including context from each of the k documents, and then process each of the multiple input sequences using the language model neural network 114.

[0102] In any of the above examples, multiple input sequences may be sampled in round robin order until N candidate output sequences have been sampled.

[0103] In some implementations, N may be a multiple of k. In other implementations, N may not be divisible by k.

[0104] The system 100 then performs reward model scoring 310 on each of the N candidate output sequences.

[0105] That is, the system 100 uses the learned reward model to assign a respective score to each of the N candidate output sequences.

[0106] The learned reward model 310 is a model, e.g., another language model neural network, that receives as input the input text query and the responses and citations generated by the neural network 114, and produces as output a score that represents the quality of the response and citation. For example, the score may represent the likelihood that a user will like the response (and accompanying citation) compared to other responses (and accompanying citations) to the same query generated by the neural network 114.

[0107] Training the reward model is described below with reference to FIG.

[0108] The system 100 then selects as the final output sequence the "best" example 312, ie, the candidate output sequence that has the highest score from the N sequences according to the learned reward model.

[0109] In some implementations, if none of the candidates have a score above the threshold, the system 100 instead outputs a default text response to the user's query, such as "I don't know" or "I'm not sure."

[0110] As described above, each candidate output sequence includes (i) a first output text subsequence that is a response to the input text query, (ii) a second output text subsequence that is one of the respective natural language identifiers for the first context text sequence, and (iii) a third output text subsequence that is text from the first context text sequence identified by the natural language identifier in the second output text subsequence.

[0111] In particular, (i), (ii), and (iii) are arranged in the output sequence according to a predefined output syntax.

[0112] As shown in the example in Figure 3, the output syntax is %<claim>%(document title)%[quote from document]% where "%<" ">%(" ")%["and"]%" are template tokens, i.e., predefined syntactic tokens that are inserted before and after the subsequences, "claim" is a placeholder for the first output text subsequence, "document title" is a placeholder for the second output text subsequence, and "quote from document" is a placeholder for the third output text subsequence.

[0113] However, any of a variety of syntaxes can be used to place the "claim", "document title", and "quote from document" placeholders at predetermined locations in the output sequence.

[0114] In some implementations, and as described above, the system samples each of the N candidates using constrained sampling to ensure that each candidate is syntactic, i.e., contains an exact quotation from the context sequence identified by the natural language identifier in the sequence.

[0115] That is, as described above, the generator 306 samples a given candidate output sequence by generating, for each time step in the output sequence, a respective score for each token in the vocabulary conditional on the first input text sequence and any token in the output sequence at any time step prior to the time step of the first output sequence, and at each time step, selects a token at the time step using the respective scores of the tokens in the vocabulary generated by the neural network for the time step.

[0116] When constrained sampling is employed, the system restricts sampling to sampling only those tokens that are valid next tokens according to the output sequence.

[0117] For example, when generating the second output text subsequence, and at each time step in the second output text subsequence after a first time step in the second output text subsequence, the generator 306 can receive the respective scores generated by the neural network at the time step, generate a constrained score distribution in one (or more) of the natural language identifiers that assigns a non-zero score only to tokens that immediately follow a token already generated in the second output text subsequence, and then sample tokens at the time step from the constrained score distribution instead of the received score distribution. That is, the system restricts sampling to assign non-zero scores only to tokens that, when added to a token already selected for the second output text subsequence, generate a valid prefix of one or more of the natural language identifiers in the corresponding input sequence.

[0118] As another example, in some cases, the second output text subsequence is preceded by one or more first predetermined syntactic tokens in the first output text sequence. For example, in the example of FIG. 3, the second output text subsequence is preceded in the output syntax by the token ">%(".

[0119] In these cases, generating the output sequence using constrained sampling includes determining that one or more first predetermined syntactic tokens at a particular time step were selected in one or more time steps immediately preceding the particular time step, and, accordingly, determining that the particular time step is the first time step in the second output text subsequence. For example, the system may determine that the token ">%(" has already been sampled, and, accordingly, determine that the next time step is the first time step in the second subsequence.

[0120] In this example, the system may receive respective scores generated by the neural network at a particular time step, and in response to determining that the particular time step is a first time step in the second output text subsequence, generate a constrained score distribution that assigns a non-zero score only to a first token in one of the natural language identifiers in the corresponding input sequence, and sample tokens at the time step from the constrained score distribution, i.e., the system restricts sampling to assign non-zero scores only to tokens that are the first token in one or more of the natural language identifiers in the corresponding input sequence.

[0121] As another example, when using constrained sampling, at each time step in the third output text subsequence after the first time step in the third output text subsequence, the system can receive the respective scores generated by the neural network at the time step and generate a constrained score distribution that assigns a non-zero score only to a token that immediately follows a token already generated in the third output text subsequence in the first context text sequence identified by the natural language identifier in the second output text subsequence. The system then samples tokens at the time step from the constrained score distribution. That is, the system restricts the sampling to assign a non-zero score only to a token that, when added to a token already selected for the third output text subsequence, directly matches a subsequence in the first context text sequence identified by the natural language identifier in the second output text subsequence. Thus, the system ensures that the third output text subsequence is a direct quotation from the context document identified by the natural language identifier in the second output text subsequence.

[0122] As another example, in some cases, the third output text subsequence is preceded by one or more second predefined syntax tokens in the first output text sequence, e.g., in the example of FIG. 3, the second output text subsequence is preceded in the output syntax by the token ")%[".

[0123] In these cases, when using constrained sampling, the system determines that at the second particular time step, one or more second predetermined syntactic tokens were selected at one or more time steps immediately preceding the second particular time step, and accordingly determines that the particular time step is the first time step in the third output text subsequence. Then, upon receiving the respective scores generated by the neural network at the particular time step, the system generates a constrained score distribution that assigns non-zero scores only to tokens that appear in the first context text sequence identified by the natural language identifier in the second output text subsequence, and samples tokens at the time step from the constrained score distribution.

[0124] The system 100 then provides at least a portion of the text from the best sample 312 for presentation to a user. For example, the system 100 may render 314 a presentation of the best sample 312 in a user interface.

[0125] As shown in FIG. 3, the presentation may include the text of the “claim”, i.e., the text of the first subsequence, citations from context documents that support the “claim”, i.e., the text of the third subsequence, and, optionally, a document identifier from the second subsequence.

[0126] FIG. 4 illustrates an exemplary user interface 400 for presenting an output sequence to a user.

[0127] In the example of FIG. 4, a user submits query 402, "What kind of animal is Scooby-Doo?"

[0128] In response, system 100 generated an output sequence that included three subsequences: (i) "Great Dane dog", (ii) "Wikipedia page: Scooby-Doo", and (iii) a quote from the Wikipedia page titled "Scooby-Doo".

[0129] In response to the user query 402 , the system then presents a first subsequence 404 , a second subsequence 406 , and a third subsequence 408 in the user interface 400 .

[0130] Additionally, the system has displayed the first subsequence 404 as a hyperlink that links to the source of the third subsequence 408, i.e., to Scooby-Doo's Wikipedia page, i.e., a web page titled "Wikipedia Page: Scooby-Doo." Including a hyperlink in the user interface 400 allows a user to navigate to the source indicated by the second subsequence, for example, to verify the accuracy of a citation or to obtain additional context regarding the response.

[0131] FIG. 5 shows an example of training the language model neural network 114.

[0132] As shown in FIG. 5, the system obtains a pre-trained language model (502).

[0133] For example, a language model may have been trained for the purposes of language modeling on a large corpus of text documents, as described above.

[0134] After obtaining a pre-trained language model (502), the system generates samples and evaluates the generated samples via human evaluation (504).

[0135] For example, to obtain each rating, the system may present the rater user with a question and two candidate answers, e.g., two samples generated using a pre-trained language model with several prompts. Each candidate answer may be divided into a "claim" section and a "supporting evidence" section, e.g., as shown above with reference to FIG. 4.

[0136] The system can then obtain input from the rater user specifying which assertions are plausible responses to the question, whether the assertions are supported by attached cited evidence, and which answers are preferred by the rater user. A plausible response to a question is one that is on-topic to the question. A supported response is one where the evidence provided is sufficient to verify the validity of the response.

[0137] An example of a user interface that can be used to obtain input from a user is shown in FIG.

[0138] That is, FIG. 6 illustrates an exemplary user interface 600 for evaluating generated samples that may, for example, receive input for human evaluation of the generated samples.

[0139] 6, a user is presented with a query 602 and two candidate responses 604 and 606 to the query 602. Each candidate response 604 and 606 includes a response to the query, supporting evidence from the response, and an identifier for the source of the supporting evidence.

[0140] For each candidate response 604 and 606, the user interface presents a corresponding selection element 608 and 610 that allows the user to submit input indicating whether the corresponding candidate response is a plausible answer (or indicates that the user is unsure) and whether the corresponding candidate response is supported by corresponding supporting evidence (or indicates that the user is unsure).

[0141] Selection elements 608 and 610 also allow a user to submit input indicating that the corresponding candidate response 604 or 606, respectively, is a preferred response (of the two candidate responses) to query 602.

[0142] The user interface 600 also allows a user to submit an input indicating that two responses are "tied" or to submit a comment about the sample.

[0143] Returning to the description of FIG. 5, the system then uses the evaluated samples to perform supervised fine-tuning (SFT) 506, in which the system trains a language model on the evaluated samples through supervised learning.

[0144] That is, for each sample used in SFT, the system trains a language model to generate the claims and supporting evidence in the sample, given a set of context sequences, including the question in the sample and a context sequence with supporting evidence text.

[0145] Optionally, when performing SFT, the system can use only samples that are deemed valid and are backed by supervised fine-tuning.

[0146] As a specific example, the system may generate an input sequence of a given sample during an SFT as follows:

[0147] For a certain percentage of samples, say 1 / 3 or 1 / 2 of the data, the system uses only a single document in the context, and this document is the same as the document from which the supporting evidence was extracted, enforcing the presence of supporting evidence in the context sequence.

[0148] For the remaining samplers, the system uses n documents in the context, where n is, for example, randomly drawn between 1 and a fixed number, for example, 5, 10, or 15. Similarly, the system enforces that the target document and its supporting evidence citations are present in the context sequence. For the remaining documents in the context sequence, the system can use, for example, the top n-1 search results for the query provided by a search engine.

[0149] The system can truncate each of the context documents such that the total token length of the input sequence does not exceed a fixed number based on the context window of the language model. This token length tolerance can be randomly divided among the documents in the prompt so that the language model recognizes different sized context sequences from different context documents in the same input sequence. In truncating a given context document to the maximum allowed length, the system can ensure that each document contains a snippet as described above.

[0150] Optionally, after performing supervised fine-tuning (SFT) 506, the system can use the SFT model to generate additional samples that are again evaluated via human evaluation.

[0151] The system then trains a reward model (RM) 508 on the generated samples, e.g., the originally generated samples or the originally generated samples and additional samples generated using the SFT model.

[0152] As described above, the trained reward model is a model, e.g., another language model neural network, that receives as inputs the input text query and the responses and citations generated by the neural network 114 and produces as output scores representing the quality of the responses and citations.

[0153] For example, the system can train the reward model as a classifier that predicts a binary variable indicating which example in a given pair was preferred given a query and response string. That is, the system can calculate the probability that the first example in the pair was preferred given the scores generated by the reward model for both examples in the pair. For example, the system can train the reward model using a cross-entropy objective, using the user's preferences as the truth values ​​and the calculated probabilities as the predicted values.

[0154] Optionally, during training, the reward model also predicts the binary supported plausible judgment of the response in the pair as an auxiliary loss. Thus, in these cases, the final loss is, for example, a combination of the average or weighted average of the pairwise preference prediction loss and the auxiliary prediction loss.

[0155] In some implementations, the system can add a set of created ("synthetic") comparisons to the RM training set. For example, the system can generate created comparisons from corroborated and refuted claims of a fact-checking dataset. An example of such a dataset is the FEVER dataset (Thorne et al., 2018). Including these created comparisons provides additional out-of-distribution question answering modes that are not extracted, allowing the reward model to better verify the evidence support. An example of such a dataset, for example the FEVER dataset, can include claims generated by modifying sentences extracted from offensive text. These claims are then classified as supported, refuted, or insufficient and marked with associated evidence. To convert such claims into example questions that compare answers, the system can use any of a variety of techniques. Some examples of the types of techniques are now described.

[0156] Type A: The system may generate questions by direct template action from the assertion (e.g., "{assertion}?", "Is {assertion} true?", "Is it correct to say {assertion}?", "Do you agree with {assertion}?"). An example would be comparing positive answers such as "Yes", "This is correct", or "That is true" paired with supporting citations to negative answers paired with the same citations. If the original assertion is supported, the positive answer is marked as preferred, supported, and plausible. If not, the negative answer is marked as preferred, supported, and plausible.

[0157] Type B: The system can convert claims into questions using a pre-trained language model neural network with a few prompts. For example, the claim "Roman Atwood is a content creator" can be converted into "Who is Roman Atwood?". As a comparison for the claims converted into questions, the system can use one answer as the corresponding claim (with supporting citations) from the dataset and a direct negation of the claim generated via the template (e.g., "{claim} is not true") as the other answer. If the original claim is supported, the answer containing the claim is marked as preferred, supported, and valid. If not, the negated claim is marked as preferred. As another example, if the original claim is supported, the system can use the original claim as one answer and a randomly generated claim as a comparison, marking the original claim as preferred, supported, and valid.

[0158] As described above, the system can then use the reward model at sampling time to assign scores to candidate output sequences.

[0159] After training the RM 508, the system can use the trained reward model to further fine-tune the SFT model through reinforcement learning 510. That is, the system uses the reward model to perform a reinforcement learning from human preferences (RLfHP) technique by training the model to maximize the expected reward predicted by the trained RM 508.

[0160] Optionally, the system can then use the further fine-tuned model to generate additional samples for human evaluation, and to re-fine-tune the model through SFT and / or RL to retrain the RM, or both, i.e., the system can repeat the described training loop multiple times to further fine-tune the language model, to further fine-tune the reward model, or both.

[0161] Furthermore, while the example in Figure 5 illustrates that the system uses both SFT and RL to fine-tune the language model, in some cases the system uses only SFT or RL rather than both. For example, when using a reward model for re-ranking, performance may be improved by using a model fine-tuned with only SFT or RL (rather than both) so that the reward model is provided for re-ranking a more diverse set of samples.

[0162] An explanation of the self-attention that can be employed by a language model neural network is as follows.

[0163] The self-attention block mentioned above is a neural network layer that includes an attention mechanism that operates on the self-attention block inputs (or inputs derived from the layer inputs) to generate the self-attention block outputs. The self-attention mechanism may be causally masked, such that any given position in the input sequence does not pay attention to (e.g., do not use data from) any positions after the given position in the input sequence. There are various possible types of attention mechanisms.Examples of self-attention layers that include attention mechanisms are "Attention is all you need" by Vaswani et al., 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; "Exploring the limits of transfer learning with a unified text-to-text transformer" by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, arXiv preprint arXiv:1910.10683, 2019; and "Towards a human-like open-domain neural network" by Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. "Language models are few-shot learners," CoRR, abs / 2001.09977, 2020; and "Language models are few-shot learners," by Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., arXiv preprint arXiv:2005.14165, 2020.

[0164] In general, attention mechanisms map a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, and the weight assigned to each value is computed by a compatibility function, e.g., the dot product or scaled dot product of the query and the corresponding key.

[0165] In general, the self-attention mechanism is configured to correlate different positions in the same sequence to determine a transformed version of the sequence as an output. For example, the input of the attention layer may comprise an element-wise vector of the input sequence. These vectors provide inputs to the self-attention mechanism and are used by the self-attention mechanism to determine a new representation of the same sequence of the attention layer output, also comprising an element-wise vector of the input sequence. The output of the self-attention mechanism may be used as the output of the attention layer or may be processed by one or more of a feedforward layer, a skip connection, or a normalization operation to provide the output of the attention layer.

[0166] In some implementations, the attention mechanism calculates a query matrix Q = XW that contains a respective query for each vector in the input sequence. Q , a key matrix K=XW that contains each key for each vector in the input sequence. K , and a value matrix V=XW that contains the respective values ​​for each vector in the input sequence. V To derive, for example, the matrix W Q A query transformation defined by, say, the matrix W K and the key transformation defined by, for example, the matrix W V to the inputs of the attention layer, which are the input data X to the inputs of the attention layer, and these are used to determine the output attention sequence. For example, the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine a respective weight for each value vector, and then combining the value vectors using the respective weights to determine the self-attention layer output for each element of the input sequence. The output of the self-attention layer may be scaled by a scaling factor, e.g., the square root of the dimensions of the query and key, to implement scaled dot product attention. Thus, for example, the output of the attention mechanism may be

[0167]

number

[0168] where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism comprises an "additive attention" mechanism that computes the compatibility function using a feedforward network with hidden layers. The output of the attention mechanism may be further processed by one or more fully connected feedforward neural network layers.

[0169] The attention mechanism may implement multi-headed attention, i.e., apply multiple different attention mechanisms in parallel. These outputs may then be combined, e.g., concatenated, with a learned linear transformation applied to reduce to the original dimensions if necessary.

[0170] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0171] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0172] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. The apparatus may also be or include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates code that establishes the execution environment for a computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0173] A computer program may be called or described as a program, software, software application, app, module, software module, script, or code, may be written in any type of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or in several coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer, or on several computers at one site or distributed across several sites and interconnected by a data communications network.

[0174] The term "database" is used broadly herein to refer to any collection of data, which need not be structured in any particular way, or even structured at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0175] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, and in other cases, multiple engines may be installed and executed on the same computer.

[0176] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0177] A computer suitable for the execution of a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, or a random access memory, or both. The essential elements of a computer are a central processing unit for executing or carrying out instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from, transfer data to, or both of these storage devices. However, a computer need not have such devices. Additionally, a computer may be incorporated in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0178] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0179] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices for providing user interaction can also be used, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Also, input from the user can be received in any form, such as acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. Also, the computer can interact with the user, for example, by sending a text message or other form of message to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0180] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling common and computationally intensive parts of machine learning training or generation, e.g., inference, workloads.

[0181] Machine learning models can be implemented and deployed using machine learning frameworks such as the TensorFlow framework or the Jax framework.

[0182] An embodiment of the subject matter described herein may be implemented in a computing system that includes back-end components, such as, for example, a data server, or includes middleware components, such as, for example, an application server, or includes front-end components, such as, for example, a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0183] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to and receiving user input from a user interacting with the device functioning as a client. Data generated at the user device, e.g., a result of a user interaction, may be received at the server from the device.

[0184] Although the present specification contains many specific implementation details, these should not be construed as limiting the scope of the invention or the scope of the claims, but as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in the context of individual embodiments herein can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, even if features are described above as functioning in a particular combination and originally claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0185] Similarly, although operations are depicted in the figures and claimed in a particular order, it should not be understood that such operations need to be performed in the particular order or sequential order shown, or that all of the operations shown need to be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0186] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily have to follow the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0187] 100 Sequence Generation System 102 Input text query 104 Context Sequence Generation System 106 Contextual Text Sequences 108 Natural Language Identifiers 108 Identifier 110 Input sequence generation system 112 First Input Sequence 114 Language Model Neural Networks 114 Autoregressive Language Model Neural Networks 116 Output Sequence 116 First output text sequence 200 processes 302 Questions 304 Internet Search 306 Generator 308 Sampling 310 Reward Model Scoring 312 Best Samples 314 Rendering 400 User Interface 402 Query 404 First Subsequence 406 Second Subsequence 408 Third Subsequence 506 Supervised Fine Tuning (SFT) 508 Reward Model (RM) 600 User Interface 602 queries 604 Candidate Response 606 Candidate Response 608 Selection elements 610 Selection elements

Claims

1. A method implemented by one or more computers, comprising: receiving an input text query; obtaining one or more first context text sequences and a respective natural language identifier for each of said first context text sequences; generating a first input sequence comprising the input text query, the one or more first context text sequences, and the respective natural language identifier for each of the one or more first context text sequences; (i) a first output text subsequence that is a response to the input text query; (ii) a second output text subsequence, the second output text subsequence being one of the respective natural language identifiers for the first context text sequence; and (iii) a third output text subsequence that is text from the first context text sequence identified by the natural language identifier in the second output text subsequence; and processing the first input text sequence using an autoregressive language model neural network to generate a first output text sequence comprising: providing at least the first output text subsequence and the third output text subsequence in response to the input text query; A method comprising:

2. providing at least the first output text subsequence and the first context text sequence in response to the input text query, comprising: The method of claim 1 , comprising providing the first output text subsequence, the second output text subsequence, and the third output text subsequence in response to the query.

3. determining from the second output text subsequence a source of the first context text sequence identified by the natural language identifier in the second output text subsequence; providing a reference to the source of the first context text sequence in response to the query; The method of claim 1 or 2, further comprising:

4. obtaining one or more second context text sequences and a respective natural language identifier for each of said second context text sequences; generating a second input sequence comprising the input text query, the one or more second context text sequences, and the respective natural language identifiers for each of the one or more second context text sequences; (i) a fourth output text subsequence that is a response to the input text query; and (ii) a fifth output text subsequence, which is one of the respective natural language identifiers for the second context text sequence; and (iii) a sixth output text subsequence that is text from the second context text sequence identified by the natural language identifier in the fifth output text subsequence; and processing the second input text sequence using the autoregressive language model neural network to generate a second output text sequence comprising: generating a respective score for each output text sequence in a set including the first and second output text sequences; determining that the first output text sequence has a higher score than any output text sequence in the set; in response to determining that the first output text sequence has the highest score, providing at least the first output text subsequence and the third output text subsequence in response to the input text query.

4. The method of claim 1 , further comprising:

5. generating a respective score for each output text sequence in a set including the first and second output text sequences, The method of claim 4 comprising the step of scoring each of the output text sequences using a trained reward model.

6. the first output sequence includes a respective token from a vocabulary of tokens at each of a plurality of time steps, and the autoregressive neural network is configured to generate, for each time step in the first output sequence, a respective score for each token in the vocabulary conditional on the first input text sequence and any token in the output sequence at any time step prior to the time step of the first output sequence, and generating the first output sequence comprises:

6. The method of claim 1, comprising at each time step selecting the token at that time step using the respective scores of the tokens in the vocabulary generated by the neural network for that time step.

7. generating the first output sequence, At each time step in the second output text subsequence after the first time step in the second output text subsequence, receiving the respective scores generated by the neural network at the time steps; generating a constrained score distribution in one of the natural language identifiers that assigns non-zero scores only to tokens that immediately follow a token already generated in the second output text subsequence; sampling the tokens from the constrained score distribution at the time steps; The method of claim 6, comprising:

8. The second output text subsequence is preceded by one or more first predetermined syntactic tokens in the first output text sequence, and generating the first output sequence includes: determining, at a particular time step, that the one or more first predefined syntactic tokens were selected in one or more time steps immediately preceding the particular time step, and accordingly determining that the particular time step is the first time step in the second output text subsequence; receiving the respective scores generated by the neural network at the particular time step; in response to determining that the particular time step is the first time step in the second output text subsequence, generating a constrained score distribution that assigns non-zero scores to only those tokens that are the first token in one of the natural language identifiers; sampling the tokens from the constrained score distribution at the time steps; The method of claim 7 comprising:

9. generating the first output sequence, At each time step in the third output text subsequence after the first time step in the third output text subsequence, receiving the respective scores generated by the neural network at the time steps; generating a constrained score distribution that assigns non-zero scores only to tokens in the first context text sequence identified by the natural language identifier in the second output text subsequence that immediately follow the token already generated in the third output text subsequence; sampling the tokens from the constrained score distribution at the time steps; 9. The method according to claim 1 , comprising:

10. the third output text subsequence is preceded by one or more second predetermined syntactic tokens in the first output text sequence, generating the first output sequence comprising: determining, at a second particular time step, that the one or more second predefined syntactic tokens were selected at one or more time steps immediately preceding the second particular time step, and accordingly determining that the particular time step is the first time step in the third output text subsequence; receiving the respective scores generated by the neural network at the particular time step; in response to determining that the particular time step is the first time step in the third output text subsequence, generating a constrained score distribution that assigns non-zero scores only to tokens that appear in the first context text sequence identified by the natural language identifier in the second output text subsequence; sampling the tokens from the constrained score distribution at the time steps; The method of claim 9 comprising:

11. obtaining one or more first context text sequences and a respective natural language identifier for each of said first context text sequences, said step comprising: submitting a query derived from the input text query to a search engine; obtaining one or more context documents from the search engine in response to the query; selecting the one or more first context sequences from the one or more context documents; The method according to any one of claims 1 to 10, comprising:

12. The method of claim 11 , wherein the respective natural language identifier for each of the first context text sequences is a title of the context document from which the first context text sequence was selected.

13. 13. The method of claim 1, wherein the neural network is pre-trained through unsupervised learning based on a language modeling objective.

14. 14. The method of claim 1, wherein the neural network is fine-tuned through supervised learning, reinforcement learning, or both.

15. One or more computers; one or more storage devices storing instructions that, when performed by the one or more computers, cause the one or more computers to perform the respective operations of any one of claims 1 to 14; A system comprising:

16. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective operations of the method described in any one of claims 1 to 14.

Images