Generating output sequences with inline evidence using language model neural networks

Abstract

The present invention provides a method, system, and apparatus for generating an output sequence using a language model neural network, which includes a computer program encoded on a computer storage medium. [Solution] The output sequence includes the response to the input query and inline evidence, including quotations from contextual documents that support the response.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Application No. 63 / 320,633, filed on March 16, 2022, which is hereby incorporated by reference in its entirety.

[0002] This specification relates to using a neural network to process an input to generate an output sequence.

Background Art

[0003] A neural network is a machine - learning model that employs one or more layers of non - linear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of each set of parameters.

Prior Art Documents

Non - Patent Documents

[0004]

Non - Patent Document 1

Non - Patent Document 2

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that uses a language model neural network to generate a response to an incoming request. In particular, the response generated by the system includes (i) the response to the request and (ii) “evidence” from one or more contextual text documents that support the response. The evidence includes direct quotations from one of the contextual text documents.

[0006] For example, this system could provide an interface between a user and an information retrieval system that accesses a corpus of documents. This interface would allow the system to leverage the information retrieval system to provide more reliable information, particularly verifiable and accurate information.

[0007] In one embodiment, the method includes the steps of: receiving an input text query; obtaining one or more first context text sequences and a respective natural language identifier for each of the first context text sequences; generating a first input sequence including an input text query, one or more first context text sequences, and a respective natural language identifier for each of the one or more first context text sequences; processing the first input text sequence using an autoregressive language model neural network to generate a first output text sequence comprising (i) a first output text subsequence that is a response to the input text query; (ii) a second output text subsequence that is one of the respective natural language identifiers for the first context text sequences; and (iii) a third output text subsequence that is text from the first context text sequences identified by the natural language identifier in the second output text subsequence; and providing at least a first output text subsequence and a third output text subsequence in response to an input text query.

[0008] In some implementations, the step of providing at least a first output text subsequence and a first context text sequence in response to an input text query comprises the step of providing a first output text subsequence, a second output text subsequence, and a third output text subsequence in response to the query.

[0009] In some implementations, the method further includes the steps of determining the source of a first context text sequence identified by a natural language identifier within the second output text subsequence from a second output text subsequence, and providing a reference to the source of the first context text sequence in response to a query.

[0010] In some implementations, the method includes the steps of: obtaining one or more second context text sequences and a natural language identifier for each of the second context text sequences; generating a second input sequence including an input text query, one or more second context text sequences, and a natural language identifier for each of the one or more second context text sequences; (i) a fourth output text subsequence which is a response to the input text query; (ii) a fifth output text subsequence which is one of the natural language identifiers for each of the second context text sequences; and (iii) a number identified by the natural language identifier in the fifth output text subsequence. The method further includes the steps of: processing a second input text sequence using an autoregressive language model neural network to generate a second output text sequence comprising a sixth output text subsequence which is text from two context text sequences; generating a score for each output text sequence in a set which includes the first and second output text sequences; determining that the first output text sequence has a higher score than any other output text sequence in the set; and, in response to the determination that the first output text sequence has the highest score, providing at least a first output text subsequence and a third output text subsequence in response to an input text query.

[0011] In some implementations, the step of generating a score for each output text sequence in a set containing the first and second output text sequences comprises the step of scoring each output text sequence using a trained reward model.

[0012] In some implementations, the first output sequence includes each token from the token vocabulary at each of multiple time steps, and the autoregressive neural network is configured to generate a score for each token in the vocabulary, conditional on the first input text sequence and any token in the output sequence at any time step prior to the time step of the first output sequence (any token in the output sequence preceding the current token), for each time step in the first output sequence (for example, at each time step of multiple time steps, the token corresponding to the current time step may be conveniently called the "current token"), and the step of generating the first output sequence includes, at each time step, a step of selecting the token at the time step (the current token) using the respective scores of the tokens in the vocabulary generated by the neural network for the time step.

[0013] In some implementations, tokens in the second output text subsequence also correspond to corresponding tokens in a second set of time steps. The step of generating the first output sequence includes receiving the respective scores generated by the neural network at each time step (each of the second set of time steps) in the second output text subsequence after the first time step in the second output text subsequence; generating a constrained score distribution that assigns a non-zero score only to tokens immediately following tokens already generated in the second output text subsequence in one of the natural language identifiers; and sampling tokens at a time step from the constrained score distribution.

[0014] In some implementations, one or more first predetermined syntactic tokens in the first output text sequence are placed before the second output text subsequence, and the step of generating the first output sequence includes: determining that one or more first predetermined syntactic tokens were selected in one or more time steps immediately preceding the particular time step, and accordingly determining that the particular time step is the first time step in the second output text subsequence; receiving the respective scores generated by the neural network at the particular time step; generating a constrained score distribution that assigns non-zero scores only to tokens that are the first tokens in one of the natural language identifiers, in response to the determination that the particular time step is the first time step in the second output text subsequence; and sampling tokens at the time step from the constrained score distribution.

[0015] In some implementations, tokens in a third output text subsequence also correspond to corresponding tokens in a third or multiple time step. The step of generating a first output sequence includes: receiving the respective scores generated by the neural network in each time step (i.e., each of the third or multiple time steps) in the third output text subsequence after the first time step in the third output text subsequence; generating a constrained score distribution that assigns non-zero scores only to tokens immediately following tokens already generated in the third output text subsequence in the first context text sequence, identified by a natural language identifier in the second output text subsequence; and sampling tokens in a time step from the constrained score distribution.

[0016] In some implementations, a third output text subsequence is preceded by one or more second predetermined syntactic tokens in the first output text sequence, and the step of generating the first output sequence includes: determining that one or more second predetermined syntactic tokens were selected in one or more time steps immediately preceding a second specific time step, and accordingly determining that the specific time step is the first time step in the third output text subsequence; receiving the respective scores generated by the neural network at the specific time step; generating a constrained score distribution that assigns non-zero scores only to tokens appearing in a first context text sequence identified by a natural language identifier in the second output text subsequence, in response to the determination that the specific time step is the first time step in the third output text subsequence; and sampling tokens at the time step from the constrained score distribution.

[0017] In some implementations, the step of obtaining one or more first context text sequences and a natural language identifier for each of the first context text sequences comprises the steps of submitting a query derived from an input text query to a search engine, obtaining one or more context documents from the search engine in response to the query, and selecting one or more first context sequences from the one or more context documents.

[0018] In some implementations, the natural language identifier for each of the first context text sequences is the title of the context document from which the first context text sequence was selected.

[0019] In some implementations, neural networks are pre-trained through unsupervised learning based on the purpose of language modeling.

[0020] In some implementations, the neural network is fine-tuned through supervised learning, reinforcement learning, or both.

[0021] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages.

[0022] The system described in this specification provides a user interface for accessing a generative language model neural network that generates responses to received requests. In particular, generative language models (LMs) are becoming increasingly useful for answering questions about the world. However, by default, LMs generate unfounded claims, so the user must choose whether to accept them blindly or verify them themselves.

[0023] This specification describes techniques that are useful for a user to evaluate responses generated by an LM by generating claims and supporting evidence. In particular, this evidence takes the form of verbatim quotes extracted from longer context documents retrieved from one or more text databases. The documents can be retrieved by an Internet search engine or any other suitable information retrieval system. Thus, the system provides a user interface between the user and the information retrieval system and increases the reliability and verifiability of the information obtained using the information retrieval system.

[0024] To ensure that the quotes are "verbatim" using generative techniques, this specification describes the special syntax used by the language model when quoting from documents and, in some cases, restricts the output of the language model based on this syntax to result in an exact quote from the retrieved document. This can ensure that the language model accurately quotes from the context document, even if the model has been pre-trained for purposes that do not require quoting from the input.

[0025] Furthermore, large-scale language models implemented as neural networks can produce excellent results in various natural language processing tasks, including question answering. However, some of these models, particularly transformer-based implementations, can have over a billion parameters, requiring considerable computing resources, power, and time to process network inputs to generate network outputs. In some cases, such models can have over 10 billion or even 100 billion parameters. When such models are used on a large scale to meet numerous user demands, a significant amount of energy is consumed.

[0026] Additional considerations arise when neural networks are implemented in backend components that communicate with digital assistant devices, such as mobile devices, via data communication networks including the internet, particularly within computing systems that include data servers. Therefore, it is necessary to optimize the computing load between the digital assistant device and the backend components. This need can be particularly severe for large language models, as their memory and computing requirements are considerably greater than those typically found in mobile devices.

[0027] The techniques described herein address these problems. In some implementations, the techniques described facilitate computational load reduction and improved load balancing, particularly when large-scale language models are implemented as neural networks in multitasking and parallel processing computer systems distributed across multiple sites and interconnected by data communication networks.

[0028] In some implementations, the techniques described enable effective distribution of computing load between local mobile computing devices and backend servers on the network. More specifically, in some implementations, conditioning language model neural networks to contexts representing documents retrieved from internet searches based on a question allows for the use of smaller language model neural networks, making it easier to implement neural networks on mobile devices with limited memory and computing resources.

[0029] Furthermore, using the techniques described herein, a system can leverage search engine results to generate predictions about input text using the most up-to-date information contained in the search engine results. Some existing systems use pre-trained neural networks to generate predictions without accessing such search engine results, and the reliability of these predictions can be low because the neural network can only encode information that was available to it during training; that is, these predictions rely on old information and are therefore inaccurate or at least outdated. Therefore, using the techniques described herein, a system can generate more accurate and timely predictions.

[0030] Furthermore, some existing systems require repeated retraining of the neural network to ensure that it encodes the most up-to-date information. The system described herein can repeatedly access the results of a new search engine, thus eliminating the need to retrain the neural network and thus significantly saving computational resources.

[0031] Using the techniques described herein, a system can generate predictions for input text using information encoded in multiple different documents provided by a search engine in response to the processing of the search engine's queries. Each of these different documents may contain different information relevant to the prediction. Therefore, the predictions generated by the system may be more accurate than those generated using a single document.

[0032] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description.

[0033] Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0034] [Figure 1] This is a diagram illustrating an exemplary sequence generation system. [Figure 2] This is a flowchart illustrating an exemplary process for generating an output sequence. [Figure 3] This is a flowchart illustrating an exemplary process for selecting candidate output sequences. [Figure 4] This figure shows an exemplary user interface that presents the output sequence to the user. [Figure 5] This figure shows an example of training a language model neural network. [Figure 6] This figure shows an exemplary user interface for evaluating the generated samples. [Modes for carrying out the invention]

[0035] Similar reference numbers and names in various drawings refer to the same elements.

[0036] Figure 1 shows an exemplary sequence generation system 100. The sequence generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, and can implement the systems, components, and techniques described below.

[0037] The sequence generation system 100 functions as a user interface to an information retrieval system that accesses one or more text databases (not shown), or it provides functionality to a user interface implemented on a user computer that is separate from the sequence generation system 100 but communicates with it. The text databases collectively form a corpus of documents. The corpus of documents may be, for example, web pages and other documents accessible via the Internet. Alternatively, the corpus of documents may be, for example, part of a proprietary text database of a scientific publisher or other organization. The sequence generation system 100 processes an input text query 102 from a user using a context sequence generation system 104, an input sequence generation system 110, and a language model neural network 114 to generate an output sequence 116.

[0038] The input text query 102 may be a query submitted to the system 100 by the user through the user computer, a question submitted to the system 100 through the user computer, or another request requiring a response from the system 100. In some cases, the system receives the query from the user computer as text. In other cases, the system receives a natural language voice query from the user and converts the voice into the input text query 102 by applying a speech recognition engine to the voice. The input text query 102 is received in the form of a sound (voice) signal captured by the microphone of the user computer and converted by a speech recognition engine, i.e., a voice-to-text converter, to form the input text query 102. Alternatively, it may be typed using a data input device on the user computer.

[0039] When system 100 receives an input text query 102, context sequence generation system 104 obtains one or more first context text sequences 106 and the respective natural language identifier 108 for each of the first context text sequences 106.

[0040] For example, each context text sequence 106 can be extracted from its own context document, and the identifier 108 can be the title of the context document. Alternatively, some or all of the context text sequences 106 can be extracted from the same context document, and the identifier 108 may be the section header or other identifier of the part of the document from which the context text sequences are extracted.

[0041] The acquisition of the context sequence will be explained in more detail below, referring to Figure 2.

[0042] Next, the input sequence generation system 110 generates a first input sequence 112 which includes an input text query 102, one or more first context text sequences 106, and a natural language identifier 108 for each of the one or more first context text sequences.

[0043] For example, the first input sequence 112 may include a query 102, a context text sequence 106, and an identifier 108 arranged according to a predetermined input syntax. In some cases, the first input sequence 112 may also include other text, such as one or more natural language "prompts," one or more separator tokens separating the various elements of the input sequence, or both. A natural language prompt is an example of an input-output pair, where the input is an example of an input that can be provided, and the output is an example of an output that will be produced. Prompts are described in more detail below.

[0044] Next, the sequence generation system 100 processes the first input sequence 112 using an autoregressive language model neural network 114 to generate a first output text sequence 116.

[0045] The output sequence 116 includes (i) a first output text subsequence which is a response to the input text query 102; (ii) a second output text subsequence which is one of the natural language identifiers 108 for the first context text sequence 106; and (iii) a third output text subsequence which is text from the first context text sequence identified by the natural language identifier in the second output text subsequence.

[0046] In particular, (i), (ii), and (iii) are placed within the output sequence according to a predetermined output syntax. An example of the predetermined syntax is described in more detail below with reference to Figure 3.

[0047] Next, the sequence generation system 100 provides at least a first output text subsequence and a third output text subsequence in response to the input text query 102. Thus, the system 100 provides a text response to the input text query 102 and text from one of the context text sequences 106 as supporting evidence for the text response.

[0048] In some implementations, the sequence generation system 100 generates multiple candidate output sequences 116 in response to an input query 102.

[0049] In these implementations, system 100 also generates a score for each candidate output sequence and, in response to a user query, provides only the text from the candidate output sequence with the highest score.

[0050] In some of these implementations, if none of the candidates have a score above a threshold, system 100 instead outputs a default text response to the user's query, such as "I don't know" or "I am not sure."

[0051] The scoring of candidate output sequences is explained below, with reference to Figure 3.

[0052] The language model neural network 114 can be any suitable language model neural network that receives an input sequence consisting of text tokens selected from a vocabulary and autoregressively generates an output sequence consisting of text tokens from the vocabulary. For example, the language model neural network 114 may be a transformer-based language model neural network or a recurrent neural network-based language model.

[0053] A token in a vocabulary can be any suitable text token representing one or more text elements in natural language, such as a word, part of a word, punctuation, etc., and optionally, a number or other text symbol found in a corpus of text. Generally, the input text query 102, the natural language identifier 108, and / or the context text sequence 106 are also sequences of tokens selected from the vocabulary.

[0054] The language model neural network 114 is called an autoregressive neural network because it autoregressively generates the output sequence of tokens by generating each particular token in the output sequence, given the current input sequence which includes any tokens that precede a particular text token in the output sequence, i.e., tokens that have already been generated for any previous position that precedes a particular position of a particular token in the output sequence, and a context input that provides the context of the output sequence.

[0055] For example, the current input sequence when generating a token at any given position in the output sequence can include the input sequence and tokens from the output sequence at any preceding position that precedes the given position in the output sequence. Specifically, the current input sequence can include the input sequence and tokens at any preceding position that precedes the given position in the subsequent output sequence. Optionally, within the current input sequence, tokens from the input sequence and the output sequence can be separated by one or more predetermined tokens, i.e., a specified set of one or more tokens from the vocabulary in the current input sequence. That is, there may be one or more predetermined tokens between the tokens from the input sequence and the tokens from the output sequence.

[0056] More specifically, in order to generate a specific token at a specific position in the output sequence, the neural network 114 can process the current input sequence to generate a score distribution, such as a probability distribution, which assigns a score, such as a probability, to each token in the token vocabulary. The neural network 114 can then use the score distribution to select a token from the vocabulary as a specific token. For example, the neural network 114 could greedily select the token with the highest score, or it could sample tokens from the distribution using, for example, nuclear sampling or another sampling technique.

[0057] As a concrete example, the language model neural network 114 could be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each applying self-attentional behavior, and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0058] Lieut The snake squirrel of the squid The snowflakes are worth it These are the results of J.S. Hoffmann, S. Borgeaud、A. Mensch、E. Buchatskaya、T. Cai、E. D. Rutherford. d. L. Houses LA Hendricks. Welbl、A. Clark, “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022, JW Rae, and S. Clark. Borgeaud、T. Cai、K. Millican, J. (1999). Hoffmann, HF Song, J. Aslanides、S. R. Henderson. Ring、S. Young、E. Rutherford;T. Hennigan、J. Menick、A. Cassirer、R. Powell、G. of the Driessche、LA Hendricks、M. Rauh、P. Huang、A. Glaese、J. Welbl、S. Dathathri、S. Huang、J. Uesato、J. Mellor、I. Higgins、A. Creswell、N. McAleese、A.Wu、E. Elsen, SM Jayakumar, E.K. Buchatskaya、D. Budden、E. Sutherland、K. Simonyan、M. Paganini、L. Sifre、L. Martens, XL Li, A. Kuncoro、A. Nematzadeh、E. Gribovskaya、D. Donato、A. Lazaridou、A. Mensch, J. Chem. Lespiau、M. Tsimpoukelli、N. Grigorev、D. Fritz、T. Sottiaux、Mr. Pajarskas、T. Pohlen、Z. Gong、D. Toyama、C. of Masson of Autumn、Y. Li、T. Terzi、V. Mikulik、I. Babuschkin, A. Clark、D. of The Houses、A. Guy、C. Jones, J.S.Bradbury、M. Johnson, BA Hechtman, L. Weidinger、I. Gabriel、WS Isaac、E. Lockhart、S. Osindero、L. Rimell、C. Dyer、O. Vinyals、K. Ayoub、J. Stanway、L. Bennett、D. Hassabis、K. Kavukcuoglu、およびG. Irving, “Scaling Language Models: Methods, Analysis, and Insights from Training Gopher” (CoRR, abs / 2112.11446, 2021), Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, and Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu transformer arXiv arXiv:1910.10683 arXiv:1910.10683 2019 Daniel Adiwardana Minh-Thang Luong David R. So Jamie Hall Noah Fiedel Roma Thoppilan Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le, and Towards a Human-Like Open-Domain chatbot CoRR, abs / 2001.09977, 2020, Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam、Girish Sastry、Amanda Askellらによる「Language models are few-shot learners can be downloaded from arXiv arXiv:2005.14165.

[0059] However, generally, transformer-based neural networks include a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives its respective input hidden state for each input token in the given input sequence. The attention block then updates each of its hidden states at least partially by applying self-attention to generate its respective output hidden state for each input token. The input hidden state of a first attention block is the embedding of the input tokens in the input sequence, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0060] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence of the last input token in the input sequence in order to generate the score distribution.

[0061] Generally, since the neural network 114 is autoregressive, the system 100 can use the same neural network 114 to generate multiple different candidate output sequences in response to the same request by, for example, using beam search decoding from the score distribution generated by the neural network 114, by using sample and rank decoding strategies, by using different random seeds for the pseudorandom number generator used when sampling different runs through the neural network 114, or by using another decoding strategy that takes advantage of the autoregressive properties of the neural network 114.

[0062] In some implementations, the language model 114 is pre-trained, i.e., trained on language modeling tasks that do not require it to respond to user questions and provide evidence, and the system 100 causes the neural network 114 to generate an output sequence according to a predetermined syntax through natural language prompts in the input sequence.

[0063] For example, system 100 or another training system pre-trains the language modeling neural network 114 on a language modeling task, such as a task where, given the current sequence of text tokens, it needs to predict the next token following the current sequence in the training data. As a specific example, the language modeling neural network 114 can be pre-trained based on a maximum likelihood goal of a large dataset of text, such as text published from the internet or another text corpus.

[0064] In some other implementations, after initial training, the system 100 fine-tunes the language model 114, for example, through supervised learning, reinforcement learning, or both, based on the objective of generating output sequences according to a specific syntax. This is explained in more detail below with reference to Figure 5.

[0065] In some of these implementations, the system 100 still includes one or more natural language prompts as input to the language model 114 during inference, i.e., after training.

[0066] As described above, a natural language prompt is an example of an input-output pair, where the input is an example of an input that can be provided, and the output is an example of an output that will be produced. Thus, each prompt includes an exemplary input sequence, which is an exemplary query; an exemplary set of one or more context sequences; and identifiers for one or more context sequences arranged according to a predetermined input syntax. Each prompt also includes an exemplary first output text subsequence, which is an exemplary response to an exemplary query, arranged according to an output syntax; an exemplary second output text subsequence, which is one of the natural language identifiers for one of the exemplary context text sequences; and an exemplary third output text subsequence, which is text from an exemplary context text sequence identified by the natural language identifier in the exemplary second output text subsequence. Optionally, the input sequences may also include one or more tokens from a vocabulary that separates each prompt and one or more tokens that separate the final prompt from the user query.

[0067] Furthermore, in some implementations, system 100 performs "constrained sampling" when selecting tokens to include in the output sequence. This ensures that the output of neural network 114 conforms to the syntax and that the sequence is internally consistent, i.e., that the evidence is a direct quote from the context text sequence 106 identified by the natural language identifier 108 in the output sequence.

[0068] When system 100 generates multiple candidate output sequences, constrained sampling eliminates the need for the system to score invalid or inconsistent output sequences, significantly reducing the number of candidates that need to be generated to guarantee high-quality output, and greatly improving the computational efficiency of system 100, i.e., reducing the amount of computational resources consumed by system 100.

[0069] An example of constrained sampling is explained in more detail below, referring to Figure 3.

[0070] Figure 2 is a flowchart of an exemplary process 200 for generating an output sequence given an input query. For convenience, the process 200 is described as being executed by a system of one or more computers located in one or more locations. For example, a sequence generation system appropriately programmed according to this specification, such as the sequence generation system 100 shown in Figure 1, can execute the process 200.

[0071] This system, for example, receives input text queries from the user using a user interface (step 202).

[0072] The system obtains one or more first contextual text sequences and a natural language identifier for each of the first contextual text sequences (step 204).

[0073] For example, the system can obtain one or more context sequences and their respective natural language identifiers by submitting a search query derived from an input text query to a search engine. The search engine is configured to access a corpus of documents and search the corpus based on the search query. For example, the search query may be the same text as the input text query, or it may be modified by the system to add synonyms or correct typos or spelling errors.

[0074] Next, the system can retrieve one or more documents from a search engine in response to a search query. These documents can then be ranked by the search engine, for example, based on their quality and relevance to the received search query.

[0075] Next, the system can select one or more first context sequences from one or more context documents, for example, by selecting one or more top-ranked search results. The system also associates a natural language identifier with each first context sequence.

[0076] In some implementations, search engines also provide snippets from corresponding contextual documents as part of the search results, identifying the corresponding contextual documents. In some of these implementations, the system can generate a contextual sequence for a given document by extracting the snippet and the text surrounding it from the corresponding contextual document. For example, the system can use snippets to extract snippet text to account that document lengths vary and often exceed the maximum context window size of the language model (as described below).

[0077] Therefore, especially in the case of a few-shot prompt when presenting multiple documents at once, the system may need to limit the number of tokens spent on document content within a given input sequence. Thus, the system can truncate documents by using snippets as described above. For example, the system can use snippets to truncate a given document into fragments of the maximum token length, so that each fragment contains the relevant search snippet.

[0078] In some implementations, this system can ensure that the truncated fragments begin at the beginning of a sentence or paragraph.

[0079] As a concrete example, during train operation, the system can randomly select such starting positions to increase input diversity. During inference, the system can allow a maximum number of characters, for example 250, 500, or 1000, before the start of the snippet fragment, identify a first sentence that begins within that range, and use that first sentence as the beginning of the truncated fragment.

[0080] A search engine can be any suitable search engine that is accessible by a system and searches any suitable corpus of documents, such as web pages, books, or other documents. For example, a search engine could be an internet search engine that searches the internet and returns results that refer to documents available on the internet. Another example is a search engine that searches a private corpus of documents, such as documents available on an internal network or stored in a collection of one or more databases.

[0081] For example, each natural language identifier for each of the first context text sequences could be the title of the context document from which the first context text sequence is selected.

[0082] The system generates a first input sequence, which includes an input text query, one or more first context text sequences, and a natural language identifier for each of the one or more first context text sequences (step 206).

[0083] This system processes a first input text sequence using an autoregressive language model neural network to generate a first output text sequence (step 208).

[0084] The first output text sequence includes a first output text subsequence which is the response to the input text query, a second output text subsequence which is one of the natural language identifiers for the first context text sequence, and a third output text subsequence which is the text from the first context text sequence identified by the natural language identifier in the second output text subsequence.

[0085] The system responds to an input text query by providing at least a first output text subsequence and a third output text subsequence (for example, to the user) (step 210).

[0086] This system can provide a first output text subsequence, a third output text subsequence, and optionally a second output text subsequence in response to a query.

[0087] Furthermore, in some implementations, the system can determine the source of a first contextual text sequence identified by a natural language identifier within the second output text subsequence from a second output text subsequence, and provide a reference to the source of the first contextual text sequence in response to a query. For example, the system can provide a reference to the source of the first contextual text sequence, such as a hyperlink linking to a web page.

[0088] An example of the output sequence generated by the system is described below with reference to Figure 4.

[0089] As described above, in some implementations, the system generates a set of multiple candidate output sequences (including the first output text sequence) and a score for each candidate output sequence, and in response to the determination that the first output text sequence has the highest score among the candidate output sequences, it provides only the first output sequence.

[0090] For example, when processing a first input text sequence, this system can generate at least a portion of the candidate output sequences in a set by sampling different candidate output sequences from the output generated by a language model neural network.

[0091] Furthermore, in some implementations, the system can generate more context sequences than fit within the "context window" of the language model neural network. That is, the language model neural network may only be able to process input sequences that do not exceed a maximum number of characters, for example, due to memory constraints or the framework in which the neural network was trained. In some implementations, this maximum number may be exceeded when natural language identifiers and tokens for all context sequences are included. In these implementations, the system generates multiple different input sequences, each containing a subset of the context sequences.

[0092] In other words, the system can also obtain one or more second context text sequences in addition to the first context text sequence, and a natural language identifier for each of the second context text sequences, and generate a second input sequence containing an input text query, one or more second context text sequences, and a natural language identifier for each of the one or more second context text sequences. The system can then process the second input text sequence using an autoregressive language model neural network to generate a second output text sequence comprising (i) a fourth output text subsequence which is a response to the input text query, (ii) a fifth output text subsequence which is one of the natural language identifiers for each of the second context text sequences, and (iii) a sixth output text subsequence which is text from the second context text sequence identified by the natural language identifier in the fifth output text subsequence.

[0093] Next, the system generates a score for each candidate output text sequence in a set, for example, a set containing the first and second output text sequences, and determines that the first output text sequence has a higher score than any other output text sequence in the set. In some cases, this can be done by using a trained reward model to score each output text sequence. The use of a trained reward model to score candidate output sequences is described below with reference to Figure 3.

[0094] Subsequently, in response to the input text query, the system can provide at least a first output text subsequence and a third output text subsequence, based on the determination that the first output text sequence has the highest score.

[0095] Figure 3 shows an example of how a sequence generation system works when it generates multiple candidate output sequences in response to a given text query.

[0096] As shown in the example in Figure 3, system 100 receives question 302 from, for example, a user computer.

[0097] System 100 performs an internet search 304 to identify the top k documents most relevant to question 302. Generally, k is an integer greater than 1, for example, 5, 10, 20, or 100. For example, System 100 can provide question 302 or a query derived from question 302 to an internet search engine and obtain search results from the internet search engine that identify the top k documents.

[0098] Next, the system uses a generator 306 to generate one or more input sequences to the language model neural network 114 and to sample N candidate output sequences using the language model neural network 114 (308). In some implementations, the number of candidate output sequences N is greater than the number of documents k.

[0099] For example, the generator 306 can generate a single input sequence containing context from all k documents in order to sample N candidate output sequences, and then process this single input sequence multiple times using the language model neural network 114.

[0100] As another example, the generator 306 can generate multiple input sequences, each containing context from a subset of k documents, in order to sample N candidate output sequences, and then process each of these input sequences multiple times using the language model neural network 114.

[0101] As another example, the generator 306 can generate multiple input sequences, each containing a context from one of k documents, and then process each of these input sequences using the language model neural network 114.

[0102] In any of the above examples, multiple input sequences may be sampled in a round-robin order until N candidate output sequences have been sampled.

[0103] In some implementations, N can be a multiple of k. In other implementations, N may not be divisible by k.

[0104] Next, system 100 performs reward model scoring 310 for each of the N candidate output sequences.

[0105] In other words, system 100 uses the learned reward model to assign a score to each of the N candidate output sequences.

[0106] The trained reward model 310 is a model, such as another language model neural network, that takes an input text query and responses and quotes generated by the neural network 114 as input and outputs a score representing the quality of the responses and quotes. For example, the score could represent the likelihood that the user prefers that response (and quote) compared to other responses (and accompanying quotes) to the same query generated by the neural network 114.

[0107] The training of the reward model is explained below, with reference to Figure 5.

[0108] Next, system 100 selects the "best" sample 312 as the final output sequence, i.e., the candidate output sequence with the highest score from N sequences according to the learned reward model.

[0109] In some implementations, if none of the candidates have a score above a threshold, the system 100 instead outputs a default text response to the user's query, such as "I don't know" or "I'm not sure."

[0110] As described above, each candidate output sequence includes (i) a first output text subsequence which is a response to an input text query, (ii) a second output text subsequence which is one of the natural language identifiers for the first context text sequence, and (iii) a third output text subsequence which is text from the first context text sequence identified by the natural language identifier in the second output text subsequence.

[0111] In particular, (i), (ii), and (iii) are placed within the output sequence according to a predetermined output syntax.

[0112] As shown in the example in Figure 3, the output syntax is %<Claim>%(Document Title)%[Quote from Document]% In the above formula, "%<" ">%(" ")%["and"]%" are template tokens, i.e., predetermined syntactic tokens that are inserted before and after subsequences, "claim" is a placeholder for the first output text subsequence, "document title" is a placeholder for the second output text subsequence, and "quote from document" is a placeholder for the third output text subsequence.

[0113] However, you can use any of the various syntaxes that place the “claim” placeholder, “document title” placeholder, and “document quote” placeholder in predetermined locations within the output sequence.

[0114] In some implementations, and as described above, the system uses constrained sampling to sample each of the N candidates in order to ensure that each candidate satisfies the syntax, i.e., contains an exact quote from a context sequence identified by a natural language identifier within the sequence.

[0115] In other words, as described above, the generator 306 samples a given candidate output sequence by generating a score for each token in the vocabulary, conditional on the first input text sequence and any token in the output sequence at any time step prior to the time step of the first output sequence, and at each time step, selects a token for that time step using the respective scores of the tokens in the vocabulary generated by the neural network for that time step.

[0116] When constrained sampling is employed, the system restricts sampling to only those tokens that will be the next valid token according to the output sequence.

[0117] For example, when generating a second output text subsequence, and in each time step within the second output text subsequence after the first time step within the second output text subsequence, the generator 306 can receive the respective scores generated by the neural network in the time step, generate a constrained score distribution that assigns a non-zero score only to tokens immediately following tokens already generated in the second output text subsequence for one or more of the natural language identifiers, and then sample tokens in the time step from the constrained score distribution rather than the received score distribution. That is, the system restricts sampling to assign a non-zero score only to tokens that, when added to tokens already selected for the second output text subsequence, generate one or more valid prefixes among the natural language identifiers in the corresponding input sequence.

[0118] As another example, in some cases, one or more first predetermined syntax tokens from the first output text sequence may precede the second output text subsequence. For example, in the example in Figure 3, the token ">%(" is placed before the second output text subsequence in the output syntax.

[0119] In these cases, the step of generating an output sequence using constrained sampling includes determining that one or more first predetermined syntactic tokens were selected in one or more time steps immediately preceding the particular time step, and accordingly determining that the particular time step is the first time step in a second output text subsequence. For example, the system may determine that the token ">%(" has already been sampled, and accordingly determine that the next time step is the first time step in a second subsequence.

[0120] In this example, the system can receive each score generated by the neural network at a given time step, generate a constrained score distribution that assigns a non-zero score only to the first token in one of the natural language identifiers in the corresponding input sequence, in response to the determination that the given time step is the first time step in the second output text subsequence, and sample tokens from the constrained score distribution at the time step. That is, the system restricts sampling to assign a non-zero score only to tokens that are one or more first tokens in the natural language identifiers in the corresponding input sequence.

[0121] As another example, when using constrained sampling, at each time step in the third output text subsequence after the first time step in the third output text subsequence, the system can receive the respective scores generated by the neural network at the time step and generate a constrained score distribution that assigns a non-zero score only to tokens immediately following tokens already generated in the third output text subsequence in the first context text sequence identified by the natural language identifier in the second output text subsequence. The system then samples tokens from the constrained score distribution at the time step. That is, the system restricts sampling to assign a non-zero score only to tokens that, if added to the tokens already selected for the third output text subsequence, directly match the subsequence in the first context text sequence identified by the natural language identifier in the second output text subsequence. Thus, the system ensures that the third output text subsequence is a direct quote from the context document identified by the natural language identifier in the second output text subsequence.

[0122] As another example, in some cases, one or more second predetermined syntax tokens from the first output text sequence may precede the third output text subsequence. For example, in the example in Figure 3, the token ")%[" is placed before the second output text subsequence in the output syntax.

[0123] In these cases, when using constrained sampling, the system determines that one or more second predetermined syntactic tokens were selected in one or more time steps immediately preceding the second specific time step, and accordingly determines that the specific time step is the first time step in the third output text subsequence. Then, at the specific time step, upon receiving the respective scores generated by the neural network, the system generates a constrained score distribution that assigns non-zero scores only to tokens that appear in the first context text sequence identified by the natural language identifier in the second output text subsequence, and samples tokens from the constrained score distribution at the time step.

[0124] The system 100 then provides the user with at least a portion of the text from the best sample 312. For example, the system 100 can render the presentation of the best sample 312 in the user interface (314).

[0125] As shown in Figure 3, the presentation may include the text of the "claim," i.e., the text of the first subsequence; a quote from a contextual document supporting the "claim," i.e., the text of the third subsequence; and optionally, a document identifier from the second subsequence.

[0126] Figure 4 shows an exemplary user interface 400 that presents the output sequence to the user.

[0127] In the example in Figure 4, the user submits query 402, "What kind of animal is Scooby-Doo?".

[0128] In response, System 100 generated an output sequence containing three subsequences: (i) "Great Dane," (ii) "Wikipedia page:Scooby-Doo," and (iii) a quote from the Wikipedia page titled "Scooby-Doo."

[0129] Next, in response to user query 402, the system presents a first subsequence 404, a second subsequence 406, and a third subsequence 408 in the user interface 400.

[0130] Furthermore, the system displays the first subsequence 404 as a hyperlink that links to the source of the third subsequence 408, namely the Wikipedia page for Scooby-Doo, specifically the webpage titled "Wikipedia page:Scooby-Doo". Including the hyperlink in the user interface 400 allows the user to navigate to the source indicated by the second subsequence, for example, to verify the accuracy of a citation or to obtain additional context regarding the response.

[0131] Figure 5 shows an example of training a language model neural network 114.

[0132] As shown in Figure 5, the system acquires a pre-trained language model (502).

[0133] For example, as mentioned above, language models may be trained for the purpose of language modeling on a large corpus of text documents.

[0134] After obtaining a pre-trained language model (502), the system generates samples and evaluates the generated samples through human evaluation (504).

[0135] For example, to obtain each evaluation, the system can present the evaluator user with a question and two candidate answers, such as two samples generated using a pre-trained language model with several prompts. Each candidate answer can be divided into a "claim" section and a "supporting evidence" section, as shown above, for example, with reference to Figure 4.

[0136] The system can then obtain input from the evaluator user specifying whether the claim is a valid response to the question, whether the claim is supported by the attached citation evidence, and which answer is preferred by the evaluator user. A valid response to a question is a topically relevant answer to the question. A supported response is one in which the provided evidence is sufficient to verify the validity of the response.

[0137] Figure 6 shows an example of a user interface that can be used to obtain input from the user.

[0138] In other words, Figure 6 shows an exemplary user interface 600 for evaluating a generated sample, which can, for example, receive input for a human to evaluate the generated sample.

[0139] As shown in Figure 6, the user is presented with query 602 and two candidate responses 604 and 606 for query 602. Each candidate response 604 and 606 includes a response to the query, supporting evidence from the response, and an identifier for the source of the supporting evidence.

[0140] For each candidate response 604 and 606, the user interface presents the corresponding selection elements 608 and 610, allowing the user to submit input indicating whether the corresponding candidate response is a valid answer (or indicating that the user is unsure), or whether the corresponding candidate response is supported by the corresponding supporting evidence (or indicating that the user is unsure).

[0141] Selection elements 608 and 610 also allow the user to submit input indicating that the corresponding candidate response 604 or 606 is the preferred response (of the two candidate responses) to query 602, respectively.

[0142] The user interface 600 also allows the user to submit input indicating that two responses are "tied," or to submit comments on the sample.

[0143] Returning to the explanation of Figure 5, the system then uses the evaluated samples to perform supervised fine-tuning (SFT) 506, in which the system trains a language model on the evaluated samples through supervised learning.

[0144] In other words, for each sample used in the SFT, the system trains a language model to generate claims and supporting evidence within the sample, given a set of context sequences containing the questions in the sample and the text of the supporting evidence.

[0145] Optionally, when performing SFT, this system can use only samples that have been evaluated as valid and supported by supervised fine-tuning.

[0146] As a concrete example, this system can generate the input sequence of a given sample during SFT as follows:

[0147] For a certain percentage of the sample, for example, 1 / 3 or 1 / 2 of the data, the system uses only a single document within the context, and this document is the same as the document from which the supporting evidence was extracted, forcing the existence of supporting evidence within the context sequence.

[0148] For the remaining samplers, the system uses n documents within the context, where n is randomly selected from a fixed number between 1 and 5, for example, 5, 10, or 15. Similarly, the system enforces that the target document and its supporting evidence citations exist within the context sequence. For the remaining documents within the context sequence, the system can use, for example, the top n-1 search results for a question provided by a search engine.

[0149] This system can truncate each context document so that the total token length of the input sequence does not exceed a fixed number based on the language model's context window. Since this token length tolerance can be randomly divided among the documents included in the prompt, the language model recognizes context sequences of different sizes from different context documents within the same input sequence. When truncating a given context document to its maximum allowable length, the system can ensure that each document contains a snippet, as described above.

[0150] Optionally, after performing Supervised Fine-Tuning (SFT) 506, the system can use the SFT model to generate additional samples that are re-evaluated through human evaluation.

[0151] Next, the system trains the reward model (RM) 508 on the generated samples, for example, the original generated samples, or additional samples generated using the original generated samples and the SFT model.

[0152] As described above, the trained reward model is a model, such as another language model neural network, that receives input text queries and responses and quotes generated by the neural network 114 as input, and outputs a score representing the quality of the responses and quotes.

[0153] For example, given a query and response string, the system can train a reward model as a classifier that predicts a binary variable indicating which example in a given pair was preferred. That is, the system can calculate the probability that the first example in a pair was preferred, taking into account the scores generated by the reward model for both examples in the pair. For example, the system can train a reward model using a cross-entropy goal, with the user's preference as the truth value and the calculated probability as the predicted value.

[0154] Optionally, during training, the reward model also predicts a reasonable binary judgment of the response in pairs as an auxiliary loss. In these cases, the final loss is, for example, a combination of the mean or weighted mean of the pairwise preference prediction loss and the auxiliary prediction loss.

[0155] In some implementations, the system can add a set of constructed ("synthesized") comparisons to the RM training set. For example, the system can generate comparisons constructed from supported and refuted claims in a fact-checking dataset. An example of such a dataset is the FEVER dataset (Thorne et al., 2018). Including these constructed comparisons provides additional out-of-distribution question-answer modes that are not extracted, allowing the reward model to more effectively verify the supporting evidence. An example of such a dataset, the FEVER dataset, could include claims generated by modifying sentences extracted from offensive text. These claims are then classified as supported, refuted, or poorly supported and marked with the associated evidence. To transform such claims into examples of questions to compare answers, the system can use one of several techniques. Some examples of the types of techniques are described below.

[0156] Type A: The system can generate questions through direct template operation from claims (e.g., "{claim}?", "Is {claim} true?", "Is it correct to say {claim}?", "{claim}. Do you agree?"). Examples compare affirmative answers such as "Yes", "This is correct", and "That is true" with supporting quotes, and negative answers with the same quotes. If the original claim is supported, the affirmative answer is marked as favorable, supported, and reasonable. Otherwise, the negative answer is marked as favorable, supported, and reasonable.

[0157] Type B: This system can translate claims into questions using a pre-trained language model neural network with a few prompts. For example, the claim "Roman Atwood is a content creator" can be translated into "Who is Roman Atwood?". As a comparison of the claims translated into questions, the system can use one answer as the corresponding claim (with supporting citations) from the dataset, and use a direct negation of the claim generated via the template (e.g., "{claim} is not true") as the other answer. If the original claim is supported, the answer containing that claim is marked as favorable, supported, and valid. Otherwise, the negated claim is marked as favorable. As another example, if the original claim is supported, the system can use the original claim as one answer, use a randomly generated claim as a comparison, and mark the original claim as favorable, supported, and valid.

[0158] As described above, the system can then use a reward model during sampling to assign scores to candidate output sequences.

[0159] After training RM508, the system can use the trained reward model to further refine the SFT model through reinforcement learning 510. That is, the system uses the reward model to perform reinforcement learning from human preferences (RLfHP) techniques by training the model to maximize the expected reward predicted by the trained RM508.

[0160] Optionally, the system can then use a further refined model to generate additional samples for human evaluation and to retrain the RM through SFT or RL or both, or to perform both. In other words, the system can repeat the described training loop multiple times to further refine the language model, further refine the reward model, or both.

[0161] Furthermore, while the example in Figure 5 illustrates how the system fine-tunes its language model using both SFT and RL, in some cases, the system may use only SFT or only RL, rather than both. For example, when using a reward model for reranking, performance may be improved by using a model fine-tuned with only SFT or only RL (rather than both), so that the reward model is provided with reranking for a wider variety of samples.

[0162] The following is an explanation of self-attention that can be employed by language model neural networks.

[0163] The self-attention block mentioned above is a neural network layer that includes an attention mechanism that operates on a self-attention block input (or an input derived from the layer input) to generate a self-attention block output. The self-attention mechanism may be causally masked, so that any given position in the input sequence does not attract attention to any positions in the input sequence after that position (for example, data from that position is not used). Various types of attention mechanisms are possible.Examples of the self-attention layer, including the attention mechanism, include "Attention is all you need" by Vaswani et al., 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, California, USA; "Exploring the limits of transfer learning with a unified text-to-text transformer" by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, arXiv preprint arXiv:1910.10683, 2019; and "Towards a human-like open-domain" by Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. "chatbot" CoRR, abs / 2001.09977, 2020, and "Language models are few-shot learners" by Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell, arXiv preprint arXiv:2005.14165, 2020.

[0164] Generally, attention mechanisms map queries and sets of key-value pairs to the output, where the query, key, and value are all vectors. The output is calculated as a weighted sum of the values, with the weights assigned to each value being calculated by a compatibility function, such as the dot product or scaled dot product of the query and its corresponding key.

[0165] Generally, a self-attention mechanism is configured to associate different positions within the same sequence in order to determine a transformed version of the sequence as its output. For example, the input to the attention layer may consist of element-wise vectors of the input sequence. These vectors provide input to the self-attention mechanism, which uses them to determine a new representation of the same sequence for the attention layer output, similarly consisting of element-wise vectors of the input sequence. The output of the self-attention mechanism may be used as the output of the attention layer and may be processed by one or more feedforward layers, skip connections, or normalization operations to provide the output of the attention layer.

[0166] In some implementations, the attention mechanism is a query matrix Q=XW containing the respective query for each vector in the input sequence. Q , a key matrix K=XW containing the respective key for each vector in the input sequence K , and the value matrix V=XW which contains the respective values ​​for each vector in the input sequence. V To derive this, for example, matrix W Q Query transformations defined by, for example, matrix W K Key transformations defined by, and for example, matrix W V Each of the value transformations defined by is configured to be applied to the inputs of the attention layer, which are the input data X to the inputs of the attention layer, and these are used to determine the attention sequence of the output. For example, the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine the respective weight for each value vector, and then combining the value vectors using the respective weights to determine the self-attention layer output for each element of the input sequence. The output of the self-attention layer may be scaled by a scaling factor, for example, the square root of the dimensions of the query and key, in order to implement scaled dot product attention. Thus, for example, the output of the attention mechanism is

[0167]

number

[0168] This can be determined as follows, where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism includes an "additive attention" mechanism that computes the compatibility function using a feedforward network with hidden layers. The output of the attention mechanism may be further processed by one or more fully connected feedforward neural network layers.

[0169] The attention mechanism can implement multi-head attention, meaning that multiple different attention mechanisms can be applied in parallel. These outputs can then be combined, for example, concatenated, with learned linear transformations applied to reduce them to the original dimension as needed.

[0170] In this specification, the term “configured” is used in relation to systems and computer program components. A system of one or more computers being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed on the system, causing the system to perform that operation or action during operation. A system of one or more computer programs being configured to perform a particular operation or action means that the program, when executed by a data processing device, contains instructions that cause the device to perform that operation or action.

[0171] The subject matter and functional operating embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including structures and structural equivalents disclosed herein. Embodiments of the subject matter described herein can be implemented as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by or control of the operation of one or more computer programs, for example, a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may also be encoded into artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0172] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be, and may include, dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include a computer program execution environment, such as processor firmware, a protocol stack, a database management system, an operating system, or code to create code that constructs one or more of these.

[0173] Computer programs may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, and may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files, for example, a file that stores one or more modules, subprograms, or parts of code. Computer programs may be deployed to run on one computer, or on multiple computers located in one site or distributed across multiple sites and interconnected by a data communication network.

[0174] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way, or even unstructured at all, and can be stored on a storage device in one or more locations. For example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0175] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same computer.

[0176] The processes and logic flows described herein can be executed by one or more programmable computers that perform one or more computer programs to perform functions by acting on input data and producing outputs. The processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0177] A computer suitable for executing computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for executing or performing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or integrated into dedicated logic circuits. Generally, a computer includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operationally coupled to receive data from or transfer data to such storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be integrated into another device, for example, a portable storage device such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or Universal Serial Bus (USB) flash drive, to name just a few.

[0178] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, which include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0179] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and pointing device such as a mouse or trackball to which the user can provide input to the computer. Other types of devices can also be used to provide user interaction, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback. Input from the user can also be received in any form, such as acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0180] Data processing units for implementing machine learning models may also include dedicated hardware accelerator units for handling the common and computationally intensive parts of machine learning training or generation, such as inference.

[0181] Machine learning models can be implemented and deployed using machine learning frameworks such as the TensorFlow framework or the Jax framework.

[0182] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component such as a data server, or a middleware component such as an application server, or a frontend component such as a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with the implementation of the subject matter described herein, or in any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0183] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for purposes such as displaying data to a user interacting with a device acting as a client, or receiving user input from the user. Data generated on the user device, such as the results of user interactions, can be received from the device to the server.

[0184] While this specification includes many details of specific implementations, these should not be construed as limiting the scope of the invention or claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of individual embodiments can also be implemented in combinations within a single embodiment. Conversely, various features described within the context of a single embodiment can also be implemented individually or in any suitable subcombination within multiple embodiments. Furthermore, even if features are described above as functioning in a particular combination and initially claimed as such, one or more features from the claimed combination may, in some cases, be removed from the combination, and the claimed combination may be directed towards a subcombination or a variation of a subcombination.

[0185] Similarly, while the drawings depict actions and the claims describe them in a specific order, it should not be understood that such actions must be performed in a specific or sequential order, or that all of the actions shown must be performed, in order to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0186] Specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. For example, the actions described in the claims may still yield the desired results even if performed in a different order. As an example, the processes shown in the accompanying drawings do not necessarily have to follow the specific order or sequence shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of Symbols]

[0187] 100 Sequence Generation Systems 102 Input Text Query 104 Context Sequence Generation System 106 Contextual Text Sequences 108 Natural Language Identifiers 108 Identifiers 110 Input Sequence Generation System 112 First Input Sequence 114 Language Model Neural Networks 114 Autoregressive Language Model Neural Networks 116 Output Sequence 116 First output text sequence 200 processes 302 Questions 304 Internet Search 306 Generator Sampling 308 times 310 Reward Model Scoring 312 Best Samples 314 Render 400 User Interfaces 402 query 404 First subsequence 406 Second subsequence 408 Third subsequence 506 Supervised Fine-Tuning (SFT) 508 Reward Model (RM) 600 User Interfaces 602 query 604 Candidate Response 606 Candidate Responses 608 selection elements 610 Selection Elements

Claims

1. A method performed by one or more computers, Steps include receiving an input text query, The steps include obtaining one or more first contextual text sequences and a natural language identifier for each of the first contextual text sequences, A step of generating a first input sequence, which includes the input text query, one or more first context text sequences, and the respective natural language identifiers for each of the one or more first context text sequences; (i) A first output text subsequence which is a response to the input text query, (ii) A second output text subsequence which is one of the respective natural language identifiers for the first context text sequence, (iii) A third output text subsequence which is the text from the first context text sequence identified by the natural language identifier in the second output text subsequence, To generate a first output text sequence comprising: processing the first input text sequence using an autoregressive language model neural network; The steps of providing at least the first output text subsequence and the third output text subsequence in response to the input text query. A method that includes [a certain feature].

2. The step of providing at least the first output text subsequence and the first context text sequence in response to the input text query is: The method according to claim 1, further comprising the step of providing the first output text subsequence, the second output text subsequence, and the third output text subsequence in response to the query.

3. A step of determining the source of the first context text sequence identified by the natural language identifier in the second output text subsequence from the second output text subsequence, The steps of providing a reference to the source of the first context text sequence in response to the query, The method according to claim 1 or 2, further comprising:

4. The steps include obtaining one or more second context text sequences and a natural language identifier for each of the second context text sequences, A step of generating a second input sequence including the input text query, one or more second context text sequences, and the respective natural language identifier for each of the one or more second context text sequences, (i) A fourth output text subsequence which is a response to the input text query, (ii) A fifth output text subsequence which is one of the respective natural language identifiers for the second context text sequence, (iii) A sixth output text subsequence which is the text from the second context text sequence identified by the natural language identifier in the fifth output text subsequence, To generate a second output text sequence comprising the steps of processing the second input text sequence using the autoregressive language model neural network, A step of generating a score for each output text sequence in a set that includes the first and second output text sequences, The steps include determining that the first output text sequence has a higher score than any other output text sequence in the set, In response to the determination that the first output text sequence has the highest score, the step of providing at least the first output text subsequence and the third output text subsequence in response to the input text query. The method according to any one of claims 1 to 3, further comprising:

5. The step of generating a score for each output text sequence in the set, which includes the first and second output text sequences, is: The method according to claim 4, further comprising the step of scoring each of the output text sequences using a trained reward model.

6. The first output sequence includes each token from the token vocabulary at each of a plurality of time steps, and the autoregressive neural network is configured to generate a score for each token in the vocabulary, conditional on the first input text sequence and any token in the output sequence at any time step prior to the time step in the first output sequence, and the step of generating the first output sequence is, The method according to any one of claims 1 to 5, comprising the step of selecting a token in a time step using the respective scores of the tokens in the vocabulary generated by the neural network for the time step.

7. The step of generating the first output sequence is In each time step in the second output text subsequence, following the first time step in the second output text subsequence, The steps include receiving the respective scores generated by the neural network in the aforementioned time step, A step of generating a constrained score distribution in which one of the natural language identifiers assigns a non-zero score only to the token immediately following the token already generated in the second output text subsequence, A step of sampling the tokens from the constrained score distribution at the time step. The method according to claim 6, comprising:

8. The step of generating the first output sequence is to place one or more first predetermined syntax tokens from the first output text sequence before the second output text subsequence, The steps include determining that, at a particular time step, one or more of the first predetermined syntactic tokens were selected in one or more time steps immediately preceding the particular time step, and accordingly determining that the particular time step is the first time step in the second output text subsequence, The steps include receiving the respective scores generated by the neural network at the aforementioned specific time step, The steps include: generating a constrained score distribution that assigns a non-zero score only to tokens that are the first token in one of the natural language identifiers, in response to the determination that the particular time step is the first time step in the second output text subsequence; A step of sampling the tokens from the constrained score distribution at the time step. The method according to claim 7, comprising:

9. The step of generating the first output sequence is In each time step in the third output text subsequence, following the first time step in the third output text subsequence, The steps include receiving the respective scores generated by the neural network in the aforementioned time step, A step of generating a constrained score distribution that assigns a non-zero score only to tokens immediately following tokens already generated in the third output text subsequence in the first context text sequence, which is identified by the natural language identifier in the second output text subsequence; A step of sampling the tokens from the constrained score distribution at the time step. The method according to any one of claims 1 to 8, comprising:

10. The third output text subsequence is preceded by one or more second predetermined syntactic tokens from the first output text sequence, and the step of generating the first output sequence is as follows: The steps include determining that in a second specific time step, one or more second predetermined syntactic tokens were selected in one or more time steps immediately preceding the second specific time step, and accordingly determining that the specific time step is the first time step in the third output text subsequence, The steps include receiving the respective scores generated by the neural network at the aforementioned specific time step, The steps include: generating a constrained score distribution that assigns non-zero scores only to tokens appearing in the first context text sequence identified by the natural language identifier in the second output text subsequence, in response to the determination that the particular time step is the first time step in the third output text subsequence; A step of sampling the tokens from the constrained score distribution at the time step. The method according to claim 9, comprising:

11. The step of obtaining one or more first contextual text sequences and a natural language identifier for each of the first contextual text sequences is: The steps include submitting a query derived from the aforementioned input text query to a search engine, The steps include obtaining one or more contextual documents from the search engine in response to the query, The steps of selecting one or more first context sequences from the one or more context documents and The method according to any one of claims 1 to 10, comprising:

12. The method according to claim 11, wherein the respective natural language identifier for each of the first context text sequences is the title of the context document from which the first context text sequence was selected.

13. The method according to any one of claims 1 to 12, wherein the neural network is pre-trained through unsupervised learning for the purpose of language modeling.

14. The method according to any one of claims 1 to 13, wherein the neural network is fine-tuned through supervised learning, reinforcement learning, or both.

15. One or more computers, When performed by the one or more computers, the one or more storage devices store instructions that cause the one or more computers to perform each of the operations described in any one of claims 1 to 14. A system that includes these features.

16. One or more computer-readable storage media that, when performed by one or more computers, store instructions causing the one or more computers to perform each of the operations of the method according to any one of claims 1 to 14.

Images