Evaluating output sequences using autoregressive language model neural networks

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for evaluating candidate output sequences using a language model neural network. Specifically, an autoregressive language model neural network is used to generate candidate output sequences. The same autoregressive language model neural network is used to evaluate the candidate output sequences to determine a rating score for each of one or more criteria. The rating score is then used to determine whether to provide the candidate output sequence.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 226,748, filed July 28, 2021. The disclosure of this prior application is considered part of the disclosure of this application and is incorporated by reference into the disclosure of this application.

[0002] This specification relates to using neural networks to process inputs to generate output sequences. [Background technology]

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, either another hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019 [Non-patent document 2] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020 [Non-patent document 3] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell and others. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020 [Non-patent document 4] Vaswani et al. “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. Summary of the Invention [Means for solving the problem]

[0005] Described herein is a system, implemented as a computer program on one or more computers at one or more locations, that uses an autoregressive language model neural network to generate output sequences and then uses the same autoregressive language model neural network to evaluate the output sequences, i.e., to determine whether the output sequences are suitable for providing as the output of the system or to select which output sequences to provide in response to a given request. Thus, the system uses the same autoregressive neural network both to generate the sequences and to "distinguish" between the generated sequences in order to determine which, if any, sequences to provide in response to a given request.

[0006] The subject matter described herein, in particular embodiments, can be implemented to realize one or more of the following advantages.

[0007] The described techniques allow an autoregressive language model neural network to be used as both a generator and a discriminator, i.e., to both generate an output sequence and to evaluate how well that output sequence satisfies one or more criteria.

[0008] Compared to having separate generator and classifier models, having a single model that can function as both a generator and a classifier allows candidate outputs to be accurately evaluated with minimal additional computational overhead compared to generating the candidates. That is, the system can evaluate generated sequences with much less additional latency and consuming much less additional memory than evaluating sequences using a separate generator model.

[0009] For example, in some cases, the language model neural network may be deployed on devices with constrained memory space, i.e., devices with significant constraints on available memory, or on devices where low decoding latency is required, such as edge devices such as mobile devices, smart speakers, or other Internet of Things (IoT) devices, or may be embedded in robots or other physical systems. In these cases, the system may perform evaluation of sequences generated by the neural network on-device without requiring additional data communication over a network and with minimal additional computational overhead, thereby improving the quality of output sequences provided in response to received requests. That is, because the language model neural network is used as both a generator and a classifier, the edge device may perform evaluation locally on-device without exceeding memory or latency constraints.

[0010] Furthermore, this single model can be used to filter task-specific fine-tuning data, and then this filtered data can be used to fine-tune the model to work well for a particular task. After fine-tuning, the quality of the output sequences produced by the language model neural network can be significantly improved because the language model has been fine-tuned on training data that has removed a significant amount of substandard outputs.

[0011] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below.

[0012] Other features, aspects, and advantages of the present subject matter will become apparent from the description, the drawings, and the claims. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram of an exemplary neural network system. [Figure 2] FIG. 1 is a diagram of an example of a benchmark evaluation system. [Figure 3] 1 is a flow diagram of an exemplary process for evaluating candidate output sequences. [Figure 4] FIG. 1 is a diagram of another example of a benchmark evaluation system. [Figure 5] 10 is a flow diagram of another exemplary process for evaluating candidate output sequences. DETAILED DESCRIPTION OF THE INVENTION

[0014] Like reference numbers and designations in the various drawings represent like elements.

[0015] 1 is a diagram of an exemplary neural network system 100. Neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0016] Neural network system 100 is a system that generates an output sequence 150 in response to a received request.

[0017] For example, system 100 can be an output sequence generation system that generates output sequences 150 unprompted, i.e., without conditioning on any user-specified contextual input. In these implementations, the output sequences generated by system 100 approximate samples of the distribution of training output sequences used during training of system 100.

[0018] As an example, system 100 may be a text generation system that generates text sequences, i.e., each output sequence 150 generated by system 100 is a sequence of text tokens from a vocabulary of text tokens that may include, for example, one or more of letters, subwords, words, punctuation marks, digits, or other symbols that occur in natural language text. For example, system 100 may generate a text sequence in response to a received request and provide the text sequence for presentation to a user.

[0019] As another example, system 100 may be an image generation system that generates images as a sequence of pixels, i.e., each output sequence 150 generated by system 100 is a sequence of color values ​​of the pixels in the output image arranged according to a specified order.

[0020] As another example, the system 100 may receive a context sequence 102 as part of a request and generate an output sequence 150 that is a response to the context sequence 102 .

[0021] As a particular example, system 100 may be part of a dialogue system, where context sequence 102 may include audio or text from the most recent conversational turn submitted by a user of the dialogue system during a dialogue, while output sequence 150 is the next turn in the dialogue, e.g., text or audio that is a response to the most recent conversational turn. Optionally, context sequence 102 may also include one or more historical conversational turns that occurred earlier in the dialogue.

[0022] As another specific example, system 100 may be part of a machine translation system, where context sequence 102 may include text written in a source language, while output sequence 150 is text written in a target language that is a translation of the source text into the target language.

[0023] As another particular example, system 100 may be part of a computer code generation system, where context sequence 102 may be a textual description of a desired piece of code or a computer code snippet written in a programming language, and output sequence 150 may be computer code, e.g., a computer program, that is a code fragment described by or follows context sequence 102.

[0024] More specifically, the system 100 receives a request for an output sequence 150 and, in response, generates one or more candidate output sequences 120 using an autoregressive language model neural network 110, i.e., an autoregressive language model neural network 110 that is either unconditioned or conditioned on a context sequence 102.

[0025] Each candidate output sequence 120 includes, at each of a plurality of positions, a respective token from a vocabulary of tokens. The vocabulary of tokens may include any of a variety of tokens representing textual or other symbols. For example, the vocabulary of tokens may include one or more of letters, subwords, words, punctuation marks, digits, or other symbols that appear in a corpus of natural language text.

[0026] The language model neural network 110 is referred to as an autoregressive neural network because the neural network 110 autoregressively generates an output sequence of tokens by generating each particular token in the output sequence conditioned on a current input sequence, which includes any tokens preceding that particular text token in the output sequence, i.e., tokens already generated for any previous positions in the output sequence preceding the particular position of that particular token. When the system 100 also receives a context sequence 102, the current input sequence can also include the context sequence 102. For example, when generating a token at any given position in the output sequence, the current input sequence can include the context sequence 102 and tokens at any previous positions preceding that given position in the output sequence. As a specific example, the current input sequence can include the context sequence 102, followed by a “sentinel” sequence representing a predetermined sequence of tokens that indicates to the neural network 110 that it should generate a response to the context sequence 102, followed by tokens at any previous positions preceding that given position in the output sequence. Optionally, within the current input sequence, subsequences can be separated by predefined tokens, for example the "<" and ">" tokens.

[0027] More specifically, to generate a particular token at a particular position in the candidate output sequence 120, the neural network 110 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of tokens. The neural network 110 may then use this score distribution to select a token from the vocabulary as the particular token. For example, the neural network 110 may greedily select the token with the highest score, or may sample the token from the distribution, e.g., using a kernel sampling technique or another sampling technique.

[0028] As a particular example, the language model neural network 110 may be an autoregressive Transformer-based neural network that includes (i) multiple attention blocks, each applying a self-attention operation, and (ii) an output sub-network that processes the output of the final attention block to generate a score distribution.

[0029] Neural network 110 can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; and Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0030] In general, however, a Transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates each hidden state, at least in part by applying self-attention, to generate a respective output hidden state for each input token. The input hidden state for the first attention block is the embedding of the input token in the input sequence, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0031] In this example, the output sub-network processes the output hidden state for the last input token in the input sequence, generated by the last attention block in the sequence, to generate a score distribution.

[0032] In some implementations, system 100 or another training system pre-trains language model neural network 110 in training data for a language modeling task, such as a task that requires predicting the next token following a current sequence of text tokens. As a particular example, language model neural network 110 can be pre-trained on a large text dataset, such as publicly available text from the Internet or another text corpus, based on a maximum likelihood objective.

[0033] In general, because neural network 110 is autoregressive, system 100 can generate multiple different candidate output sequences 120 using the same neural network 110 in response to the same request, for example, by using beam search decoding from the score distribution generated by neural network 110, by using a Sample-and-Rank decoding strategy, or by using another decoding strategy that exploits the autoregressive nature of the neural network.

[0034] When multiple candidate output sequences 120 are generated, the system 100 must determine which of the multiple sequences should be provided as the output sequence 150 in response to a request. Furthermore, even if only one output sequence is generated, some candidate output sequences 120 may not be suitable for provision as the final output sequence 150.

[0035] To determine how to respond to a given request, i.e., to determine which, if any, candidate output sequences 120 to provide in response to the request, system 100 maintains data specifying a set of one or more criteria ("output sequence criteria"), each measuring a corresponding characteristic of a given output sequence.

[0036] The set of criteria may include any of a variety of criteria that measure characteristics relevant to the quality of a given candidate output sequence in terms of the task that system 100 is configured to perform.

[0037] For example, the set of criteria may include a criterion corresponding to the sensibility of the output sequence. The sensibility criterion measures whether the model's response is conscious, i.e., whether a given candidate output sequence generated by the neural network 110 makes sense in the context of the current interaction, i.e., as a response to the context sequence 102, and is consistent with any information in the context sequence 102 or in previous sequences during the current interaction. Thus, if an output sequence is conscious, it satisfies this criterion.

[0038] As another example, the set of criteria may include a criterion corresponding to the specificity of the output sequence. The specificity criterion measures whether the model's response is specific, i.e., whether a given candidate output sequence generated by the neural network 110 is specific to the corresponding context sequence 102, rather than being a general response that could apply to any of a variety of context sequences. For example, if the context sequence 102 is "I like television" and the model responds "Me too," this response will receive a low score for specificity, e.g., a score of 0, because this response can be used in many different contexts. If the model responds "Me too. I like watching reality TV," this response will receive a higher score for specificity.

[0039] As another example, the set of criteria may include a criterion corresponding to the interestingness of an output sequence. The interestingness criterion measures whether a model's response is interesting, i.e., whether a given candidate output sequence generated by the neural network 110 would be interesting to the human who generated the corresponding context sequence 102. For example, a response to "How do you throw a ball?" could be "You can throw a ball by first picking it up and then tossing it." An alternative, more interesting answer could be "One way to toss a ball is to grasp it firmly with both hands, then swing your arms down and up again, extending your elbows, and then release the ball upward." The initial response, while meaningful and specific to the question, may not be found interesting or informative by the person asking the question. On the other hand, the alternative response provides more information that answers the question and therefore would score higher on the interestingness criterion than the initial response.

[0040] Other criteria may be included instead of or in addition to the above criteria, for example, suitability criteria, which measure the degree to which a given output sequence is suitable for being provided in response to a request.

[0041] The system 100 uses a criteria evaluation system 130 to generate a respective rating score 140 for each of one or more criteria in the set. The rating score 140 for a given criterion measures the degree to which the candidate output sequence 120 generated by the autoregressive language model neural network 110 satisfies that criterion.

[0042] In general, the benchmark evaluation system 130 uses the output from the language model neural network 110 to generate the rating scores 140 in a computationally efficient yet accurate manner.

[0043] Exemplary techniques for using neural network 110 to generate rating scores 140 for one or more criteria in a set are described below with reference to FIGS.

[0044] That is, rather than simply using a separately trained machine learning model to generate rating scores 140, system 130 leverages neural networks 110 to improve both the accuracy of rating scores 140 and the computational efficiency of generating rating scores 140.

[0045] After the rating score 140 is generated, the system 100 can use the rating score 140 to determine whether to provide a candidate output sequence 120 in response to a request in any of a variety of ways.

[0046] For example, in an implementation in which system 100 generates multiple candidate output sequences, system 100 may generate, for each candidate output sequence, a respective quality score from that candidate output sequence's respective rating scores 140 for one or more of the criteria in the set. For example, the quality score for a given candidate output sequence may be the sum or weighted sum of the rating scores 140 for one or more criteria.

[0047] The system 100 may then select the candidate output sequence 120 with the highest respective quality score as the sequence to provide in response to the request.

[0048] As another example, system 100 may maintain respective thresholds for at least a subset of the criteria. In this example, for each criterion in the subset, system 100 may determine, for a given candidate output sequence 120, whether the quality score of each of the given candidate output sequences for that criterion in the set satisfies, e.g., exceeds, the threshold for that criterion, and then determine not to provide the given candidate output sequence in response to the request (even though the given candidate output sequence would otherwise have the highest quality score) when the quality score of each of the given candidate output sequences for the first output criterion in the set does not satisfy the threshold.

[0049] In addition to or instead of using system 130 to determine how to respond to a request, system 100 can also use system 130 to filter task-specific fine-tuning data and then use this filtered data to fine-tune neural network 110 to perform well for a particular task.

[0050] Specifically, system 100 may obtain task-specific fine-tuning data including a set of training output sequences and, optionally, a corresponding context sequence for each training output sequence. System 130 may then generate a rating score for each training output sequence and then use the rating score to filter one or more training output sequences from the fine-tuning data. For example, system 130 may determine to filter out any training output sequences having a quality score below a threshold. As another example, system 130 may determine to filter out any training output sequences having a rating score below another threshold for each criterion in a specified subset of criteria. By filtering out these training output sequences, the system may improve the overall quality of the output sequences in the fine-tuning data, and thus, the quality of the output sequences generated by language model neural network 110 after fine-tuning may be significantly improved.

[0051] FIG. 2 is a diagram of an example of a benchmark evaluation system 130.

[0052] In the example of FIG. 2, the criteria evaluation system 130 includes a respective criteria engine 220A-N for each of one or more criteria.

[0053] Each criteria engine 220A-N receives a candidate output sequence 120 and generates a rating score 140 for the corresponding criteria. The rating score 140 represents the degree to which the candidate output sequence 120 generated by the autoregressive language model neural network 110 satisfies the criteria.

[0054] To generate the rating score 140, each criteria engine 220A-N receives the candidate output sequence 120 and generates an input sequence 230 based on the candidate output sequence 120 and a predetermined sequence of one or more tokens from the vocabulary that specifies the criteria, i.e., the sequence of tokens that indicates to the autoregressive neural network 110 which criteria are being evaluated. More specifically, before evaluating one or more criteria using the neural network 110, the system receives as input a respective sequence of tokens for each criterion. For example, the sequence of tokens may be a natural-language name or other identifier of the corresponding criterion; e.g., the word "specificity" may represent a specificity criterion, while the word "interestingness" may represent an interest criterion.

[0055] Specifically, the input sequence 230 for a given criterion includes the candidate output sequence 120 followed by one or more tokens that specify the output sequence criterion. When the system 100 also receives a context sequence 102, the input sequence 230 can also include the context sequence.

[0056] As a particular example, the input sequence 230 may be of the following form: <context> <sentinel> <response> <attribute name>, where "context" represents the context sequence 102, "sentinel" represents a predetermined sequence of tokens that indicates to the neural network 110 that it should generate a response to the context sequence 102, "response" is a candidate output sequence 120, and "attribute name" is a token that specifies the criteria.

[0057] The system 130 then processes the input sequence 230 using the neural network 110 to generate a respective score for each token in the vocabulary. Because the neural network 110 has already processed the context sequence 102 to generate the candidate output sequence 120, this process (also called "evaluating the classifier") simply processes one or more additional tokens "<attribute name>" appended to the end of the candidate output sequence 120, reusing the hidden states generated during the generation of the candidate output sequence 120. This contrasts with, and is significantly more computationally efficient than, using a separate neural network to evaluate the criteria, which would need to process all tokens in the context 102 and the candidate output sequence 120 in order to accurately score the candidate output sequence 102.

[0058] Thus, each engine 220A-N generates a different input sequence 230 and uses the neural network 110 to generate a corresponding score distribution across the vocabulary of tokens.

[0059] Each engine 220A-N then determines a rating score 140 from the scores of each of the tokens in the corresponding proper subset of the token's vocabulary.

[0060] That is, each criterion has a corresponding proper subset of one or more tokens in the vocabulary.

[0061] When each rating score has the same range, the corresponding true subsets may be the same.

[0062] When different rating scores have different ranges, the corresponding true subsets may be different. For example, when the rating scores range from 0 to 1, engines 220A-N may use a token's score of "1" as rating score 140. More generally, engines 220A-N of the system may calculate rating score 140 as equal to the weighted sum of each possible ground truth value of the rating scores, where the weight for each ground truth value is the score of the token representing that ground truth value in the vocabulary.

[0063] Thus, as shown in FIG. 2, the system 130 uses the neural network 110 to generate a rating score 140 for one or more criteria in a computationally efficient manner, i.e., generating the rating score 140 requires the neural network 110 to process only a small number of additional tokens per criterion.

[0064] Before using neural network 110 to generate rating scores, and after neural network 110 has been pre-trained, system 130 or another training system fine-tunes neural network 110, i.e., by repeatedly performing a training step on batches of training examples from a set of training data for fine-tuning, in order to cause neural network 110 to accurately generate rating scores.

[0065] Each training example includes (i) a training output sequence followed by (ii) a training input sequence that includes one or more tokens specifying a particular criterion from a set of criteria. Each training example also includes, for a corresponding output sequence criterion, a ground truth rating score that represents the degree to which the training output sequence satisfies the particular output sequence criterion. The ground truth rating score can be obtained, for example, as a result of a user manually labeling the output sequence or as the output of another automatic labeling system.

[0066] To perform the training step, the system, i.e., system 130 or another training system, obtains one or more batches of training examples. For example, the system may sample the batches from a larger set of training examples for the fine-tuning training process.

[0067] For each training example in the batch, the system uses an autoregressive language model neural network to process the training input sequence in the training example to generate a respective score for each token in the vocabulary.

[0068] The system then trains an autoregressive language model neural network to minimize, for each training example in the batch, a loss function that measures the error between (i) the score for each token in the vocabulary generated for that training example and (ii) a ground truth set of scores for tokens in the vocabulary that result in a ground truth rating score for that training example. For example, the loss function can be a negative log-likelihood loss or a cross-entropy loss.

[0069] Specifically, the system may calculate gradients of the loss function with respect to the parameters of the neural network 110, for example by backpropagation. The system may then update the parameters of the neural network 110 by applying an optimizer to the calculated gradients to generate updates, and then adding or subtracting the updates from the current values ​​of the parameters.

[0070] The system is able to perform this fine-tuning computationally efficiently because the loss is applied only to the error measured by the loss function, i.e., the predicted rating score, without applying the loss to any predictions at any previous positions in the training input sequence.

[0071] As described above, after fine-tuning is performed, system 130 or another training system may also use engines 210A-N to generate a filtered set of training data that is used to further fine-tune neural network 110.

[0072] 3 is a flow diagram of an exemplary process 300 for evaluating candidate output sequences. For convenience, process 300 is described as being performed by a system of one or more computers located at one or more locations. For example, a neural network system, such as neural network system 100 shown in FIG. 1 and suitably programmed in accordance with this specification, can perform process 300.

[0073] The system receives a request for an output sequence, step 302. In some implementations, the request includes a context sequence, and the request is for a response to the context sequence.

[0074] The system uses an autoregressive language model neural network to generate a first candidate output sequence including a plurality of tokens, each selected from a vocabulary of tokens (step 304). As described above, in some implementations, the system uses the neural network to generate multiple candidate output sequences, i.e., the first candidate output sequence as well as one or more additional candidate output sequences.

[0075] For each output sequence criterion in the set of one or more output sequence criteria, the system uses an autoregressive language model neural network to process the input sequence, including (i) a first candidate output sequence followed by (ii) one or more tokens that specify that output sequence criterion, to generate a respective score for each token in the vocabulary (step 306). When the request also specifies a conditioning sequence, the input sequence also includes the conditioning sequence.

[0076] For each output sequence criterion, the system determines a rating score for each of the first candidate output sequences from the scores of each of the tokens in the corresponding proper subset of the tokens' vocabulary, the rating score representing the degree to which the first candidate output sequence generated by the autoregressive language model neural network satisfies the output sequence criterion (step 308).

[0077] The scores may be generated in parallel, for example, by processing multiple input sequences using multiple copies of an autoregressive language model neural network, each comprising (i) a first candidate output sequence followed by (ii) one or more tokens specifying the respective output sequence criteria for obtaining the score. Such parallelization may allow rating scores to be determined more quickly, thereby enabling outputs based on the candidate output sequences to be provided more quickly.

[0078] As additional candidate output sequences are generated, the system again performs steps 306 and 308 to generate a respective rating score for each additional candidate output sequence.

[0079] The system then determines whether the first candidate output sequence should be provided in response to the request using the rating score of each of the first candidate output sequences with respect to one or more output sequence criteria (step 310). For example, the system may make this determination as described above with reference to FIG.

[0080] FIG. 4 is a diagram of another example of a benchmark evaluation system 130 .

[0081] In the example of FIG. 4, the criteria evaluation system 130 includes a classifier 460 that generates a respective rating score 140 for each of the one or more criteria.

[0082] Specifically, as shown in FIG. 2, the language model neural network 110 includes a first sub-network 430 and an output sub-network 450 .

[0083] The first sub-network 430 is configured to process an input sequence including multiple input tokens to generate a respective hidden state 440 for each input token.

[0084] The output sub-network 450 is configured to process each hidden state for the input token at the position immediately preceding the particular position to generate a respective score for each token in the vocabulary of tokens, and to use those respective scores to select one of the tokens in the vocabulary as the token at that particular position in the first candidate output sequence.

[0085] For example, as described above, neural network 110 can be a Transformer-based neural network that includes multiple attention blocks, each applying a self-attention mechanism. More specifically, in this example, the Transformer-based neural network includes a sequence of attention blocks, such that during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence and updates each hidden state, at least in part, by applying self-attention, to generate a respective output hidden state for each input token. The input hidden state for the first attention block is the embedding of the input token, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0086] In this example, the attention block is part of the first sub-network 410, and the hidden state 440 is the output hidden state produced by the last attention block in the sequence.

[0087] The output sub-network 450 may include, for example, one or more fully connected layers that process the hidden state 440 for the final input token in the input sequence to generate a respective score for each token in the vocabulary of tokens. As a particular example, the output sub-network 450 may include one or more linear neural network layers followed by a softmax output layer.

[0088] In the example of FIG. 4, to generate rating score 140, system 130 processes input sequence 420 using first sub-network 430 to generate a respective hidden state 440 for each input in input sequence 420.

[0089] Specifically, input sequence 420 includes a plurality of input tokens, including all tokens at all positions in candidate output sequence 120. That is, input sequence 420 includes the entire candidate output sequence 120. When system 100 also receives a context sequence 102, the input tokens also include tokens from that context sequence 102.

[0090] For example, the input sequence 420 may be of the following form: <context> <sentinel> <response>, where “context” represents the context sequence 102, “sentinel” represents a predetermined sequence of tokens that indicates to the neural network 110 that it should generate a response to the context sequence 102, and “response” is the candidate output sequence 120.

[0091] In many autoregressive neural networks 110 that generate variable-length output sequences, a given output sequence terminates when a predetermined end-of-sequence token is selected at a given time step. That is, once the end-of-sequence token is selected, the system 100 determines that the output sequence is the token at each time step preceding the given time step during the generation of the output sequence. In these implementations, the system 130 does not need to perform any additional processing to generate a hidden state for the input sequence 410, since the input sequence 410 has already been processed to select the end-of-sequence token that terminated the generation of the candidate output sequence 120.

[0092] The system 130 then processes one or more of the hidden states 440 using a classifier 460 to generate a respective rating score 140 for each of the one or more criteria. For example, the one or more hidden states 440 processed by the classifier 460 may be the respective hidden states for a token at a final position in the first candidate output sequence generated by the first sub-network. As another example, the one or more hidden states 440 processed by the classifier 460 may be the respective hidden states for a specified input token at a specified position among the input tokens, e.g., the first input token in the input sequence, generated by the first sub-network 430. As yet another example, the one or more hidden states 440 may include all hidden states 440, and the classifier 460 may be configured to average the hidden states 440.

[0093] The classifier 460 is a neural network that includes one or more classifier layers and is configured to process one or more hidden states using the one or more classifier layers to generate a respective rating score 140 for each of the one or more criteria. For example, the classifier layers may include one or more fully connected layers, with the final layer having a respective node, e.g., a logistic regression or sigmoid node, corresponding to each criterion, and each node configured to generate a respective rating score 140 for the corresponding criterion.

[0094] Thus, in the example of FIG. 4, system 130 can generate scores for all criteria by processing the hidden states (already generated as part of generating the candidate output sequences) using a small, computationally efficient classifier 460.

[0095] Before classifier 460 is used to generate rating scores, and after neural network 110 has been pre-trained, system 130 or another training system trains classifier 460 to accurately generate rating scores, i.e., by repeatedly performing a training step on batches of training examples from a set of training data for training.

[0096] To perform the training step, the system, i.e., system 130 or another training system, obtains one or more batches of training examples. For example, system 130 can sample the batches from a larger set of training examples for the fine-tuning training process.

[0097] For each training example in the batch, the system uses one or more classifier layers to process one or more of the respective hidden states generated by processing inputs including a plurality of input tokens including all tokens at all positions in the training output sequence in the training example by the first sub-network to generate a respective rating score for each output sequence criterion.

[0098] The system then trains one or more classifier layers to minimize, for each training example in the batch, a loss function that measures the error between (i) the respective rating score generated for that training example and (ii) the respective ground truth rating score for that training example.

[0099] Specifically, the system can calculate the gradient of the loss function with respect to the parameters of the classifier layer. The system can then update the parameters of the classifier layer by applying an optimizer to the calculated gradient to generate updates, and then adding or subtracting the updates from the current values ​​of the parameters.

[0100] The system can perform this training computationally efficiently because the classifier 460 has a relatively small number of parameters, i.e., relatively few compared to the neural network 110, and because the system keeps the input sub-network frozen while the classifier 460 is being trained.

[0101] As described above, after classifier 460 has been trained, system 130 or another training system may also use classifier 460 to generate a filtered set of training data that is used to further fine-tune neural network 110.

[0102] 3 is a flow diagram of another exemplary process 500 for evaluating candidate output sequences. For convenience, process 500 is described as being performed by a system of one or more computers located at one or more locations. For example, a neural network system, such as neural network system 100 shown in FIG. 1 and suitably programmed in accordance with this specification, can perform process 500.

[0103] The system receives a request for an output sequence, step 502. In some implementations, the request includes a context sequence, and the request is for a response to the context sequence.

[0104] The system uses an autoregressive language model neural network to generate a first candidate output sequence including a plurality of tokens, each selected from a vocabulary of tokens (step 504).

[0105] As described above, the language model neural network includes a first sub-network and an output sub-network.

[0106] The first sub-network is configured to process, at each particular position of the plurality of positions in the candidate output sequence, an input comprising a plurality of input tokens to generate a respective hidden state for each input token, the input tokens comprising tokens at each position preceding the particular position in the candidate output sequence and, optionally, tokens in the context sequence.

[0107] The output sub-network is configured, at each particular position of the plurality of positions, to process a respective hidden state for the input token at the position immediately preceding the particular position to generate a respective score for each token in the vocabulary of tokens, and to use those respective scores to select one of the tokens in the vocabulary as the token at that particular position in the first candidate output sequence.

[0108] As described above, in some implementations, the system uses a neural network to generate multiple candidate output sequences, i.e., to generate a first candidate output sequence as well as one or more additional candidate output sequences.

[0109] The system uses one or more classifier layers to process one or more of the respective hidden states generated by processing an input including all tokens at all positions in the first candidate output sequence by the first sub-network to generate a respective rating score for each criterion in the set of one or more criteria (step 506).

[0110] As additional candidate output sequences are generated, the system also performs step 506 to generate a respective rating score for each additional candidate output sequence.

[0111] The system then determines whether the first candidate output sequence should be provided in response to the request using the rating score of each of the first candidate output sequences with respect to one or more output sequence criteria (step 508). For example, the system may make this determination as described above with reference to FIG.

[0112] An "embedding," as used herein, is a vector of numeric values, eg, floating point or other types of numeric values, having a given dimension, eg, having a given number of values.

[0113] The self-attention block described above is a neural network layer that includes an attention mechanism that operates over the self-attention block input (or input derived from the layer input) to generate the self-attention block output. The self-attention mechanism may be causally masked, such that any given position in the input sequence does not attend to any positions after that given position in the input sequence (e.g., does not use data from any positions after that given position in the input sequence). Many different attention mechanisms are possible. Some examples of self-attention layers that include attention mechanisms are described in the following references: Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0114] In general, an attention mechanism maps a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed, for example, by a compatibility function between the query and the corresponding key, such as the dot product or scaled dot product.

[0115] In general, a self-attention mechanism is configured to associate different positions within the same sequence and determine a transformed version of the sequence as an output. For example, the attention layer input can comprise a vector for each element of the input sequence. These vectors provide input to the self-attention mechanism, which uses these vectors to determine a new representation of the same sequence for the attention layer output, which also comprises a vector for each element of the input sequence. The output of the self-attention mechanism can be used as the attention layer output, or it can be processed by one or more of a feedforward layer, a skip connection, or a normalization operation to provide the attention layer output.

[0116] In some implementations, the attention mechanism may be implemented using, for example, a matrix W Q A query transformation defined by, say, the matrix W K and the key transformation defined by, for example, the matrix W V Each of the value transformations defined by the following is applied to the attention layer input, which is the input data X to the attention layer, to obtain a query matrix Q=XW containing each query for each vector in the input sequence. Q , a key matrix K=XW containing the respective keys for each vector in the input sequence K, and a value matrix V=XW containing the respective values ​​for each vector in the input sequence. V These matrices are used to determine the attention sequence for output. For example, the attention mechanism can be a dot-product attention mechanism applied by applying each query vector to each key vector to determine a respective weight for each value vector, and then combining the value vectors using the respective weights to determine the self-attention layer output for each element of the input sequence. This self-attention layer output can be scaled by a scaling factor, e.g., the square root of the dimension of the query and key, to implement scaled dot-product attention. Thus, for example, the output of the attention mechanism can be scaled by a softmax

[0117]

number

[0118] where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism comprises an "additive attention" mechanism that computes the fitness function using a feedforward network with one hidden layer. The output of the attention mechanism can be further processed by one or more fully connected feedforward neural network layers.

[0119] The attention mechanism can implement multi-headed attention, i.e., it can apply multiple different attention mechanisms in parallel, and then combine, e.g., concatenate, the outputs of these attention mechanisms and, if necessary, apply a learned linear transformation to reduce the original dimensionality.

[0120] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers is configured to perform particular operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform those operations or actions. One or more computer programs are configured to perform particular operations or actions means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0121] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware containing the structures disclosed herein and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described herein can be implemented as one or more computer programs, e.g., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated, propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus.

[0122] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0123] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, such as in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple linked files, such as files containing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0124] The term "database" is used broadly herein to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized differently and accessed differently.

[0125] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same one or more computers.

[0126] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0127] A computer suitable for running a computer program can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0128] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0129] To enable user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can input to the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device being used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message in return from the user.

[0130] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units, e.g., for handling common computationally intensive portions of machine learning training workloads or production, e.g., inference, workloads.

[0131] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0132] Embodiments of the present subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the present subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0133] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server may send data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying the data to and receiving input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of a user interaction, may be received from the device at the server.

[0134] While this specification contains many specific details of implementations, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features that are described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination, and may even be initially claimed as such, one or more features from a claimed combination can, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0135] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0136] Specific embodiments of the present subject matter have been described above. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0137] 100 Neural Network Systems 102 Context sequence, candidate output sequence, context 110 Autoregressive Language Model Neural Network, Autoregressive Neural Network 120 candidate output series 130 Criteria Evaluation System 140 Rating Score 150 output series 220A~N Standard engine 230 Input Series 300 processes 410 First Subnetwork, Input Sequence 420 Input Series 430 First Subnetwork 440 Hidden 450 output subnetworks 460 Classifier 500 processes

Claims

1. 1. A method implemented by one or more computers, comprising: receiving a request for an output sequence; generating a first candidate output sequence using an autoregressive language model neural network, the first candidate output sequence comprising a plurality of tokens each selected from a vocabulary of tokens; For each output sequence criterion in the set of one or more output sequence criteria: using the autoregressive language model neural network to process an input sequence comprising (i) the first candidate output sequence followed by (ii) one or more tokens specifying the output sequence criteria to generate a respective score for each token in the vocabulary; determining a rating score for each of the first candidate output sequences from the respective scores of the tokens in the corresponding proper subset of the vocabulary of tokens, the rating score representing the degree to which the first candidate output sequence generated by the autoregressive language model neural network satisfies the output sequence criteria; providing the first candidate output sequence in response to the request using the respective rating scores of the first candidate output sequence with respect to the one or more output sequence criteria; A method comprising:

2. 2. The method of claim 1 , wherein the request is for an output sequence that follows a context sequence, and wherein generating the first candidate output sequence comprises conditioning the autoregressive language model neural network with an input comprising the context sequence, and wherein the input sequence further comprises the context sequence, the input sequence comprising (i) the first candidate output sequence followed by (ii) one or more tokens specifying the output sequence criteria.

3. generating one or more additional candidate output sequences using the autoregressive language model neural network; For each additional candidate output sequence, For each output sequence criterion in said set of one or more output sequence criteria: using the autoregressive language model neural network to process an input sequence comprising (i) the additional candidate output sequence followed by (ii) the one or more tokens specifying the output sequence criteria to generate a respective score for each token in the vocabulary; determining, from the respective scores of tokens in the corresponding subset of vocabulary of tokens, respective rating scores representing the degree to which the additional candidate output sequences generated by the autoregressive language model neural network satisfy the output sequence criteria; further comprising providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria, using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences to select one of the first candidate output sequence and the one or more additional candidate output sequences to provide in response to the request.

2. The method of claim 1, comprising:

4. using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences to select one of the first candidate output sequence and the one or more additional candidate output sequences to provide in response to the request, generating, for each said candidate output sequence, a respective quality score from said respective rating scores of said candidate output sequence for each of one or more output sequence criteria in said set; selecting the candidate output sequence having the highest respective quality score as the sequence to be provided in response to the request; 4. The method of claim 3, comprising:

5. providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria, determining whether the quality score of each of the first candidate output sequences for a first output criterion in the set satisfies a threshold; determining not to provide the first candidate output sequence in response to the request when the respective quality scores of the first candidate output sequence for the first output criterion in the set do not satisfy the threshold; 5. The method of claim 4, comprising:

6. 10. A method for training an autoregressive language model neural network according to claim 1, comprising: obtaining one or more batches of training examples, each training example comprising: a training input sequence comprising (i) a training output sequence followed by (ii) one or more tokens specifying a particular output sequence criterion from said set of output sequence criteria; and a ground truth rating score for the corresponding output sequence criterion that indicates the degree to which the training output sequence satisfies the particular output sequence criterion; and For each training example in the batch, using the autoregressive language model neural network to process the training input sequences in the training examples to generate a respective score for each token in the vocabulary; training the autoregressive language model neural network to minimize, for each training example in the batch, a loss function that measures the error between (i) the respective scores of the tokens in the vocabulary generated for the training example and (ii) a ground truth set of scores for the tokens in the vocabulary, resulting in the ground truth rating score for the training example; A method comprising:

7. training the autoregressive language model neural network to minimize a loss function, applying a loss only to the error without applying a loss to any previous positions in the training input sequence.

7. The method of claim 6, comprising:

8. 8. The method of claim 6 or claim 7, further comprising, prior to the step of training the autoregressive language model neural network to minimize a loss function, training the autoregressive language model neural network on a language modeling task requiring predicting a next token in a sequence of tokens given a previous token in the sequence.

9. 1. A method implemented by one or more computers, comprising: receiving a request for an output sequence; generating a first candidate output sequence using an autoregressive language model neural network, the first candidate output sequence comprising a respective token at each of a plurality of positions, the autoregressive language model neural network a first sub-network, at each particular location of the plurality of locations, a first sub-network configured to process an input comprising a plurality of input tokens to generate a respective hidden state for each said input token, said input tokens comprising said token at each position preceding said particular position in said first candidate output sequence; and an output sub-network, at each particular location of the plurality of locations, processing the respective hidden states for the input tokens at a position immediately preceding the particular position to generate a respective score for each token in a vocabulary of tokens; using the respective scores to select one of the tokens in the vocabulary as the token at the particular position in the first candidate output sequence. The output subnetwork is configured as follows: and processing, using one or more classifier layers, one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens by the first sub-network, including all of the tokens at all of the positions in the first candidate output sequence, to generate, for each output sequence criterion in a set of one or more output sequence criteria, a respective rating score representing the degree to which the first candidate output sequence generated by the autoregressive language model neural network satisfies the output sequence criterion; providing the first candidate output sequence in response to the request using the respective rating scores of the first candidate output sequence with respect to the one or more output sequence criteria; A method comprising:

10. 10. The method of claim 9, wherein the request is for an output sequence that follows a context sequence, and wherein generating the first candidate output sequence includes conditioning the autoregressive language model neural network with input comprising the context sequence, and wherein for each particular position, the input tokens further comprise tokens from the context sequence.

11. generating one or more additional candidate output sequences using the autoregressive language model neural network; generating, for each additional candidate output sequence, a respective rating score for each output sequence criterion in the set of one or more output sequence criteria that represents the degree to which the additional candidate output sequence generated by the autoregressive language model neural network satisfies the output sequence criterion; further comprising providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria, using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences to select one of the first candidate output sequence and the one or more additional candidate output sequences to provide in response to the request.

10. The method of claim 9, comprising:

12. using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences to select one of the first candidate output sequence and the one or more additional candidate output sequences to provide in response to the request, generating, for each said candidate output sequence, a respective quality score from said respective rating scores of said candidate output sequence for each of one or more output sequence criteria in said set; selecting the candidate output sequence having the highest respective quality score as the sequence to be provided in response to the request; 12. The method of claim 11, comprising:

13. providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria, determining whether a quality score of each of the first candidate output sequences for a first output criterion in the set satisfies a threshold; determining not to provide the first candidate output sequence in response to the request when the respective quality scores of the first candidate output sequence for the first output criterion in the set do not satisfy the threshold; 10. The method of claim 9, comprising:

14. processing one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens including all of the tokens at all of the positions in the first candidate output sequence by the first sub-network, processing the respective hidden states for the tokens in their final positions in the first candidate output sequence produced by the first sub-network; 10. The method of claim 9, comprising:

15. processing one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens including all of the tokens at all of the positions in the first candidate output sequence by the first sub-network, processing the respective hidden states generated by the first sub-network for a specified input token at a specified position among the input tokens; 10. The method of claim 9, comprising:

16. During said generating of said first candidate output sequence, for one or more designated positions prior to a final position in said first candidate output sequence: processing one or more of the respective hidden states generated by the first sub-network at the designated position in the first candidate output sequence using the one or more classifier layers to generate, for each output sequence criterion in a set of one or more output sequence criteria, a respective rating score that represents the degree to which the portion of the first candidate output sequence generated at the designated position satisfies the output sequence criterion; determining whether (i) the first candidate output sequence should continue to be generated past the designated location, or (ii) whether any portion of the first candidate output sequence should not be provided in response to the request based on the respective rating scores for the output sequence criteria in the set generated at the designated location; 10. The method of claim 9, further comprising:

17. 10. A method for training an autoregressive language model neural network according to claim 9, comprising: obtaining one or more batches of training examples, each training example comprising: the training output sequence, and a respective ground truth rating score for each of one or more of the output sequence criteria, the ground truth rating score representing the degree to which the training output sequence satisfies the particular output sequence criterion; and For each training example in the batch, processing, using the one or more classifier layers, one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens including all tokens at all positions in the training output sequence within the training examples by a first sub-network to generate a respective rating score for each output sequence criterion; training the one or more classifier layers to minimize, for each training example in the batch, a loss function that measures the error between (i) the respective rating score generated for the training example and (ii) the respective ground truth rating score for the training example; A method comprising:

18. training the one or more classifier layers to minimize a loss function, keeping an input sub-network frozen during the training of the one or more classifier layers; 18. The method of claim 17, comprising:

19. 18. The method of claim 17, further comprising, prior to training the one or more classifier layers, training the autoregressive language model neural network on a language modeling task requiring predicting a next token in a sequence of tokens given a previous token in the sequence.

20. The method of claim 1 , wherein the output sequence is a text sequence and the vocabulary of tokens comprises a plurality of text tokens.

21. The method of claim 2 or claim 10, wherein the context sequence is a text sequence.

22. one or more computers; one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of claim 1; A system comprising:

23. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of claim 1.

Images