Evaluating the output sequence using an autoregressive language model neural network.

Abstract

To provide a method, system, and apparatus including a computer program encoded on a computer storage medium for evaluating candidate output sequences using a language model neural network. [Solution] An autoregressive language model neural network is used to generate candidate output sequences. The same autoregressive language model neural network is used to evaluate the candidate output sequences and determine a rating score for each of one or more criteria. The rating scores are then used to determine whether or not to provide the candidate output sequences.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 226,748, filed Jul. 28, 2021. The disclosure of this prior application is considered a part of the disclosure of this application and is incorporated herein by reference.

[0002] This specification relates to using a neural network to process an input to generate an output sequence.

Background Art

[0003] A neural network is a machine - learning model that uses one or more layers of non - linear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of each set of parameters.

Prior Art Documents

Non - Patent Documents

[0004]

Non - Patent Document 1

Non - Patent Document 2

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations, which uses an autoregressive language model neural network to generate output sequences and then uses the same autoregressive language model neural network to evaluate those output sequences, that is, to determine whether the output sequences are suitable to be provided as output of the system, or to select which output sequences should be provided in response to a given request. Thus, in this system, the same autoregressive neural network is used both for generating sequences and for "identifying" the generated sequences from each other in order to determine which sequences should be provided in response to a given request, if sequences exist.

[0006] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages:

[0007] The techniques described make it possible to use autoregressive language model neural networks as both generators and discriminators, that is, for both generating output sequences and evaluating how well those output sequences satisfy one or more criteria.

[0008] Having a single model that can function as both a generator and a discriminator, compared to having separate generator and discriminator models, allows for accurate evaluation of candidate outputs with minimal additional computational overhead compared to generating candidates. In other words, this system can evaluate generated sequences with much less additional latency and consume much less additional memory compared to evaluating sequences using separate generator models.

[0009] For example, in some cases, a language model neural network can be deployed on a device with a constrained memory space, i.e., a significant constraint on available memory; or on an edge device such as a mobile device, smart speaker, or other Internet of Things (IoT) device, where low decoding latency is required; or embedded in a robot or other physical system. In these cases, the system can perform the evaluation of sequences generated on-device by the neural network on-device, without requiring additional data communication over the network, and with minimal additional computational overhead, thereby improving the quality of the output sequences provided in response to received requests. That is, since the language model neural network is used as both a generator and a discriminator, the edge device can perform the evaluation locally on-device without exceeding memory or latency constraints.

[0010] Furthermore, this single model can be used to filter task-specific fine-tuning data, and then this filtered data can be used to fine-tune the model to work well for a particular task. After fine-tuning, the quality of the output sequences produced by the language model neural network can be significantly improved because the language model has been fine-tuned on the training data, which has had a considerable amount of substandard output removed.

[0011] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description.

[0012] Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0013] [Figure 1] It is a diagram of an exemplary neural network system. [Figure 2] It is a diagram of an example of a reference evaluation system. [Figure 3] It is a flowchart of an exemplary process for evaluating a candidate output sequence. [Figure 4] It is a diagram of another example of a reference evaluation system. [Figure 5] It is a flowchart of another exemplary process for evaluating a candidate output sequence.

Best Mode for Carrying Out the Invention

[0014] Like reference numerals and names in the various drawings represent like elements.

[0015] FIG. 1 is a diagram of an exemplary neural network system 100. The neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations that can implement the systems, components, and techniques described below.

[0016] The neural network system 100 is a system that generates an output sequence 150 in response to a received request.

[0017] For example, the system 100 can be an output sequence generation system that generates the output sequence 150 without being given a prompt, that is, without conditioning on any user-specified context input. In these implementations, the output sequence generated by the system 100 approximates a sample of the distribution of the training output sequences used during the training of the system 100.

[0018] As an example, system 100 can be a text generation system that generates a text sequence, that is, each output sequence 150 generated by system 100 is a sequence of text tokens from a vocabulary of text tokens that includes one or more of, for example, characters, subwords, words, punctuation marks, numbers, or other symbols that appear in natural language text. For example, system 100 can generate a text sequence in response to a received request and provide that text sequence for presentation to a user.

[0019] As another example, system 100 can be an image generation system that generates an image as a sequence of pixels, that is, each output sequence 150 generated by system 100 is a sequence of color values of pixels in an output image arranged in a specified order.

[0020] As another example, system 100 can receive a context sequence 102 as part of a request and generate an output sequence 150 that is a response to that context sequence 102.

[0021] As a particular example, system 100 can be part of a dialogue system, where the context sequence 102 can include audio or text from the most recent conversation turn submitted by a user of the dialogue system during a conversation, while the output sequence 150 is text or audio that is a response to the next turn in the conversation, for example, the most recent conversation turn. Optionally, the context sequence 102 can also include one or more historical conversation turns that occurred earlier in the conversation.

[0022] As another particular example, system 100 can be part of a machine translation system, where the context sequence 102 can include text written in a source language, while the output sequence 150 is text written in a target language that is a translation of the source text into the target language.

[0023] As another specific example, system 100 may be part of a computer code generation system, context sequence 102 may be a textual description of a desired code snippet or a computer code snippet written in a programming language, and output sequence 150 may be computer code, for example, a computer program, which may be the code snippet described by context sequence 102 or a code snippet that conforms to context sequence 102.

[0024] More specifically, system 100 receives a request for an output sequence 150 and, in response, generates one or more candidate output sequences 120 using an autoregressive language model neural network 110, i.e., an autoregressive language model neural network 110 that is either unconditioned or conditioned by the context sequence 102.

[0025] Each candidate output sequence 120 contains, at each of its multiple positions, a token from the token vocabulary. The token vocabulary can include any of a variety of tokens that represent text symbols or other symbols. For example, the token vocabulary can include one or more of the following symbols that appear in a corpus of natural language text: letters, subwords, words, punctuation marks, numbers, or other symbols.

[0026] The language model neural network 110 is called an autoregressive neural network because it autoregressively generates an output sequence of tokens by generating each specific token in the output sequence conditioned by the current input sequence, which includes any tokens that precede that particular text token in the output sequence, i.e., tokens that have already been generated for any preceding position in the output sequence that precedes that particular token at a particular location. The current input sequence may also include the context sequence 102 when the system 100 also receives the context sequence 102. For example, when generating a token at any given position in the output sequence, the current input sequence may include the context sequence 102 and tokens at any preceding position that precedes that given position in the output sequence. As a particular example, the current input sequence may include the context sequence 102, a “sentinel” sequence representing a given sequence of tokens that indicates to the neural network 110 that a response to the context sequence 102 should be generated, and tokens at any preceding position that precedes that given position in the output sequence. Optionally, subsequences within the current input sequence can be separated by predetermined tokens, such as the "<" and ">" tokens.

[0027] More specifically, in order to generate a specific token at a particular position in the candidate output sequence 120, the neural network 110 can process the current input sequence to generate a score distribution, such as a probability distribution, which assigns a score, such as a probability, to each token in the token vocabulary. The neural network 110 can then use this score distribution to select a token from the vocabulary as that specific token. For example, the neural network 110 could greedily select the token with the highest score, or it could sample tokens from the distribution using, for example, a nuclear sampling technique or another sampling technique.

[0028] As a specific example, the language model neural network 110 can be an autoregressive Transformer-based neural network that includes (i) a number of attention blocks, each to which a self-attention operation is applied, and (ii) an output subnetwork that processes the output of the final attention block to generate a score distribution.

[0029] Neural networks 110 can have any of the diverse Transformer-based neural network architectures. Examples of such architectures are described in the following literature: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0030] However, generally, Transformer-based neural networks include a sequence of attention blocks, and during the processing of a given input sequence, each attention block in the sequence receives its respective input hidden state for each input token in that given input sequence. The attention block then updates each hidden state by applying self-attention to at least some of them to generate its respective output hidden state for each input token. The input hidden state for a first attention block is the embedding of the input tokens in the input sequence, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0031] In this example, the output subnetwork processes the output hidden states for the last input token in the input sequence, which are generated by the last attention block in the sequence, to produce a score distribution.

[0032] In some implementations, system 100 or another training system pre-trains the language model neural network 110 on training data for a language modeling task, such as a task that requires predicting the next token following the current sequence of text tokens, given the current sequence of text tokens. In a specific example, the language model neural network 110 can be pre-trained on a large text dataset, such as publicly available text from the internet or another text corpus, based on the maximum likelihood objective.

[0033] Generally, since the neural network 110 is autoregressive, the system 100 can generate multiple different candidate output sequences 120 in response to the same request using the same neural network 110, for example by using beam search decoding from the score distribution generated by the neural network 110, by using a sample-and-rank decoding strategy, or by using another decoding strategy that leverages the autoregressive nature of the neural network.

[0034] When multiple candidate output sequences 120 are generated, the system 100 needs to decide which of the multiple sequences should be provided as the output sequence 150 in response to the request. Furthermore, even if only one output sequence is generated, some of the candidate output sequences 120 are not suitable to be provided as the final output sequence 150.

[0035] In order to determine how to respond to a given request, that is, to determine which candidate output sequence 120 to provide in response to the request when there are candidate output sequences 120, the system 100 maintains data specifying a set of one or more criteria ("output sequence criteria") that each measure the corresponding characteristics of a given output sequence.

[0036] The set of criteria may include any of a variety of criteria that measure characteristics relevant to the quality of a given candidate output sequence in relation to the task that the system 100 is configured to perform.

[0037] For example, the set of criteria may include a criterion corresponding to the sensibleness of the output sequence. The sensibleness criterion measures whether the model's response is sensible, that is, whether a given candidate output sequence generated by the neural network 110 makes sense in the context of the current interaction, i.e., as a response to the context sequence 102, and is consistent with any information in the context sequence 102 or previous sequences during the current interaction. Therefore, if an output sequence is sensible, it satisfies this criterion.

[0038] As another example, the set of criteria could include a criterion corresponding to the specificity of the output sequence. The specificity criterion measures whether the model's response is specific, that is, whether a given candidate output sequence generated by the neural network 110 is specific to the corresponding context sequence 102, rather than being a general response that could apply to any of a variety of context sequences. For example, if the context sequence 102 is "I like television" and the model responds "Me too," this response would receive a low score for specificity, e.g., a score of 0, because this response could be used in many different contexts. If the model responds "Me too. I like watching reality shows," this response would receive a higher score for specificity.

[0039] As another example, a set of criteria could include a criterion corresponding to the interestingness of the output sequence. The interestness criterion measures whether the model's response is interesting, i.e., whether a given candidate output sequence generated by the neural network 110 is interesting to the person who generated the corresponding context sequence 102. For example, a possible response to "How do you throw a ball?" is "You can throw a ball by first picking it up and then throwing it." A more interesting alternative answer might be, "One way to toss a ball is to firmly grasp it with both hands, then swing your arms down and up again, extending your elbows, and then release the ball upwards." The first response makes sense and is specific to the question, but it may not be considered interesting or informative to the person who asked the question. The alternative response, on the other hand, provides further information that answers the question and therefore would score higher on the interestness criterion than the first response.

[0040] In place of or in addition to the above criteria, other criteria may be included. For example, other criteria may include a suitability criterion that measures the degree to which a given output sequence is suitable to be provided in response to a request.

[0041] System 100 uses a criterion evaluation system 130 to generate a rating score 140 for each of one or more criteria in the set. The rating score 140 for a given criterion measures the extent to which the candidate output sequence 120 generated by the autoregressive language model neural network 110 satisfies that criterion.

[0042] Generally, the standard evaluation system 130 uses the output from the language model neural network 110 to generate a rating score 140 computationally efficient but accurate.

[0043] An exemplary technique for generating rating scores 140 for one or more criteria in a set using a neural network 110 is described below with reference to Figures 2 to 5.

[0044] In other words, instead of simply using a separately trained machine learning model to generate a rating score of 140, system 130 leverages the neural network 110 to improve both the accuracy of the rating score 140 and the computational efficiency of generating the rating score 140.

[0045] After the rating score 140 is generated, the system 100 can use the rating score 140 to determine, in response to the request, whether to provide a candidate output sequence 120, in one of several ways.

[0046] For example, in an implementation where system 100 generates multiple candidate output sequences, system 100 can generate a quality score for each candidate output sequence from its respective rating scores 140 for one or more criteria within the set. For example, the quality score of a given candidate output sequence can be the sum or weighted sum of the rating scores 140 for one or more criteria.

[0047] Next, the system 100 can select candidate output sequences 120 having the highest respective quality scores as sequences to be provided in response to the request.

[0048] As another example, system 100 may maintain a threshold for each of the criteria for at least a subset of the criteria. In this example, for each criterion in the subset, system 100 may determine, for a given candidate output sequence 120, whether the quality score of each of the given candidate output sequences for that criterion in the set satisfies, for example, exceeds, the threshold for that criterion, and then decide not to provide the given candidate output sequence in response to the request if the quality score of each of the given candidate output sequences for a first output criterion in the set does not satisfy the threshold (even if the given candidate output sequence would normally have the highest quality score).

[0049] In addition to using system 130 to determine how to respond to a request, or alternatively, system 100 can also use system 130 to filter task-specific fine-tuning data, and then use this filtered data to fine-tune the neural network 110 to work well for a particular task.

[0050] Specifically, system 100 can obtain task-specific fine-tuning data, which includes a set of training output sequences and, optionally, a corresponding context sequence for each training output sequence. System 130 can then generate a rating score for each training output sequence and use the rating scores to filter one or more training output sequences from the fine-tuning data. For example, system 130 may decide to filter out any training output sequences with a quality score lower than a threshold. As another example, system 130 may decide to filter out any training output sequences with a rating score below another threshold for each criterion within a specified subset of criteria. By filtering out these training output sequences, the system improves the overall quality of the output sequences in the fine-tuning data, and therefore, the quality of the output sequences generated by the language model neural network 110 after fine-tuning may be significantly improved.

[0051] Figure 2 is a diagram of an example of the standard evaluation system 130.

[0052] In the example shown in Figure 2, the standard evaluation system 130 includes each of the standard engines 220A to N for each of one or more standards.

[0053] Each criterion engine 220A~N receives a candidate output sequence 120 and generates a rating score 140 for the corresponding criterion. The rating score 140 represents the degree to which the candidate output sequence 120 generated by the autoregressive language model neural network 110 satisfies the criterion.

[0054] To generate a rating score 140, each criterion engine 220A-N receives a candidate output sequence 120 and generates an input sequence 230 based on the candidate output sequence 120 and a predetermined sequence of one or more tokens from a vocabulary that specify the criterion, i.e., a sequence of tokens that explicitly tells the autoregressive neural network 110 which criterion is being evaluated. More specifically, before evaluating one or more criteria using the neural network 110, the system receives a sequence of tokens for each criterion as input. For example, the sequence of tokens could be the natural language name or other identifier of the corresponding criterion; for example, the word "specificity" could represent the specificity criterion, while the word "interest" could represent the interest criterion.

[0055] Specifically, the input sequence 230 for a given criterion includes a candidate output sequence 120, followed by one or more tokens specifying the output sequence criterion. When the system 100 also receives a context sequence 102, the input sequence 230 may also include that context sequence.

[0056] As a specific example, input sequence 230 can be in the following format: <Context> <Sentinel> <Response> <Attribute Name>, However, "context" represents the context sequence 102, "sentinel" represents a predetermined sequence of tokens that indicate to the neural network 110 that it should generate a response to the context sequence 102, "response" is the candidate output sequence 120, and "attribute name" is a token that specifies the criterion.

[0057] Next, system 130 processes the input sequence 230 using the neural network 110 to generate a score for each token in the vocabulary. Since the neural network 110 has already processed the context sequence 102 to generate the candidate output sequence 120 (also called "classifier evaluation"), this process simply involves processing one or more additional tokens "<attribute name>" appended to the candidate output sequence 120, while reusing the hidden states generated during the generation of the candidate output sequence 120. This is significantly more computationally efficient than using a separate neural network for criterion evaluation, because a separate neural network would need to process the context 102 and all tokens in the candidate output sequence 120 in order to accurately score the candidate output sequence 102.

[0058] Therefore, each engine 220A~N generates a different input sequence 230 and uses the neural network 110 to generate a corresponding score distribution across the token vocabulary.

[0059] Next, each engine 220A-N determines a rating score of 140 from the scores of each token in the corresponding proper subset of the token's vocabulary.

[0060] In other words, each criterion has a corresponding proper subset consisting of one or more tokens within the vocabulary.

[0061] When each rating score has the same range, the corresponding proper subsets can be the same.

[0062] When different rating scores have different ranges, the corresponding proper subsets can be different. For example, when the range of rating scores is from 0 to 1, engines 220A~N can use a token score of "1" as a rating score of 140. More generally, engines 220A~N of the system can calculate a rating score of 140 as equal to the weighted sum of each possible ground truth value of the rating score, where the weight for each ground truth value is the score of the token representing that ground truth value in the vocabulary.

[0063] Therefore, as shown in Figure 2, the system 130 uses the neural network 110 to computationally generate a rating score 140 for one or more criteria, because generating the rating score 140 requires the neural network 110 to process only a small number of additional tokens per criterion.

[0064] Before generating a rating score using the neural network 110, and after the neural network 110 has been pre-trained, system 130 or another training system fine-tunes the neural network 110 to cause it to accurately generate a rating score, that is, by repeatedly performing the training step on batches of training examples from a set of training data for fine-tuning.

[0065] Each training example includes a training input sequence containing (i) a training output sequence followed by (ii) one or more tokens specifying a particular criterion from a set of criteria. Each training example also includes a ground truth rating score for the corresponding output sequence criterion, representing the extent to which the training output sequence satisfies that particular output sequence criterion. The ground truth rating score can be obtained, for example, as a result of a user manually labeling the output sequences, or as output from another automated labeling system.

[0066] To carry out the training steps, the system, i.e., system 130 or another training system, acquires a batch of one or more training examples. For example, the system may sample a batch from a larger set of training examples for a fine-tuning training process.

[0067] For each training example in a batch, the system uses an autoregressive language model neural network to process the training input sequence within the training example and generate a score for each token in the vocabulary.

[0068] Next, for each training example in a batch, the system trains an autoregressive language model neural network to minimize a loss function that measures the error between (i) the score of each token in the vocabulary generated for that training example and (ii) the ground truth set of scores for the tokens in the vocabulary that yields the ground truth rating score for that training example. For example, the loss function can be a negative log-likelihood loss or a cross-entropy loss.

[0069] Specifically, the system can calculate the gradient of the loss function with respect to the parameters of the neural network 110, for example, by backpropagation. The system can then update the parameters of the neural network 110 by applying an optimizer to the calculated gradient to generate an update, and then adding the update to the current value of the parameter or subtracting the update from the current value of the parameter.

[0070] The system can perform this fine-tuning computationally efficient because the loss is applied only to the error measured by the loss function, i.e., only to the predicted rating score, and not to any prediction at any previous position in the training input sequence.

[0071] As described above, after fine-tuning has been performed, system 130 or another training system may also use engines 210A-N to generate a filtered set of training data used to further fine-tune the neural network 110.

[0072] Figure 3 is a flowchart of an exemplary process 300 for evaluating candidate output sequences. For convenience, process 300 is described as being performed by a system consisting of one or more computers located in one or more locations. For example, a neural network system, such as the neural network system 100 shown in Figure 1 and appropriately programmed according to this specification, can perform process 300.

[0073] The system receives a request for an output sequence (step 302). In some implementations, the request includes a context sequence, and the request seeks a response to that context sequence.

[0074] The system uses an autoregressive language model neural network to generate a first candidate output sequence containing multiple tokens, each selected from the token vocabulary (step 304). As described above, in some implementations, the system uses a neural network to generate multiple candidate output sequences, i.e., one or more additional candidate output sequences in addition to the first candidate output sequence.

[0075] For each output sequence criterion in a set of one or more output sequence criteria, the system uses an autoregressive language model neural network to process an input sequence containing (i) a first candidate output sequence followed by (ii) one or more tokens specifying that output sequence criterion, and generates a score for each token in the vocabulary (step 306). If the request also specifies a conditional sequence, the input sequence also includes that conditional sequence.

[0076] For each output sequence criterion, the system determines a rating score for each first candidate output sequence, which represents the extent to which the first candidate output sequence generated by the autoregressive language model neural network satisfies that output sequence criterion, from the scores of each token in the corresponding proper subset of the token's vocabulary (step 308).

[0077] The scores can be generated in parallel, for example, by processing multiple input sequences using multiple copies of an autoregressive language model neural network, each having (i) a first candidate output sequence followed by (ii) one or more tokens specifying the respective output sequence criteria for obtaining the score. Such parallelization may allow for faster determination of rating scores, thereby enabling faster provision of outputs based on candidate output sequences.

[0078] When additional candidate output sequences are generated, the system again performs steps 306 and 308 to generate a rating score for each additional candidate output sequence.

[0079] Next, the system determines whether to provide the first candidate output sequence in response to the request, using the respective rating scores of the first candidate output sequence for one or more output sequence criteria (step 310). For example, the system can make this determination as described above with reference to Figure 1.

[0080] Figure 4 is a diagram of another example of the standard evaluation system 130.

[0081] In the example shown in Figure 4, the criteria evaluation system 130 includes a classifier 460 that generates a rating score 140 for each of one or more criteria.

[0082] Specifically, as shown in Figure 2, the language model neural network 110 includes a first subnetwork 430 and an output subnetwork 450.

[0083] The first subnetwork 430 is configured to process an input sequence containing multiple input tokens and generate a corresponding hidden state 440 for each input token.

[0084] The output subnetwork 450 is configured to process the hidden state of each input token at the position immediately preceding a particular position to generate a score for each token in the token vocabulary, and to use those scores to select one of the tokens in the vocabulary as the token at that particular position in the first candidate output sequence.

[0085] For example, as described above, the neural network 110 can be a Transformer-based neural network that includes multiple attention blocks, each to which a self-attention mechanism is applied. More specifically, in this example, the Transformer-based neural network includes a sequence of attention blocks, and during the processing of a given input sequence, each attention block in the sequence receives its respective input hidden state for each input token in that given input sequence, updates each hidden state by applying self-attention to at least part of it, and generates its respective output hidden state for each input token. The input hidden state for the first attention block is the embedding of the input token, and the input hidden state for each subsequent attention block is the output hidden state generated by the preceding attention block.

[0086] In this example, the attention block is part of the first subnetwork 410, and the hidden state 440 is an output hidden state generated by the last attention block in the sequence.

[0087] The output subnetwork 450 may include, for example, one or more fully connected layers that process the hidden state 440 for the last input token in the input sequence to generate a score for each token in the token vocabulary. In a specific example, the output subnetwork 450 may include one or more linear neural network layers followed by a softmax output layer.

[0088] In the example in Figure 4, to generate a rating score 140, the system 130 processes the input sequence 420 using the first subnetwork 430 to generate a corresponding hidden state 440 for each input in the input sequence 420.

[0089] Specifically, the input sequence 420 includes multiple input tokens, each containing all tokens at all positions within the candidate output sequence 120. In other words, the input sequence 420 includes the entire candidate output sequence 120. When the system 100 also receives the context sequence 102, the input tokens also include tokens from that context sequence 102.

[0090] For example, input sequence 420 can be in the following format: <Context> <Sentinel> <Response>, However, "context" represents the context sequence 102, "sentinel" represents a predetermined sequence of tokens that indicates to the neural network 110 that it should generate a response to the context sequence 102, and "response" is the candidate output sequence 120.

[0091] In many autoregressive neural networks 110 that generate variable-length output sequences, a given output sequence ends when a predetermined end-of-sequence token is selected at a given time step. That is, when the end-of-sequence token is selected, the system 100 determines that the output sequence consists of the tokens at each time step preceding that given time step during the generation of the output sequence. In these implementations, the system 130 does not need to perform any additional processing to generate a hidden state for the input sequence 410, because the input sequence 410 has already been processed to select the end-of-sequence token of the sequence that terminated the generation of the candidate output sequences 120.

[0092] Next, the system 130 processes one or more of the hidden states 440 using the classifier 460 to generate a rating score 140 for each of one or more criteria. For example, the one or more hidden states 440 processed by the classifier 460 may be the respective hidden states for the tokens at the final position in the first candidate output sequence, generated by the first subnetwork. As another example, the one or more hidden states 440 processed by the classifier 460 may be the respective hidden states for a specified input token at a specified position in the input tokens, for example, the first input token in the input sequence, generated by the first subnetwork 430. As yet another example, the one or more hidden states 440 may include all hidden states 440, and the classifier 460 may be configured to average those hidden states 440.

[0093] The classifier 460 is a neural network comprising one or more classifier layers, configured to process one or more hidden states using these one or more classifier layers to generate a respective rating score 140 for each of the one or more criteria. For example, the classifier layers may include one or more fully connected layers, the final layer having a corresponding node for each criterion, such as a logistic regression or sigmoid node, and each node configured to generate a respective rating score 140 for the corresponding criterion.

[0094] Therefore, in the example in Figure 4, system 130 can generate scores for all criteria by processing the hidden states (which have already been generated as part of the candidate output sequence generation) using a small, computationally efficient classifier 460.

[0095] Prior to generating rating scores using classifier 460, and after the neural network 110 has been pre-trained, system 130 or another training system trains classifier 460 to accurately generate rating scores, that is, by repeatedly performing the training step on batches of training examples from a set of training data for training.

[0096] To carry out the training steps, the system, i.e., system 130 or another training system, acquires a batch of one or more training examples. For example, system 130 may sample a batch from a larger set of training examples for a fine-tuning training process.

[0097] For each training example in a batch, the system processes one or more of the respective hidden states generated by processing an input containing multiple input tokens, including all tokens at all positions in the training output sequence within the training example, using one or more classifier layers to generate the respective rating score for each output sequence criterion.

[0098] Next, for each training example in a batch, the system trains one or more classifier layers to minimize a loss function that measures the error between (i) the rating score generated for that training example and (ii) the ground truth rating score for that training example.

[0099] Specifically, the system can compute the gradient of the loss function with respect to the parameters of the classifier layer. The system can then update the parameters of the classifier layer by applying an optimizer to the computed gradient to generate an update, and then adding the update to the current value of the parameter or subtracting the update from the current value of the parameter.

[0100] The system can perform this training computationally efficient because classifier 460 has a relatively small number of parameters, i.e., relatively few compared to neural network 110, and the system freezes the input subnetwork while training classifier 460.

[0101] As described above, after the classifier 460 has been trained, system 130 or another training system can also use the classifier 460 to generate a filtered set of training data that can be used to further fine-tune the neural network 110.

[0102] Figure 3 is a flowchart of another exemplary process 500 for evaluating candidate output sequences. For convenience, process 500 is described as being performed by a system consisting of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system 100 shown in Figure 1 and appropriately programmed according to this specification, can perform process 500.

[0103] The system receives a request for an output sequence (step 502). In some implementations, the request includes a context sequence, and the request seeks a response to that context sequence.

[0104] The system uses an autoregressive language model neural network to generate a first candidate output sequence containing multiple tokens, each selected from the token vocabulary (step 504).

[0105] As described above, the language model neural network includes a first subnetwork and an output subnetwork.

[0106] The first subnetwork is configured to process an input comprising multiple input tokens at each specific location within a candidate output sequence, and to generate a corresponding hidden state for each input token, the input tokens comprising tokens at each location preceding that particular location within the candidate output sequence, and optionally tokens within the context sequence.

[0107] The output subnetwork is configured to process the respective hidden states of the input tokens at the position immediately preceding each of multiple locations to generate a score for each token in the token vocabulary, and to use those scores to select one of the tokens in the vocabulary as the token at that particular location in the first candidate output sequence.

[0108] As mentioned above, in some implementations, the system uses a neural network to generate multiple candidate output sequences, that is, one or more additional candidate output sequences in addition to the first candidate output sequence.

[0109] The system processes one or more of the hidden states generated by processing the input, which includes all tokens at all positions in the first candidate output sequence, with a first subnetwork, using one or more classifier layers to generate a rating score for each criterion in a set of one or more criteria (step 506).

[0110] When additional candidate output sequences are generated, the system again performs step 506 to generate a rating score for each additional candidate output sequence.

[0111] Next, the system determines whether to provide the first candidate output sequence in response to the request, using the respective rating scores of the first candidate output sequence for one or more output sequence criteria (step 508). For example, the system can make this determination as described above with reference to Figure 1.

[0112] "Embedding" in this specification means a vector of numerical values, such as floating-point numbers or other types of numerical values, having a predetermined dimension, such as a predetermined number of values.

[0113] The self-attention block described above is a neural network layer that includes an attention mechanism that acts across the self-attention block input (or input obtained from the layer input) to produce a self-attention block output. The self-attention mechanism may be causally masked so that any given position in the input sequence does not direct attention to any position in the input sequence after that given position (e.g., does not use data from any position in the input sequence after that given position). There are many different possible attention mechanisms. Some examples of self-attention layers that include attention mechanisms are described in the following literature: Vaswani et al. "Attention is all you need", 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0114] Generally, the attention mechanism maps queries and sets of key-value pairs to an output, where the query, key, and value are all vectors. The output is calculated as a weighted sum of the values, where the weights assigned to each value are calculated, for example, by a compatibility function between the query and the corresponding key, such as a dot product or a scaled dot product.

[0115] Generally, a self-attention mechanism is configured to associate different positions within the same sequence to determine a transformed version of that sequence as an output. For example, the attention layer input may consist of vectors for each element of the input sequence. These vectors provide input to the self-attention mechanism, which uses these vectors to determine a new representation of the same sequence for the attention layer output, which also consists of vectors for each element of the input sequence. The output of the self-attention mechanism can be used as the attention layer output, or the output of the self-attention mechanism can be processed by one or more of the following: a feedforward layer, skip connections, or normalization operations, to obtain the attention layer output.

[0116] In some implementations, the attention mechanism is, for example, matrix W Q Query transformations defined by, for example, matrix W K Key transformations defined by, and, for example, matrix W V Each of the value transformations defined by is applied to the attention layer input, which is the input data X to the attention layer, to form a query matrix Q = XW containing the respective queries for each vector in the input sequence. Q , a key matrix K=XW containing the respective keys for each vector in the input sequence. K, and a value matrix V=XW containing the respective values ​​for each vector in the input sequence. V The system is configured to obtain the following matrices, which are used to determine the sequence to focus attention on for the output. For example, the attention mechanism can be a dot product attention mechanism, which is applied by applying each query vector to each key vector to determine the respective weights for each value vector, and then using the respective weights to combine the value vectors to determine the self-attention layer output for each element of the input sequence. This self-attention layer output can be scaled by a scaling factor, for example by the square root of the dimensions of the query and key, to implement scaled dot product attention. Thus, for example, the output of the attention mechanism is softmax

[0117]

number

[0118] This can be determined as follows, where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism includes an "additive attention" mechanism that computes the fitting function using a feedforward network with one hidden layer. The output of the attention mechanism can be further processed by one or more fully connected feedforward neural network layers.

[0119] The attention mechanism can implement multi-head attention, meaning that the attention mechanism can apply multiple different attention mechanisms in parallel. The outputs of these attention mechanisms can then be combined, for example, concatenated, and a learned linear transformation can be applied to reduce the original dimensionality if necessary.

[0120] In this specification, the term “configured” is used in relation to systems and computer program components. A system consisting of one or more computers is configured to perform a particular operation or action if the system has software, firmware, hardware, or a combination thereof installed on it that causes the system to perform that operation or action when it is in operation. A computer program is configured to perform a particular operation or action if the program contains instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0121] The subject matter and functional operating embodiments described herein can be implemented in the form of digital electronic circuits, in the form of tangibly embodied computer software or computer firmware, in the form of computer hardware including the structures disclosed herein and their structural equivalents, or in the form of one or more of these combinations. Embodiments of the subject matter described herein can be implemented as one or more computer programs, for example, as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium to be executed by a data processing device or to control the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage board, a random-access or serial-access memory device, or one or more of these combinations. Alternatively or in addition, the program instructions can also be encoded on artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information so that it can be transmitted to a suitable receiver device for the purpose of execution by a data processing device.

[0122] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, a single programmable processor, a single computer, or multiple processors or computers. A device may also be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include FPGAs or ASICs. Optionally, in addition to hardware, a device may also include code that creates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code comprising one or more of these.

[0123] Computer programs, also called or sometimes described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may correspond to a file in a file system, although this is not required. A program can be stored in a single file dedicated to the program, or in a set of linked files, such as a file containing one or more modules, subprograms, or portions of code, which are part of a file that holds other programs or data, for example, in one or more scripts stored within a markup language document. A computer program can be deployed to run on one computer, or on multiple computers located in one site or distributed across multiple sites and interconnected by a data communication network.

[0124] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way or not at all, and which can be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each organized differently and each accessible differently.

[0125] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or software components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed on and running on the same one or more computers.

[0126] The processes and logic flows described herein can be implemented by one or more programmable computers that execute one or more computer programs to perform their functions by acting on input data and generating outputs. The processes and logic flows can also be implemented by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0127] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are the central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or integrated into dedicated logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to them to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio player or mobile video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0128] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks.

[0129] To enable user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can input to the computer. Interaction with the user can also be enabled using other types of devices; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic input, voice input, or haptic input. In addition, the computer can interact with the user by sending documents to and receiving documents from the device being used by the user, for example, by sending web pages to a web browser on the user's device in response to a request received from that web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user.

[0130] Data processing devices for implementing machine learning models may also include dedicated hardware accelerator units for handling common, computationally intensive parts of the workload, such as the machine learning training workload or the production workload, such as the inference workload.

[0131] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0132] Embodiments of the Subject Matter described herein can be implemented, for example, in a computing system including a backend component as a data server, or in a computing system including a middleware component, such as an application server, or in a computing system including a frontend component, such as a graphical user interface, web browser, or application having an implementation of the Subject Matter described herein and a user interacting with it, or in a computing system including one or more such backend components, middleware components, or frontend components in any combination. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0133] A computing system can include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device, for example, to display data to a user interacting with the device acting as a client, and to receive input from that user. Data generated on the user device, such as the results of user interaction, can be received from the device to the server.

[0134] This specification includes many specific details of implementations, which should not be interpreted as limitations on the scope of any invention or the scope of what is claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, features may be described above as acting in a particular combination, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may cover a partial combination or a variation of a partial combination.

[0135] Similarly, while the operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific order or sequence as shown, or that all indicated operations be performed, in order to achieve the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0136] Specific embodiments of this subject matter have been described above. Other embodiments are included in the scope of the appended claims. For example, the actions described in the claims can be performed in a different order, and the desired results can still be achieved. As an example, the process shown in the appended figures does not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of Symbols]

[0137] 100 Neural Network Systems 102 Context sequence, candidate output sequence, context 110 Autoregressive Language Model Neural Network, Autoregressive Neural Network 120 candidate output series 130 Criteria Evaluation System 140 Rating Score 150 output series 220A~N Standard Engine 230 Input Sequences 300 processes 410 First subnetwork, input sequence 420 input sequences 430 First Subnetwork 440 Hidden state 450 Output Subnetwork 460 Classifier 500 processes

Claims

1. A method carried out by one or more computers, The step of receiving a request for the output sequence, A step of generating a first candidate output sequence using an autoregressive language model neural network, wherein the first candidate output sequence comprises a plurality of tokens, each selected from a token vocabulary; For each output sequence criterion in a set of one or more output sequence criteria, Using the autoregressive language model neural network, the steps include processing an input sequence comprising (i) the first candidate output sequence followed by (ii) one or more tokens specifying the output sequence criteria, and generating a score for each token in the vocabulary; A step of determining a rating score for each of the first candidate output sequences, which represents the degree to which the first candidate output sequences generated by the autoregressive language model neural network satisfy the output sequence criteria, from the respective scores of the tokens in the corresponding proper subset of the vocabulary of the tokens; A step of providing the first candidate output sequence in response to the request, using the respective rating scores of the first candidate output sequence for the one or more output sequence criteria. Methods that include...

2. The method according to claim 1, wherein the request is for an output sequence following a context sequence, and the step of generating the first candidate output text sequence includes the step of conditioning the autoregressive language model neural network with an input comprising the context sequence, the input sequence further comprising the context sequence, wherein the input sequence comprises (i) the first candidate output sequence followed by (ii) one or more tokens specifying the output sequence criteria.

3. The steps include generating one or more additional candidate output sequences using the aforementioned autoregressive language model neural network, For each additional candidate output sequence, For each output sequence criterion in the set of one or more output sequence criterions, Using the autoregressive language model neural network, the steps include processing an input sequence comprising (i) the additional candidate output sequence followed by (ii) one or more tokens specifying the output sequence criteria, and generating a score for each token in the vocabulary; The steps include determining, from the respective scores of the tokens in the corresponding subset of the vocabulary of the tokens, each rating score representing the degree to which the additional candidate output sequences generated by the autoregressive language model neural network satisfy the output sequence criteria, and It further includes, The step of providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria is: Step 1: Using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences, select a sequence to be provided in response to the request from among the first candidate output sequence and the one or more additional candidate output sequences. The method according to claim 1 or claim 2, including the method described in claim 1 or 2.

4. The step of selecting a sequence to be provided in response to the request from among the first candidate output sequence and the one or more additional candidate output sequences, using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences, For each candidate output sequence, the step of generating a quality score from the respective rating scores of the candidate output sequence for each of the output sequence criteria within the set, The steps include selecting the candidate output series having the highest respective quality score as the series to be provided in response to the aforementioned request, and The method according to claim 3, including the method described in claim 3.

5. The step of providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria is: A step of determining whether the quality score of each of the first candidate output sequences satisfies a threshold for the first output criterion in the set, The step of determining not to provide the first candidate output sequence in response to the request if the quality score of each of the first candidate output sequences with respect to the first output criterion in the set does not satisfy the threshold; The method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

6. A method for training an autoregressive language model neural network according to any one of claims 1 to 5, A step of obtaining a batch of one or more training examples, wherein each training example is A training input sequence comprising (i) a training output sequence and (ii) one or more tokens specifying a particular output sequence criterion from the set of output sequence criteria, and The ground truth rating score represents the degree to which the training output sequence satisfies the corresponding output sequence criterion. It has steps, For each training example in the aforementioned batch, The steps include using the autoregressive language model neural network to process the training input sequence in the training example and generate a score for each token in the vocabulary, For each training example in the batch, the steps include: (i) training the autoregressive language model neural network to minimize the loss function that measures the error between (i) the respective scores of the tokens in the vocabulary generated for the training example and (ii) the ground truth set of scores for the tokens in the vocabulary that yields the ground truth rating score for the training example; Methods that include...

7. The step of training the autoregressive language model neural network to minimize the loss function is: Steps to apply the loss only to the error, without applying the loss to any previous position in the training input sequence. The method according to claim 6, including the method described in claim 6.

8. The method according to claim 6 or 7, further comprising the step of training the autoregressive language model neural network on a language modeling task that requires predicting the next token in a sequence of tokens, given a preceding token in the sequence of tokens, to minimize the loss function.

9. A method carried out by one or more computers, The step of receiving a request for the output sequence, A step of generating a first candidate output sequence using an autoregressive language model neural network, wherein the first candidate output sequence comprises a token at each of a plurality of positions, and the language model neural network A first subnetwork, wherein at each specific location of the plurality of locations, A first subnetwork is configured to process an input comprising multiple input tokens and generate a corresponding hidden state for each of the input tokens, wherein the input tokens comprise the tokens at each position preceding the particular position in the first candidate output sequence, and An output subnetwork, wherein at each specific location of the plurality of locations, The respective hidden states for the input token at the position immediately preceding the specific position are processed to generate the respective scores for each token in the token vocabulary. Using each of the aforementioned scores, one of the tokens in the vocabulary is selected as the token at the specific position in the first candidate output sequence. Output subnetwork configured as follows: It has steps, The steps of processing one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens, including all of the tokens at all of the positions in the first candidate output sequence, using one or more classifier layers, in order to generate a rating score for each output sequence criterion in a set of one or more output sequence criteria, representing the degree to which the first candidate output sequence generated by the autoregressive language model neural network satisfies the output sequence criterion; A step of providing the first candidate output sequence in response to the request, using the respective rating scores of the first candidate output sequence for the one or more output sequence criteria. Methods that include...

10. The method according to claim 9, wherein the request is for an output sequence following a context sequence, and the step of generating the first candidate output text sequence includes the step of conditioning the autoregressive language model neural network with an input comprising the context sequence, wherein for each particular position the input token further comprises a token from the context sequence.

11. The steps include generating one or more additional candidate output sequences using the aforementioned autoregressive language model neural network, For each additional candidate output sequence, the steps include generating a rating score for each output sequence criterion in the set of one or more output sequence criteria, representing the degree to which the additional candidate output sequence generated by the autoregressive language model neural network satisfies the output sequence criterion; It further includes, The step of providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria is: Step 1: Using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences, select a sequence to be provided in response to the request from among the first candidate output sequence and the one or more additional candidate output sequences. The method according to claim 9 or claim 10, including the method according to claim 9 or 10.

12. The step of selecting a sequence to be provided in response to the request from among the first candidate output sequence and the one or more additional candidate output sequences, using the respective rating scores of the first candidate output sequence and the one or more additional candidate output sequences, For each candidate output sequence, the step of generating a quality score from the respective rating scores of the candidate output sequence for each of the output sequence criteria within the set, The steps include selecting the candidate output series having the highest respective quality score as the series to be provided in response to the aforementioned request, and The method according to claim 11, including the method described in claim 11.

13. The step of providing the first candidate output sequence in response to the request using the respective rating scores for the one or more output sequence criteria is: A step of determining whether the quality score of each of the first candidate output sequences satisfies a threshold for the first output criterion in the set, The step of determining not to provide the first candidate output sequence in response to the request if the quality score of each of the first candidate output sequences with respect to the first output criterion in the set does not satisfy the threshold; The method according to any one of claims 9 to 12, including the method described in any one of claims 9 to 12.

14. The step of processing one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens, including all of the tokens at all of the positions in the first candidate output sequence, by the first subnetwork, The step of processing the respective hidden states for the token at the final position in the first candidate output sequence, which is generated by the first subnetwork. The method according to any one of claims 9 to 13, including the method described in any one of claims 9 to 13.

15. The step of processing one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens, including all of the tokens at all of the positions in the first candidate output sequence, by the first subnetwork, The step of processing the respective hidden states for a specified input token at a specified location among the input tokens generated by the first subnetwork. The method according to any one of claims 9 to 14, including the method described in any one of claims 9 to 14.

16. During the generation of the first candidate output sequence, for one or more designated positions prior to the final position in the first candidate output sequence, The steps include processing one or more of the respective hidden states generated by the first subnetwork at the designated position in the first candidate output sequence using one or more classifier layers to generate respective rating scores for each output sequence criterion in a set of one or more output sequence criteria, representing the extent to which the portion of the first candidate output sequence generated at the designated position satisfies the output sequence criterion; (i) whether the first candidate output sequence should continue to be generated past the designated location, or (ii) whether any portion of the first candidate output sequence should not be provided in response to the request, based on the respective rating scores for the output sequence criteria in the set generated at the designated location. The method according to any one of claims 9 to 15, further comprising:

17. A method for training an autoregressive language model neural network according to any one of claims 9 to 16, A step of obtaining a batch of one or more training examples, wherein each training example is Training output sequence, and For each of the output sequence criteria (one or more of the aforementioned output sequence criteria), the ground truth rating score represents the degree to which the training output sequence satisfies the particular output sequence criterion. It has steps, For each training example in the aforementioned batch, The steps include: processing one or more of the respective hidden states generated by processing an input comprising a plurality of input tokens, including all tokens at all positions in the training output sequence within the training example, using one or more classifier layers, in order to generate a rating score for each output sequence criterion; For each training example in the batch, the steps include: (i) training one or more classifier layers to minimize the loss function that measures the error between the respective rating score generated for the training example and (ii) the respective ground truth rating score for the training example; Methods that include...

18. The step of training one or more classifier layers to minimize the loss function is: The step of keeping the input subnetwork frozen during the training of one or more of the classifier layers. The method according to claim 17, including the method described in claim 17.

19. The method according to claim 17 or 18, further comprising the step of training the autoregressive language model neural network on a language modeling task that requires predicting the next token in a sequence of tokens, given the preceding token in the sequence of tokens.

20. The method according to any one of claims 1 to 19, wherein the output sequence is a text sequence and the vocabulary of the token comprises a plurality of text tokens.

21. The method according to claim 20, which is also dependent on claim 2 or claim 10, wherein the context sequence is a text sequence.

22. One or more computers, One or more storage devices that store instructions, when executed by the one or more computers, cause the one or more computers to perform each of the operations described in any one of claims 1 to 21, and A system that includes these features.

23. One or more computer-readable storage media that, when executed by one or more computers, stores instructions causing one or more computers to perform each of the operations according to any one of claims 1 to 21.

Images