Self-improving training of generative neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a generative neural network configured to generate output sequences from input sequences. In one aspect, a method comprises: obtaining a reward model training dataset comprising a plurality of training tuples, wherein each training tuple comprises an input sequence, a first output sequence, and a second output sequence; a reward model on the reward model training dataset to train the reward model to generate, for each training tuple included in the reward model training dataset, a reward model output that is a prediction of a difference between (i) a first quality measure of the first output sequence and (ii) a second quality measure of the second output sequence; and using a reinforcement learning technique that uses rewards derived from reward model outputs generated by the trained reward model to train the generative neural network.

Description

[0001] Attorney Docket No.56113-0546WO1 SELF-IMPROVING TRAINING OF GENERATIVE NEURAL NETWORKS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No.63 / 541,290, filed on September 28, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to training neural networks.

[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0004] Some neural networks are generative neural networks. Generative neural networks are neural networks that can generate new and original content, such as text, photos, videos, code, data, or 3D renderings, from the vast amounts of data they are trained on.

[0005] During training, a number of techniques have been used to improve the quality of the outputs generated by such neural networks. For example, a human-in-the-loop training paradigm can be used to improve the quality of the outputs by collecting more diverse and high-quality data and improve the alignment. However, this approach requires extra human effort and may not be realistic in real-world scenarios, especially for a specific domain that requires expert knowledge. As another example, prompting methods can also be used. However, prompting usually requires explicitly and thoroughly written rubrics to define the improvement goal, and it is hard to infer and write all possible rubrics for humans with a real-world complex improvement goal. SUMMARY

[0006] This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a generative neural network. The generative neural network can be configured through training to generate output sequences from input sequences. Attorney Docket No.56113-0546WO1

[0007] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0008] This specification describes a system that implements a self-improving training framework to train a generative neural network, e.g., a language model neural network, to improve the quality of the output sequences generated by the neural network. The training framework may be referred to as “self-improving” because the training process reuses existing training data that is already available to the system, and thus requires no human intervention that would otherwise be required in generating additional, annotated training data.

[0009] The self-improving training framework makes use of a reinforcement learning training objective which evaluates a numeric quality measure difference between two output sequences, e.g., rather than a likelihood of the preference of one output sequence over another. In particular, by leveraging the known preference between different output sequences given an input sequence that is provided in each training tuple within the existing training data, the system can train a reward model to predict a quality difference between the different output sequences in each training tuple represented as a numeric value. Once trained, the system can then train the generative neural network by using a reinforcement learning technique that uses rewards derived from the quality difference predictions computed by the reward model.

[0010] Existing training data can thus be utilized in a way that increases the value of the data for training a generative neural network. This can, in turn, reduce the amount of labeled training data needed to effectively train neural networks and improve the quality of the output sequences generated by those neural networks. Thus, the cost associated with human intellect necessary for generating annotated training data can be reduced. Moreover, the amount of computing resources necessary for the training of the neural networks can be reduced. For example, the amount of memory required for storing the annotated training data can be reduced, the amount of processing resources used to generate the annotated training data can be reduced, or both. The improved effectiveness in the training of generative neural networks can be especially significant for large language model neural networks that are harder to train due to their open-ended generative configuration.

[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. Attorney Docket No.56113-0546WO1 BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG.1 shows an example training system.

[0013] FIG.2A is an example illustration of operations to implement a self-improving training framework to train a generative neural network.

[0014] FIG.2B is an example illustration of operations to use a generative neural network.

[0015] FIG.3 is a flow diagram of an example process for training a generative neural network configured to generate output sequences from input sequences.

[0016] FIG.4 is a flow diagram of sub-steps of one of the steps of the process of FIG.3.

[0017] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] FIG.1 is a diagram of an example training system 100. The training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0019] The training system 100 trains a generative neural network 110 that is used to perform one or more generative tasks. In particular, the generative neural network 110 is a neural network having parameters 116 and that can be configured through training to process an input sequence 102 that is made up of tokens from a vocabulary in accordance with the parameters 116 to generate, based on the input sequence 102, an output sequence 104 for a generative task that is made up of tokens from the vocabulary. For example, the parameters 116 include weights and, optionally, biases of the layers of the generative neural network 110.

[0020] After training, the training system 100 or another inference system 150 can deploy the generative neural network 110 on one or more computing devices to perform inference for the one or more generative tasks, i.e., to generate new output sequences 104 for the generative tasks based on new input sequences 102.

[0021] The vocabulary of tokens can include any of a variety of tokens that represent text symbols or other symbols. For example, the vocabulary of tokens can include one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code.

[0022] Additionally, or alternatively, the vocabulary of tokens can include tokens that can represent data other than text. For example, the vocabulary of tokens can include image Attorney Docket No.56113-0546WO1 tokens that represent a discrete set of image patch embeddings of an image that can be generated by an image encoder neural network based on processing the image patches of the image. As another example, the vocabulary of tokens can include audio tokens that represent code vectors in a codebook of a quantizer, e.g., a residual vector quantizer.

[0023] In some implementations, the generative neural network 110 can be configured as an auto-regressive language model neural network. The language model neural network is referred to as an auto-regressive neural network when the language model neural network auto-regressively generates an output sequence 104 of tokens by generating each particular token in the output sequence 104 conditioned on a current input sequence that includes any (e.g. all) tokens that precede the particular token in the output sequence 104, i.e., tokens that have already been generated for any previous positions in the output sequence 104 that precede the particular position of the particular token, and the input sequence 102.

[0024] For example, the current input sequence when generating a token at any given position in the output sequence 104 can include the input sequence 102 and the tokens at any preceding positions that precede the given position in the output sequence 104. As a particular example, the current input sequence can include the input sequence 102 followed by the tokens at any (e.g. all) preceding positions that precede the given position in the output sequence 104. Optionally, the input sequence 102 and the current output sequence can be separated by one or more predetermined tokens within the current input sequence.

[0025] More specifically, to generate a particular token at a particular position within an output sequence 104, the generative neural network 110 can process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the vocabulary of tokens. The generative neural network 110 can then select, as the particular token, a token from the vocabulary using the score distribution. For example, the generative neural network 110 can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0026] As a particular example, the generative neural network 110 can be or comprise an auto-regressive Transformer-based neural network that includes (i) a sequence comprising a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0027] The generative neural network 110 can have any of a variety of Transformer-based language model neural network architectures. Examples of such neural network Attorney Docket No.56113-0546WO1 architectures include those described in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020; Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311; Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023; and Gemini Team, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023).

[0028] Generally, however, the Transformer-based language model neural network includes a sequence of attention blocks, and, during the processing of a given input sequence 102, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates at least the hidden state for the last token in a given input sequence at least in part by applying self- attention to generate a respective output hidden state for the last token. The input hidden states for the first attention block are embeddings of the input tokens in the input sequence and the input hidden states for each subsequent attention block are the output hidden states generated by the preceding attention block.

[0029] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate the score distribution.

[0030] In some implementations, the generative neural network 110 can be configured as diffusion model neural network that generates an output sequence 104 for a generative task from random noise across multiple updating iterations by performing a reverse diffusion process conditioned on an input sequence 102.

[0031] Examples of such neural networks include those described in Chitwan Saharia, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022; Aditya Ramesh, et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125; Robin Rombach, et al. High-resolution image synthesis with latent Attorney Docket No.56113-0546WO1 diffusion model, Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2022; and Huiwen Chang, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.

[0032] As an example, the generative neural network 110 can generate text sequences, i.e., each output sequence 104 generated by the generative neural network 110 is a sequence of text tokens from a vocabulary of text tokens that includes, e.g., one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in natural language text. For example, the inference system 150 can use the generative neural network 110 to generate text sequences and provide the text sequences for presentation to users.

[0033] As another example, the generative neural network 110 can generate images or videos that have multiple frames (where each frame is an image) by generating images, e.g., either as sequences of pixels or through an iterative denoising process. For example, the output sequence 104 generated by the generative neural network 110 includes a plurality of color values for pixels in an image arranged according to a specified order. As another example, the output sequence 104 generated by the generative neural network 110 includes a plurality of tokens that represent image patch embeddings of an image which can then be processed by a decoder neural network to generate the image. For example, the inference system 150 can use the generative neural network 110 to generate an image or a video conditioned on an input sequence 102 that includes a text description of the content of the image or the video.

[0034] As another example, the input sequence 102 is a sequence of text and the output sequence 104 is another sequence of text, e.g., a completion of the input sequence of text, a paraphrase of the input sequence of text, a response to a question posed in the input sequence, or a sequence of text that is about a topic specified by the input sequence of text. As another example, the input sequence 102 can be an input other than text, e.g., a plurality of pixels included in an image, and the output sequence 104 can be a text sequence that describes the input.

[0035] As another example, the input sequence 102 represents data to be compressed, e.g., image data, text data, audio data, or any other type of data; and the output sequence 104 is a compressed version of the data. The tokens included in the output sequence 104 can include any representation of compressed data, e.g., symbols or embeddings to be decoded by a respective neural network. Attorney Docket No.56113-0546WO1

[0036] As a particular example, the inference system 150 can be part of a dialog system and the input sequence 102 can include audio or text from the most recent conversational turn submitted by a user of the dialog system during the dialog while the output sequence 104 is the next turn in the conversation, e.g., either text or audio that is a response to the most recent conversational turn. Optionally, the input sequence 102 can also include one or more historical conversational turns that occurred earlier in the conversation.

[0037] As another particular example, the inference system 150 can be part of a machine translation system and the input sequence 102 can include text in a source language while the output sequence 104 can include text in a target language that is a translation of the source text into the target language.

[0038] As another particular example, the inference system 150 can be part of a natural language processing system. For example, if the input sequence 102 is a sequence of words in an original language, e.g., a sentence or phrase, the output sequence 104 can be a summary of the input sequence in the original language, i.e., a sequence that has fewer words than the input sequence but that retains the essential meaning of the input sequence. As another example, if the input sequence 102 is a sequence of words that form a question, the output sequence 104 can be a sequence of words that form an answer to the question.

[0039] As another particular example, the inference system 150 can be part of a computer- assisted medical diagnosis system. For example, the input sequence 102 can be a sequence of data from an electronic medical record and the output sequences 104 can each be a sequence of predicted treatments.

[0040] As another particular example, the inference system 150 can be part of a computer code generation system and the input sequence 102 can include a text description of a desired piece of code or a snippet of computer code in a programming language and the output sequence 104 can include computer code, e.g., a snippet of code that is described by the input sequence 102 or a snippet of code that follows the input sequence 102 in a computer program.

[0041] As another particular example, the inference system 150 can be part of a multi- modal system that processes multi-modal input sequences 102, e.g., both text and image input sequences, or both text and audio input sequences, and generates the output sequences 104 that are either in a single data modality or in multiple data modalities, e.g., text and image output sequences, or text and audio output sequences. Examples of such multi- modal systems include an image captioning system, a text-based image search system, an image-based question answering system, and so on. Attorney Docket No.56113-0546WO1

[0042] As another particular example, the inference system 150 can be part of or associated with a robotic control system, i.e., a system for controlling one or more mechanical agents. The input sequence 102 can comprise a natural language description of one or more tasks for a the one or more mechanical agents and the output sequence can comprise a sequence of instructions (e.g., joint angles, torques, velocities, etc.) for the one or more mechanical agents that cause the one or more mechanical agents to perform the one or more tasks described in the input sequence 102.

[0043] As another particular example, the inference system 150 can be part of or associated with a search system that facilitates searching of resources on the Internet. A resource can be any data that can be provided over the Internet. A resource can be identified by a resource address that is associated with the resource. Resources include web pages, word processing documents, portable document format (PDF) documents, images, video, and news feed sources, to name a few.

[0044] In this particular example, the search system can receive search queries submitted by client devices and, in response, identify resources that are relevant to the search query in the form of search results and return the search results to the user devices in search results pages. A search result page can include search result data generated by the search system that identifies a resource responsive to a search query, and includes a link to the resource. The search result page can additionally include a result in the form of an output sequence 104 that is generated by the inference system 150 based on an input sequence 102 derived from the search query.

[0045] The generative neural network 110 is typically trained using a two-stage approach: a pre-training stage followed by a fine-tuning stage, where at least the fine-tuning stage takes place at the training system 100. For example, the training system 100 can receive data specifying a pre-trained generative neural network 110 from another system, and then perform the fine-tuning of the pre-trained generative neural network 110.

[0046] In the pre-training stage, the generative neural network 110 is pre-trained by the training system 100 or another system based on optimizing one or more unsupervised or self-supervised objective functions, e.g., a maximum-likelihood objective function, on one or more large datasets and then, in some cases, adjusted to the generative tasks, which can include any combination of one or more of the generative tasks mentioned below and possibly other tasks, through fine-tuning adaptation based on supervised learning, reinforcement learning from human feedback (RLHF), reinforcement learning from AI Attorney Docket No.56113-0546WO1 feedback (RLAIF), prompt tuning, instruction tuning, and the like, that use different training objectives, different datasets, or both.

[0047] The one or more large datasets used during the pre-training stage can include a large dataset of text in one or more natural languages, e.g., text that is publicly available from the Internet or another text corpus, a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, and so on, e.g., computer code that is publicly available from the Internet or another code repository, a large dataset of audio samples, e.g., audio recordings or waveforms that represent the audio recordings, a large dataset of images where each image includes an array of pixels, a large dataset of videos where each video includes a temporal sequence of frames, or a large multi-modal dataset that includes a combination of two or more of these datasets.

[0048] In the fine-tuning stage, the training system 100 implements a self-improving training framework to train the generative neural network 110 to improve the quality of the output sequences 104 generated by the generative neural network 110. The self-improving training framework uses a reinforcement learning training objective based on rewards derived from reward model outputs generated by a reward model 120.

[0049] To implement the self-improving training framework, the training system 100 uses a reward model optimization engine 130 to train the reward model 120 (i.e., to update the values of the parameters of the reward model 120). Once the reward model 130 is trained, the training system 100 then uses a generative neural network optimization engine 140 to train the generative neural network 110 (i.e., to update the values of the parameters 116 of the generative neural network 110 that have been determined as a result of the pre-training stage to reinforcement learning fine-tuned values) through reinforcement learning based on reward model outputs generated by the trained reward model 120.

[0050] The reward model optimization engine 130 is configured to train the reward model 130 on a reward model training dataset 132 based on optimizing a reward model loss function. The reward model training dataset 132 includes a plurality of training tuples. Each training tuple includes an input sequence, a first output sequence, and a second output sequence.

[0051] The reward model 130 can have any appropriate architecture that allows the reward model 130 to process a reward model input that includes a first output sequence and a second output sequence and, optionally, other data, e.g., a predetermined prompt, to generate a reward model output that is a prediction of a difference between a first quality Attorney Docket No.56113-0546WO1 measure of the first output sequence and a second quality measure of the second output sequence.

[0052] Notably, the reward model 130 is not configured to directly predict either the first quality measure of the first output sequence or the second quality measure of the second output sequence. Rather, it is configured to predict the difference, i.e., the “gap,” between the first quality measure and the second quality measure.

[0053] For example, the reward model 130 can include any appropriate types of neural network layers (e.g., convolutional layers, fully-connected layers, attention layers, etc.) in any appropriate number (e.g., 5 layers, 10 layers, or 20 layers) and connected in any appropriate configuration (e.g., as a linear sequence of layers).

[0054] As a particular example, the reward model 130 can similarly have a Transformer- based neural network architecture. In some implementations, the reward model 130 can be initialized using a Transformer-based neural network that has been pre-trained based on optimizing one or more unsupervised or self-supervised objective functions on one or more large datasets.

[0055] Once trained, the reward model 130 can then be used by the generative neural network optimization engine 140 to train the generative neural network 110 through reinforcement learning on a reinforcement learning training dataset 142.

[0056] In particular, the generative neural network optimization engine 140 trains the generative neural network 110 based on optimizing a reinforcement learning training objective which includes one or more terms dependent on the reward model outputs generated by the trained reward model 120.

[0057] Like the reward model training dataset 132, the reinforcement learning training dataset 142 includes a plurality of training tuples. Each training tuple includes an input sequence, a first output sequence, and a second output sequence.

[0058] In some implementations, prior to the reinforcement learning and after the pre- training of the generative neural network 110, the generative neural network optimization engine 140 generates a supervised fine-tuned instance of the generative neural network by fine-tuning an instance of the generative neural network 110 (i.e., updates the pre-trained values of the parameters 116 of the instance of the generative neural network 110 that have been determined as a result of the pre-training stage to supervised fine-tuned values) on a supervised fine-tuning training dataset 144 based on optimizing a supervised fine-tuning loss function. The supervised fine-tuned values will generally be different from the reinforcement learning fine-tuned values mentioned above. Attorney Docket No.56113-0546WO1

[0059] Like the reward model training dataset 132 and the reinforcement learning training dataset 142, the supervised fine-tuning training dataset 144 includes a plurality of training tuples. Each training tuple includes an input sequence, a first output sequence, and a second output sequence.

[0060] The training system 100 can obtain the reward model training dataset 132, and the reinforcement learning training dataset 142, the supervised fine-tuning training dataset 144 in any of a variety of ways.

[0061] In some implementations, the training system 100 receives the training datasets as new data. For example, the training system 100 can receive training datasets as one or more uploads from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the training system 100. As another example, the training system 100 can receive one or more inputs from a user specifying which data that is already maintained by the system 100 should be used as the training datasets.

[0062] In other implementations, the training system 100 reuses existing training data that is already available to the system as the training datasets, e.g., either as-is or by rearranging or reorganizing the existing training data (but without needing to request new data from a user). For example, the existing training data can include an annotated training dataset that has been used during the pre-training stage of the generative neural network 110.

[0063] In these other implementations, the annotated training dataset can be partitioned by the training system 100 to generate the reward model training dataset 132, and the reinforcement learning training dataset 142, the supervised fine-tuning training dataset 144.

[0064] As an example, the annotated training dataset can be ^^ ൌ ^^ ^^, ^^௪, ^^^^^ଷ^, where ^^ is the total number of training tuples. For each training tuple, ^^ is an input sequence, ^^௪is a first output sequence that has a relatively higher quality measure, and ^^^is a second output sequence that has a relatively lower quality measure than the first output sequence ^^௪.

[0065] For example, for each training tuple, the input sequence ^^ is a context sequence, and the first output sequence ^^௪and the second output sequence ^^^are each a respective output sequence, e.g., a respective response to the input sequence ^^, that has been generated by the generative neural network 110 conditioned on the input sequence ^^ during the pre-training stage.

[0066] Notably, the annotated training dataset need not include or be otherwise associated with the quality measure of any of the output sequences included in the annotated training Attorney Docket No.56113-0546WO1 dataset. That is, each training tuple included in the annotated training dataset includes a first output sequence ^^௪that has a first quality measure that is higher than a second quality measure of a second output sequence ^^^that is also included in the training tuple, but neither the first quality measure nor the second quality measure is provided or otherwise specified by the training tuple.

[0067] For example, the first output sequence ^^௪and the second output sequence ^^^can be arranged in a descending order of quality within each training tuple included in the annotated training dataset, such that the output sequence in the earlier position is the one that has the relatively higher quality measure while the output sequence in the latter position is the one that has the relatively lower quality measure.

[0068] The quality measure of an output sequence 104 generated by the generative neural network 110 can be defined with respect to any aspect of the output sequence 104. Generally, however, the quality measure may affect the suitability of the generative neural network 110 for deployment; for example the higher the quality measure, the more suitable the generative neural network 110 is for deployment in the inference system 150.

[0069] As an example, the quality measure can be or be dependent on a helpfulness of the output sequence 104. As another example, the quality measure can be or be dependent on a conciseness of the output sequence 104. As another example, the quality measure can be or be dependent on a harmlessness of the output sequence 104. As another example, the quality measure can be or be dependent on a combination of two or more of the above aspects, e.g., a combination of both helpfulness and harmlessness, and possibly other aspects.

[0070] In some situations, the quality measure can be determined based on evaluating the output sequence 104 against a set of one or more criteria. For example, the one or more criteria may specify that the text included in the output sequences should not include certain content or information. For example, the criteria may specify that the output sequences should not include one or more of: offensive content, misinformation, or confidential or private information, such as personal contact information.

[0071] As another example, the criteria may specify that the output sequences should not result in undesired data leakage. For example, in context of federated learning where different organizations train the network together by using local data that is private to each organization, the text from any private training data or other protectable data (e.g., copyrighted data) that is used to pre-train the generative neural network 110 should not be included in the output sequences. Attorney Docket No.56113-0546WO1

[0072] By implementing the self-improving training framework as described in this specification, the training system 100 can train the generative neural network 110 to improve the quality measure of the output sequences 104, and hence improve the suitability of the generative neural network 110 for deployment in the inference system 150.

[0073] FIG.2A is an example illustration of operations performed by the training system 100 to implement a self-improving training framework to train the generative neural network 110.

[0074] During training, the training system 100 trains the generative neural network 110 on the reinforcement learning training dataset 142 by using a reinforcement learning technique that uses rewards derived from reward model outputs generated by the reward model 120.

[0075] The generative neural network 110 generates, for each training tuple included in the reinforcement learning training dataset 142, a predicted output sequence (“Output 2”) 206 based on processing at least an input sequence 202 included in the training tuple.

[0076] The reward model 120 then processes a reward model input that includes (i) the input sequence 202 included in the training tuple, (ii) a reference output sequence (“Output 1”) 204, and (iii) the predicted output sequence (“Output 2”) 206 to generate a reward model output that is a prediction of the difference between a first quality measure of the reference output sequence (“Output 1”) 204 and a second quality measure of the predicted output sequence (“Output 2”) 206.

[0077] The reward model output generated by the reward model 120 is then used in the reinforcement learning technique to train the generative neural network 110.

[0078] In particular, the training system 100 trains the generative neural network 110 based on optimizing a reinforcement learning training objective to learn to generate predicted output sequences which tend to increase the prediction of the difference between the first quality measure of the reference output sequence (“Output 1”) 204 and the second quality measure of the predicted output sequence (“Output 2”) 206. That is, the training system 100 trains the generative neural network 110 to maximize the quality gap between the reference output sequence and the predicted output sequence.

[0079] Training the generative neural network 110 in this way is different from training the generative neural network 110 to learn to generate higher quality predicted output sequences by optimizing a conventional reinforcement learning from human feedback (RLHF) training objective that is based on rewards that would be generated by some reward model which is configured to predict the quality of the predicted output sequences generated by the generative neural network 110. Attorney Docket No.56113-0546WO1

[0080] FIG.2B is an example illustration of operations performed by the inference system 150 to use the generative neural network 110 after the generative neural network 110 has been trained using the self-improving training framework.

[0081] Once trained, the inference system 150 can use the generative neural network 110 to generate output sequences based on input sequences using a self-improving technique. The inference system 150 receives an input sequence 212 and generates a first output sequence 214 based on processing the input sequence 212 using the generative neural network 110. Then, inference system 150 performs a self-improving iteration to generate a second output sequence 216 based on processing (i) the input sequence 212 and (ii) the first output sequence 214 using the generative neural network 110.

[0082] That is, the inference system 150 uses the same generative neural network 110 twice to generate two different output sequences in response to the same input sequence 212 and, because of the way the generative neural network 110 has been trained using the self- improving training framework, the second output sequence 216 is likely to have a higher quality than the first output sequence 214.

[0083] In some implementations, the inference system 150 generates a prompt based on (i) the input sequence 212 and (ii) the first output sequence 214, and then processes the prompt using the generative neural network 110 to generate the second output sequence 216.

[0084] Unlike some existing prompting-based methods, e.g., the self-refine technique which requires explicitly and thoroughly written rubrics that define what a high quality output sequence is, this prompt need not be lengthy and instead need only include the input sequence 212 and the first output sequence 214 and a few additional tokens. Therefore, generating multiple output sequences by using the self-improving technique has lower computational overhead than generating multiple output sequences using those existing prompting-based techniques.

[0085] For example, the prompt for processing by the generative neural network 110 to generate the second output sequence 216 can be: { ^^ } Assistant: << Candidate >> { ^^^^^} << Improved >> where ^^ is the input sequence 212, ^^^^^is the first output sequence 214, “{” and “}” are special tokens that surround the input sequence 212 and the first output 214, “<< Candidate >>” are tokens separating the input sequence 212 from the first output sequence 214, and “<< Improved >>” are tokens indicating to the generative neural network 110 that it should generate the second output sequence 216. Attorney Docket No.56113-0546WO1

[0086] In principle the self-improving iteration can be repeated multiple times during inference. In other words, after having generated the second output sequence 216, the inference system 150 can perform another self-improving iteration to generate a third output sequence by generating a prompt that includes (i) the input sequence 212, (ii) the first output sequence 214, and (iii) the second output sequence 216 using the generative neural network 110, and then using the generative neural network 110 to process the prompt, and so on.

[0087] Thus, the inference system 150 can use the same generative neural network 110 to generate a series of multiple output sequences in response to the same input sequence, each output sequence generated conditioned on any output sequence in the series that has previously been generated by the generative neural network 110.

[0088] In some implementations, the inference system 150 continues performing the self- improving iterations until termination criteria for the self-improving of the generative neural network 110 have been satisfied, e.g., until a threshold amount of wall clock time has elapsed, until a threshold number of self-improving iterations have been performed, or until the quality of the output sequences generated by the generative neural network 110 have converged.

[0089] The output sequences generated by the inference system 150 can be used in any of a variety of ways. For example, the inference system 150 can provide one or more of the output sequences generated over the multiple self-improving iterations for presentation to the user on a client device. As another example, the inference system 150 can provide one or more of the output sequences generated over the multiple self-improving iterations to another component in the system for further processing. As yet another example, the inference system 150 can store one or more of the output sequences generated over the multiple self-improving iterations in a data repository for some future purpose.

[0090] For example, in some implementations, the inference system 150 can provide the output sequence generated in the last self-improving iteration for presentation to the user on the client device.

[0091] In some other implementations, the inference system 150 can “discriminate” between the generated output sequences, e.g., by using the same generative neural network 110 or another output sequence quality evaluation engine of the inference system 150, to generate an output selecting determine which output sequence, if any, to provide for presentation to the user. Attorney Docket No.56113-0546WO1

[0092] For example, the inference system 150 can use the same generative neural network 110 to process an input that includes multiple output sequences to generate an output selecting one of the multiple output sequences.

[0093] FIG.3 is a flow diagram of an example process 300 for training a generative neural network configured to generate output sequences from input sequences. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG.1, or another training system, appropriately programmed in accordance with this specification, can perform the process 300.

[0094] The system obtains a reward model training dataset (step 302). The reward model training dataset includes a plurality of training tuples. Each training tuple includes an input sequence, a first output sequence, and a second output sequence.

[0095] The system trains a reward model on the reward model training dataset (step 304). The reward model is trained by the system to generate, for each training tuple included in the reward model training dataset, a reward model output that is a prediction of a difference between (i) a first quality measure of the first output sequence and (ii) a second quality measure of the second output sequence. The quality measure of an output sequence generated can be defined with respect to any aspect of the output sequence.

[0096] In some implementations, the prediction of the difference can be a numeric value within a predefined range (e.g., 0.0-1.0, 0-10, negative 1.0-1.0, and so on). For example, the numeric value can be represented using a floating-point format or an integer format (or another fixed-point format).

[0097] In some implementations, the prediction of the difference can be a binned score. For example, a prediction of a first range (e.g., 0.0-0.33) may be output as “low,” a prediction of a second range (e.g., 0.34-0.66) may be output as “medium,” and a prediction of a third range (e.g., 0.67-1.0) may be output as “high.” The reward model output can also be configured to define the prediction of the difference in other ways.

[0098] The system trains the reward model by updating the values of parameters of the reward model based on optimizing a reward model loss function. The system can do this by computing, for each training tuple in a batch of training tuples obtained from the reward model training dataset, respective gradients of the reward model loss function with respect to the parameters of the reward model by backpropagation through the parameters of the reward model. The system can then determine the updates by applying an update rule, e.g., Attorney Docket No.56113-0546WO1 an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the respective gradients.

[0099] The reward model loss function includes one or more terms that, for each training tuple in a batch, are dependent on the reward model outputs generated by the reward model based on the first output sequence and the second output sequence included in the training tuple.

[0100] In some implementations, the reward model loss function includes a first term that is dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the second quality measure of the second output sequence.

[0101] To generate (a), the reward model processes a reward model input that includes the input sequence, the first output sequence, and the second output sequence to generate a reward model output, which can be used as (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output sequence.

[0102] To generate (b), the reward model processes a reward model input that includes the input sequence, the second output sequence, and the second output sequence to generate a reward model output, which can be used as (b) the difference between (i) the second quality measure of the second output sequence and (ii) the second quality measure of the second output sequence.

[0103] In some implementations, the reward model loss function includes a second term that is dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the first quality measure of the first output sequence and (ii) the first quality measure of the first output sequence.

[0104] To generate (b), the reward model processes a reward model input that includes the input sequence, the first output sequence, and the first output sequence to generate a reward model output, which can be used as (b) the difference between (i) the first quality measure of the first output sequence and (ii) the first quality measure of the first output sequence.

[0105] In some implementations, the reward model loss function includes a third term that is dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output Attorney Docket No.56113-0546WO1 sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

[0106] To generate (b), the reward model processes a reward model input that includes the input sequence, the second output sequence, and the first output sequence to generate a reward model output, which can be used as (b) the difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

[0107] In some implementations, the reward model loss function includes a fourth term that is dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the first quality measure of the first output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

[0108] In some implementations, the reward model loss function includes a fifth term that is dependent on a difference between (a) the difference between (i) the second quality measure of the second output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

[0109] In some implementations, the reward model loss function includes two or more of the first, second, third, fourth, or fifth term mentioned above. For example, the reward model loss function can include all of the first, second, third, fourth, and fifth terms.

[0110] For example, the reward model loss function can be: is the third term, is the fourth term, and is the fifth term. For example, ^^ ^^ ^^ represents a logarithmic function, a sigmoid function, and ^^^௪,^^^is the shortcut of ^^^^^^ ^^, ^^௪, ^^^^ that is the difference between (i) the first quality measure of the first output sequence ^^௪and (ii) the second quality measure of the second output sequence ^^^and that is generated by the Attorney Docket No.56113-0546WO1 reward model from processing a reward model input that includes the input sequence ^^, the first output sequence ^^௪, and the second output sequence ^^^.

[0111] After having trained the reward model, i.e., after termination criteria for the training of the reward model have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of forward / backward passes through the parameters of the reward model have been performed, the system uses the reward model together with a reinforcement learning technique to train the generative neural network (step 306).

[0112] FIG.4 is a flow diagram of sub-steps 402-404 of step 306 of the process of FIG.3.

[0113] The system obtains a reinforcement learning training dataset (step 402). The reinforcement learning training dataset includes a plurality of training tuples. Each training tuple includes an input sequence, a first output sequence, and a second output sequence.

[0114] The system trains the generative neural network by updating the values of parameters of the generative neural network based on optimizing a reinforcement learning loss function (step 404). In doing so, the system determines reinforcement learning fine- tuned values of the parameters of the generative neural network from their pre-trained values that have been determined as a result of the pre-training stage.

[0115] The system can do this by computing, for each training tuple in a batch of training tuples obtained from the reinforcement learning training dataset, respective gradients of the reinforcement learning loss function with respect to the parameters of the generative neural network by backpropagation through the parameters of the generative neural network. The system can then determine the updates by applying an update rule, e.g., an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the respective gradients.

[0116] The reinforcement learning loss function includes one or more terms that, for each training tuple in a batch, are dependent on an output sequence generated by the generative neural network based at least on the input sequence included in the training tuple.

[0117] In some implementations, the reinforcement learning loss function includes a first term that is dependent on a reward model output of the reward model that is a prediction of a difference between (i) a quality measure of a predicted output sequence generated by the generative neural network from processing the input sequence and a reference output sequence (which can be any appropriate output sequence, as explained further below), and (ii) a quality measure of the reference output sequence. Attorney Docket No.56113-0546WO1

[0118] To generate the prediction of the difference between (i) and (ii), the reward model processes a reward model input that includes the input sequence, the predicted output sequence, and the reference output sequence to generate a reward model output, which can be used as the prediction of the difference between (i) and (ii).

[0119] In some implementations, the reinforcement learning loss function includes a second term that is dependent on a predicted output sequence generated by the supervised fine-tuned instance of the generative neural network from processing the input sequence and the reference output sequence.

[0120] As mentioned above, the supervised fine-tuned instance of the generative neural network can be generated by training an instance of a pre-trained generative neural network on a supervised fine-tuning training dataset based on optimizing a supervised fine-tuning loss function.

[0121] The supervised fine-tuning loss function measures, for each training tuple included in the supervised fine-tuning training dataset, a difference between (i) a predicted output sequence that is generated by the supervised fine-tuned instance of the generative neural network from processing the input sequence and the second output sequence included in the training tuple and (ii) the first output sequence included in the training tuple.

[0122] For example, the supervised fine-tuning loss function can be: where ^^ is an higher quality measure, and ^^^is a second output sequence that has a relatively lower quality measure than the first output sequence ^^௪, and ^^^ூ்represent the supervised fine-tuned instance of the generative neural network.

[0123] In this way, the supervised fine-tuned instance of the generative neural network is trained to generate, for each training tuple included in the supervised fine-tuning training dataset, and based on processing the input sequence and the second output sequence included in the training tuple, a predicted output sequence that is not too much different from the first output sequence included in the training tuple.

[0124] In some implementations, the reinforcement learning loss function includes a third term that is dependent on a predicted output sequence generated by the generative neural network from processing the input sequence and the reference output sequence. Attorney Docket No.56113-0546WO1

[0125] In some implementations, the reinforcement learning loss function includes two or more of the first, second, or third term mentioned above. For example, the reinforcement learning loss function can include all of the first, second, and third terms.

[0126] The reference output sequence can be any appropriate output sequence. For example, the reference output sequence can be the first output sequence included in the training tuple. As another example, the reference output sequence can be the second output sequence included in the training tuple. As another example, the reference output sequence can be a predicted output sequence generated by the generative neural network from processing the input sequence. As yet another example, the reference output sequence can be a predicted output sequence generated by the generative neural network from processing the input sequence and another predicted output sequence generated by the generative neural network, where the other predicted output sequence is generated by the generative neural network from processing the input sequence.

[0127] For example, when the reference output sequence is either the first output sequence ^^௪or the second output sequence ^^^included in the training tuple, the reinforcement learning loss function can be: where ^^^^^^ ^^, output sequence ^^ generated by the generative neural network from processing the input sequence ^^ and the reference output sequence ^^^^^and (ii) the quality measure of the reference output sequence ^^^^^; ^^^^^^ ^^, ^^, ^^^^^^ is generated by the reward model from processing a reward model input that includes the input sequence ^^, the predicted output sequence ^^, and the reference output sequence ^^^^^. is a weighted Kullback–Leibler divergence term a Kullback–Leibler divergence between (i) a predicted output sequence generated by the generative neural network ^^^ோூ^் from processing the input sequence ^^ and the reference output sequence ^^^^^and (ii) the predicted output sequence generated by the supervised fine-tuned instance of the generative neural network ^^^ௌூி்்from processing the input sequence ^^ and the reference output sequence ^^^^^. Attorney Docket No.56113-0546WO1

[0128] As another example, when the reference output sequence is a predicted output sequence generated by the generative neural network from processing the input sequence ^^, the reinforcement learning loss function can be:

[0130] In some implementations, the system refrains from adjusting the supervised fine- tuned values of the supervised fine-tuned instance of the generative neural network when updating the values of the parameters of the generative neural network based on optimizing the reinforcement learning loss function. Thus, the parameters of the supervised fine-tuned instance of the generative neural network are held fixed to their supervised fine-tuned values while the parameters of the generative neural network are being updated.

[0131] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0132] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is Attorney Docket No.56113-0546WO1 generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0133] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0134] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0135] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0136] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a Attorney Docket No.56113-0546WO1 central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0137] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0138] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0139] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. Attorney Docket No.56113-0546WO1

[0140] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework.

[0141] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0142] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0143] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0144] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be Attorney Docket No.56113-0546WO1 performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0145] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0146] What is claimed is:

Claims

Attorney Docket No.56113-0546WO1 CLAIMS 1. A method of training a generative neural network configured to generate output sequences from input sequences, the method comprising: obtaining a reward model training dataset comprising a plurality of training tuples, wherein each training tuple comprises an input sequence, a first output sequence, and a second output sequence; training a reward model on the reward model training dataset to train the reward model to generate, for each training tuple included in the reward model training dataset, a reward model output that is a prediction of a difference between (i) a first quality measure of the first output sequence and (ii) a second quality measure of the second output sequence; and after training the reward model, using a reinforcement learning technique that uses rewards derived from reward model outputs generated by the trained reward model to train the generative neural network.

2. The method of claim 1, wherein training the reward model on the reward model training dataset comprises training the reward model by optimizing a reward model loss function that comprises: a first term dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the second quality measure of the second output sequence.

3. The method of any one of claims 1-2, wherein the reward model loss function comprises: a second term dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the first quality measure of the first output sequence and (ii) the first quality measure of the first output sequence.Attorney Docket No.56113-0546WO1 4. The method of any one of claims 1-3, wherein the reward model loss function comprises: a third term dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

5. The method of any one of claims 1-4, wherein the reward model loss function comprises: a fourth term dependent on a difference between (a) the difference between (i) the first quality measure of the first output sequence and (ii) the first quality measure of the first output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

6. The method of any one of claims 1-5, wherein the reward model loss function comprises: a fifth term dependent on a difference between (a) the difference between (i) the second quality measure of the second output sequence and (ii) the second quality measure of the second output sequence, and (b) a difference between (i) the second quality measure of the second output sequence and (ii) the first quality measure of the first output sequence.

7. The method of any one of claims 1-6, wherein the prediction of the difference comprises a numeric value.

8. The method of claim 7, wherein the numeric value is represented using a floating point format.Attorney Docket No.56113-0546WO1 9. The method of any one of claims 1-8, further comprising, prior to training the reward model: obtaining a supervised fine-tuning training dataset comprising a plurality of training tuples, each training tuple comprising an input sequence, a first output sequence, and a second output sequence; training the generative neural network on the supervised fine-tuning training dataset to obtain a supervised fine-tuned instance of the generative neural network by optimizing a supervised fine-tuning loss function that measures, for each training tuple included in the supervised fine-tuning training dataset, a difference between (i) a predicted output sequence that is generated by the generative neural network from processing the input sequence and the second output sequence included in the training tuple and (ii) the first output sequence included in the training tuple.

10. The method of any one of claims 1-9, wherein using the reinforcement learning technique to train the generative neural network comprises: obtaining a reinforcement learning training dataset comprising a plurality of training tuples, each training tuple comprising an input sequence, a first output sequence, and a second output sequence; and training the generative neural network on the reinforcement learning training dataset by optimizing a reinforcement learning loss function that comprises, for each training tuple included in the reinforcement learning training dataset: a first term dependent on a reward model output of the trained reward model that is a prediction of a difference between (i) a quality measure of a predicted output sequence generated by the generative neural network from processing the input sequence and a reference output sequence, and (ii) a quality measure of the reference output sequence.

11. The method of claim 10, wherein the reinforcement learning loss function comprises, for each training tuple included in the reinforcement learning training dataset: a second term dependent on a predicted output sequence generated by the supervised fine-tuned instance of the generative neural network from processing the input sequence and the reference output sequence.Attorney Docket No.56113-0546WO1 12. The method of any one of claims 10-11, wherein the reinforcement learning loss function that comprises, for each training tuple included in the reinforcement learning training dataset: a third term dependent on a predicted output sequence generated by the generative neural network from processing the input sequence and the reference output sequence.

13. The method of any one of claims 10-12, wherein the reference output sequence is the first output sequence included in the training tuple.

14. The method of any one of claims 10-12, wherein the reference output sequence is the second output sequence included in the training tuple.

15. The method of any one of claims 10-12, wherein the reference output sequence is a predicted output sequence generated by the generative neural network from processing the input sequence.

16. The method of any one of claims 10-12, wherein the reference output sequence is a predicted output sequence generated by the generative neural network from processing the input sequence and another predicted output sequence generated by the generative neural network, wherein the other predicted output sequence is generated by the generative neural network from processing the input sequence.

17. The method of any one of claims 9-16, wherein using the reinforcement learning technique to train the generative neural network on the generative neural network training dataset comprises: adjusting parameter values of the generative neural network based on optimizing the reinforcement learning loss function while holding parameter values of the supervised fine- tuned instance of the generative neural network fixed.

18. The method of any one of claims 1-17, wherein training the generative neural network comprises: obtaining an annotated dataset comprising a plurality of training tuples, wherein each training tuple comprises an input sequence, a first output sequence, and a second output sequence; and generating, from the annotated dataset, the supervised fine-tuning training dataset, the reward model training dataset, and the reinforcement learning training dataset.Attorney Docket No.56113-0546WO1 19. The method of claim 18, wherein for each training tuple included in the annotated dataset, the first output sequence has a first quality measure that is lower than a second quality measure of the second output sequence, but neither the first quality measure nor the second quality measure is provided by the training tuple.

20. A method of using a generative neural network of any one of preceding claims to generate output sequences, the method comprising: receiving an input sequence; generating, by processing the input sequence using the generative neural network, a first output sequence; and generating, by processing (i) the input sequence and (ii) the first output sequence using the generative neural network, a second output sequence.

21. The method of claim 20, further comprising providing the second output sequence for display on a client device as a response to the input sequence.

22. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-21.

23. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-21.

Images