Reinforcement learning with preference feedback

Abstract

To provide a system apparatus, a method, and a program for training a neural network by comparing qualities between network outputs.SOLUTION: A method includes: a step of receiving one or more network inputs; a step of processing each of the network inputs using a target neural network to generate a first network output for each network input; a step of processing each of the network inputs using an alternative neural network to generate a second network output; a step of applying a preference function to the first network output and the second network output to generate a preference score that compares the first network output and the second network output; a step of updating a weight of the target neural network using an objective function that encourages the first network output to be prioritized over the second network output.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This specification relates to processing data using machine learning models. [Background technology]

[0002] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the parameters of the model.

[0003] Some machine learning models are deep models that use multiple model layers to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output. Summary of the Invention [Problem to be solved by the invention]

[0004] Described herein are systems and methods, implemented as computer programs on one or more computers at one or more locations, that can train neural networks using preference functions that compare measures of quality between network outputs. [Means for solving the problem]

[0005] According to a first aspect, there is provided a method for training a target neural network, the method being executed by one or more computers and having a plurality of target neural network weights, configured to process network inputs according to the target neural network weights to generate network outputs. In each of a plurality of training steps, the method includes receiving one or more network inputs. For each of the network inputs, the method includes processing the network input using the target neural network to generate a first network output, processing the network input using an alternative neural network for the training steps to generate a second network output, and applying a preference function to the first network output and the second network output to generate a preference score indicating the likelihood that the first network output is a higher quality output than the second network output. For each training step, the method includes updating the target neural network weights using an objective function including a first term that encourages a preference score to favor the first network output over a corresponding second network output.

[0006] In some implementations, the method further includes, in each of the plurality of training steps, and for each of the one or more network inputs, (i) determining a first likelihood score to be assigned to the first network output by the target neural network given the network input, and (ii) determining a second likelihood score to be assigned to the first network output by a reference neural network for the training step given the network input, wherein the objective function includes a second term that penalizes the target neural network for producing a first likelihood score that deviates from the corresponding second likelihood score.

[0007] In some implementations, in each of the multiple training steps, the reference neural network for the training step is an alternative neural network for the training step.

[0008] In some implementations, applying a preference function to the first network output and the second network output to generate a preference score indicating the likelihood that the first network output is a higher quality output than the second network output includes (i) processing the first network output and the second network output using a preference model neural network to generate preference network outputs, the preference model neural network being trained to process pairs of neural network outputs and generate corresponding preference network outputs for each processed pair of neural network outputs that indicate the likelihood that one member of the processed pair is a higher quality output than the other member of the processed pair; and (ii) generating a preference score based on the preference network outputs that indicates the likelihood that the first network output is a higher quality output than the second network output.

[0009] In some implementations, in each of the plurality of training steps, and for each of the one or more network inputs, determining a second likelihood score to be assigned to the first network output by the reference neural network for the training step given the network input includes determining the second likelihood score to be assigned to the first network output based on a geometric mixture of the first likelihood score of the first network output and the likelihood score of the first network output under a fixed reference distribution, and based on a normalization constant for the training step.

[0010] In some implementations, in each of the multiple training steps, the alternative neural network for the training step has multiple alternative neural network weights for the training step and has the same architecture as the target neural network, and the weights of the alternative neural network for the training step are determined by an exponential moving average of the weights of the target neural network over the current and previous training steps.

[0011] In some implementations, the objective function is based on a preference score determined by a preference function that indicates, for each network input and corresponding first and second network outputs, the likelihood that the first network output is a higher quality output than the second network output, a first likelihood score for the first network output determined by the target neural network, and a second likelihood score for the first network output determined by a fixed reference distribution.

[0012] In some implementations, updating the weights of the current target neural network using an objective function including a first term that encourages a preference for a first network input over a corresponding second network input according to the preference score includes determining a gradient of the objective function with respect to the weights of the target neural network, and updating the weights of the target neural network based on the gradient of the objective function.

[0013] In some implementations, determining the gradient of the objective function with respect to the current target neural network weights includes calculating, for each network input and corresponding first and second network outputs, a gradient of a first likelihood score of a first network output determined by the target neural network with respect to the current target neural network weights, a preference score determined by a preference function indicating the likelihood that the first network output is a higher quality output than the second network output, a first likelihood score of the first network output determined by the target neural network, and a likelihood score of the first network output determined by a fixed reference distribution.

[0014] In some implementations, the target neural network is a large language model.

[0015] In some implementations, the preference neural network is a large language model.

[0016] In some implementations, the first network output is a first network output sequence, the second network output is a second network output sequence, and processing the network input using an alternative neural network to generate the second network output includes autoregressively generating the second network output sequence using the alternative neural network.

[0017] In some implementations, in each of the plurality of training steps, and for each of the one or more network inputs, determining a second likelihood score to be assigned to the first network output by the reference neural network for the training step given the network input includes: for each output element of the first network output sequence, determining a first likelihood score for the output element using the target neural network given the network input; determining a second likelihood score for the output element based on a geometric mixture of the first likelihood score of the output element and the likelihood score of the output element under a fixed reference distribution and based on a normalization constant for the training step; and determining a second likelihood score to be assigned to the first network output based on the second likelihood score of the output element in the first network output sequence.

[0018] In some implementations, the preference function can model non-transitive preference relationships.

[0019] According to another aspect, a system is provided comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the aforementioned method.

[0020] According to another aspect, one or more non-transitory computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the aforementioned methods.

[0021] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0022] The described system can use pairwise preference functions to train a target neural network that processes pairs of network outputs to determine preference scores. In some implementations, the described system can train a preference model to model the pairwise preference functions. Unlike conventional methods for training neural networks based on preferences, which often rely on fitting reward models to assign numerical values ​​to individual network outputs aligned with preferences, the described system can model non-transitive preferences. The use of pairwise preference functions can also model transitive preferences more accurately than conventional methods. Thus, the described system can accurately train neural networks based on preferences for a wider range of applications (e.g., modeling human preferences, predicting the outcomes of competitive games, etc.) than conventional methods.

[0023] By using pairwise preference functions to more accurately model preferences for conditional generation tasks, the described system can train a target neural network to achieve a particular threshold of performance (e.g., determined based on the preference functions) in the conditional generation task with fewer training iterations than conventional methods. Thus, the described system can train a neural network based on preferences more efficiently (e.g., in terms of training time, computational resources, etc.) than conventional methods.

[0024] Conventional methods often train neural networks to optimize an adapted reward model based on preference data and can be susceptible to reward hacking, in which network outputs that maximize rewards under the reward model are not preferred over network outputs that receive worse rewards. In contrast, the described system can train a target neural network to approach a Nash equilibrium of preferences with respect to a set of alternative neural networks (e.g., to produce network outputs that are at least as preferable as those produced by the set of alternative neural networks).

[0025] In particular, the described system can train a target neural network by comparing outputs from the target neural network with corresponding outputs of alternative neural networks. The described system can use an objective function that includes a term indicating a preference for the target neural network output compared to the corresponding alternative neural network output. While training the target neural network, the described system can update the alternative neural network at each training step based on the training of the target neural network. By comparing the preference for the target neural network output with the corresponding network output from the updated alternative neural network at each training step, the described system's objective function can encourage the target neural network to approach a Nash equilibrium of preferences for the set of alternative neural networks. By training the target neural network to approach a Nash equilibrium of preferences, the described system can be less susceptible to reward hacking compared to conventional methods.

[0026] The described systems can also regularize the training of the target neural network with respect to a reference neural network. In particular, the described systems can use an objective function that includes a term that penalizes the target neural network for being too different from the reference neural network. For example, the reference neural network can produce known safe or desired outputs, and the described systems can train the target neural network based on preferences while producing outputs similar to those of the reference neural network.

[0027] Thus, the described system can enable more accurate and efficient training of neural networks based on preferences while ensuring that the target neural network is capable of producing the preferred output.

[0028] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 is a block diagram of an exemplary training system. [Figure 2] FIG. 1 is a block diagram of an exemplary target neural network. [Figure 3] FIG. 1 is a flow diagram of an exemplary process for training a target neural network. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for training a preference model neural network. [Figure 5] FIG. 1 is a flow diagram of an exemplary process for determining an objective function for a training step. DETAILED DESCRIPTION OF THE INVENTION

[0030] Like reference numbers and designations in the various drawings indicate like elements.

[0031] 1 illustrates an exemplary training system 100. Training system 100 is an example of a system in which the systems, components, and techniques described below are implemented as computer programs on one or more computers at one or more locations.

[0032] The training system 100 can train a target neural network 102 using a set of training data 104 to perform a conditional generation task. In particular, the system 100 can train the target neural network 102 to perform a conditional generation task tailored to task preferences. The target neural network 102 has a set of target neural network weights, including, for example, weights for the target neural network layers and, optionally, biases. The target neural network 102 can be a neural network that has been trained (e.g., pre-trained) to perform a conditional generation task, and the system 100 can further train (e.g., fine-tune) the target neural network 102 to perform a conditional generation task tailored to task preferences.

[0033] The training system 100 includes a weight update system 106 that can update the weights of the target neural network 102 to train the target neural network 102 over a series of training steps. In each training step, the training system 100 can train the target neural network 102 to perform a conditional generative task on a set (e.g., a mini-batch) of network inputs 108.

[0034] The target neural network 102 may be any generative model suitable for generating output samples based on the network inputs 108. The target neural network 102 is described in more detail below with reference to FIG.

[0035] The conditional generation task may be any of a variety of tasks. As an example, the network input 108 may be a text prompt, and the conditional generation task may generate an output sample as described by the text prompt. As a further example, the target neural network 102 may be configured to output, for example, a sample image, a sample video, a sample audio, etc., as described by the text prompt from the network input 108.

[0036] The conditional generation task may be a style transfer task to match the style of, e.g., image, video, audio, etc., specified by the network input 108. For example, the target neural network 102 may be configured to output, e.g., a sample image, sample video, sample audio, etc., that matches the style of the network input 108.

[0037] A conditional generation task may be a transcription task in which the target neural network 102 is configured to generate a textual transcription of, for example, video, audio, etc., specified by the network input 108.

[0038] A conditional generation task may be a summarization task in which the target neural network 102 is configured to generate a text summary of, for example, text, images, video, audio, etc., specified by the network inputs 108.

[0039] The conditional generation task may be a natural language processing (NLP) task, and the target neural network 102 may be a large-scale language model configured to perform the language processing task. For example, the target neural network 102 may be configured to generate text responses to user-provided network inputs 108. As another example, the target neural network 102 may be configured to generate code (e.g., in a programming language such as Python, Java, C++, etc.) for a computer program to perform the task specified by the user-provided network inputs 108.

[0040] The conditional generative task can be an action selection task for an agent interacting with an environment. For example, the network input 108 can be, e.g., sensor data for the agent characterizing observations of the environment, observed trajectories of other agents in the environment, etc., and the target neural network 102 can be configured to sample actions for the agent based on the observations of the environment. As a further example, the network input 108 can be, e.g., LIDAR data, camera data, etc. collected for an autonomous vehicle, and the target neural network 102 can be configured to select actions for controlling the autonomous vehicle. As another example, the network input 108 can characterize a game state, and the target neural network 102 can be configured to select actions for a player of the game.

[0041] The training system 100 trains the target neural network 102 to perform a conditional generation task using a pairwise preference function 109. The pairwise preference function 109 can compare a given output sample from the target neural network 102 with a corresponding output sample from an alternative neural network 110 to generate a preference score 111 that indicates the likelihood that the output sample from the target neural network 102 is a higher quality output than the output sample from the alternative neural network 110.

[0042] The alternative neural network 110 can be any of a variety of networks capable of providing alternative output samples (to the target neural network 102) for a conditional generation task. Examples of alternative neural networks 110 are described below with reference to FIG. 5.

[0043] As an example, the preference score 111 may indicate whether an output sample from the target neural network 102 or a corresponding output sample from the alternative neural network 110 is more likely to be drawn from the ground truth distribution.

[0044] As another example, the preference score 111 may indicate, for a conditional generation task, whether the output sample from the target neural network 102 or the corresponding output sample from the alternative neural network 110 is more likely to have, for example, less distortion, perceptual distortion, etc.

[0045] As another example, the preference score 111 may indicate whether the output sample from the target neural network 102 or the corresponding output sample from the alternative neural network 110 is more likely to receive a larger reward for a conditional generation task.

[0046] As yet another example, preference score 111 may quantify a human preference between an output sample from target neural network 102 and a corresponding output sample from an alternative neural network 110 .

[0047] In particular, preference function 109 can model non-transitive preferences between network outputs. That is, for network outputs A, B, and C, if the preference function determines that A is preferable to B (i.e., more likely to be a high-quality output) and that B is preferable to C, then preference function 109 can determine that C is preferable to A. In the above example, a model limited to modeling transitive preferences would necessarily determine that A is preferable to C. Non-transitive preferences are found in a variety of applications (e.g., modeling human preferences, predicting player performance in competitive games, etc.). Thus, preference function 109 can accurately model preferences for a wider range of applications compared to models limited to modeling transitive preferences.

[0048] In each training step, system 100 uses pairwise preference functions 109 to compare target network outputs 112, which include output samples from target neural network 102, with corresponding alternative network outputs 114, which include output samples from alternative neural networks 110, to generate preference scores 111 for the training step. Based on preference scores 111, weight update system 106 generates weight updates 116 for target neural network 102 and updates 118 for alternative neural networks 110. Weight update system 106 can generate weight updates 116 and 118 to train target neural network 102 to perform a conditional generation task. The process of training target neural network 102 is described in more detail below with reference to FIG. 3.

[0049] FIG. 2 illustrates an exemplary target neural network 102.

[0050] The target neural network 102 includes a generative model 202 that can process the network inputs 108 and generate output samples 204 to perform a conditional generation task.

[0051] The generative model 202 may have any suitable architecture for processing the network inputs 108 and generating the output samples 204 to perform a conditional generation task. As an example, the generative model 202 may be an autoregressive generative model (e.g., a Transformer, a recurrent neural network, etc.) that can autoregressively generate a sequence as the output samples 204 based on the network inputs 108. The generative model may be, for example, a large-scale language model (LLM) that can autoregressively generate tokenized representations of text data, a visual language model (VLM) that can autoregressively generate tokenized representations of image or video data, an audio language model that can autoregressively generate tokenized representations of text data, etc.

[0052] As another example, the generative model 202 may be a diffusion model (e.g., a denoising diffusion model, a score-based diffusion model, a latent diffusion model, etc.) that can generate the output samples 204 by repeatedly transforming samples from a noise distribution (e.g., a Gaussian distribution) based on the network input 108 over a series of iterations. For example, the generative model 202 may be a diffusion model that transforms samples from a noise distribution using a denoising neural network having any suitable architecture (e.g., a convolutional neural network, a recurrent neural network, etc.).

[0053] As another example, the generative model 202 may be a neural network that can generate the output samples 204 by transforming samples from a noise distribution (e.g., a Gaussian distribution). The generative model 202 may be, for example, a generator network of a generative adversarial network, a decoder of a variational autoencoder, a normalization flow, etc.

[0054] The generative model 202 may generate a network output 208 that defines a conditional probability or likelihood of each of the output samples 204 given the corresponding network input 108. For example, the network output 208 may include a conditional probability for a set of values ​​for each of the output samples 204 given the corresponding network input 108. As another example, the network output 208 may determine statistical properties (e.g., mean, variance, etc.) related to the conditional distribution for each of the output samples 204 given the corresponding network input 108. As another example, the generative model 202 may include multiple processing layers and multiple layer activation functions, and the network output 208 may include output values ​​from the processing layers and layer activation functions that result from the generative model 202 processing the corresponding network input 108.

[0055] The target neural network 102 may include a sample likelihood system 206. The sample likelihood system 206 may process data specifying the conditional distributions 208 to generate likelihood scores 210 that specify the likelihood of each of the output samples 204 under the conditional distributions 208 given the corresponding network inputs 108.

[0056] The likelihood score 210 may be any of a variety of numerical values ​​that indicate the conditional probability of the likelihood of the output sample 204 given the corresponding network input 108. For example, the likelihood score 210 may be the conditional probability or likelihood of the output sample 204 given the corresponding network input 108. As another example, the likelihood score 210 may be a transform of the conditional probability or likelihood (e.g., log-probability, log-likelihood, negative log-likelihood, non-normal value, etc.) of the output sample 204 given the network input 108. As another example, the sample likelihood system 206 may determine an auxiliary distribution that approximates the conditional distribution of the output sample 204 given the corresponding network input 108, and the likelihood score 210 may be the probability or likelihood of the output sample 204 under the auxiliary distribution.

[0057] The sample likelihood system 206 can determine a likelihood score 210 based on the network output 208 in any of a variety of ways. For example, the network output 208 can include a set of conditional probabilities for the output samples 204, and the sample likelihood system can return the conditional probabilities of the output samples 204 as the likelihood score 210. As another example, the network output 208 can include data specifying a conditional distribution of the output samples 204, and the sample likelihood system 206 can return the likelihood of the output samples 204 under the specified conditional distribution as the likelihood score 210. As another example, the network output 208 can include data specifying the conditional distribution of the output samples 204, and the sample likelihood system 206 can determine an approximate likelihood of the output samples 204 under the specified conditional distribution and return it as the likelihood score 210. As a particular example, the network outputs 208 may include data specifying layer outputs from the generative model 202, and the sample likelihood system may determine approximate likelihoods (e.g., by using importance weights to determine an approximation of the conditional distribution) to return as likelihood scores 210. As another particular example, the sample likelihood system 206 may include a neural network trained to approximate the conditional distribution of the output samples 204 given the network inputs 108, and may process the network inputs 108 to determine approximate likelihoods of the output samples 204 to return as likelihood scores 210.

[0058] 3 is a flow diagram of an exemplary process for training a target neural network to perform a conditional generative task. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a training system such as training system 100 of FIG. 1 suitably programmed in accordance with this specification can perform process 300.

[0059] The training system can train the target neural network over multiple training steps.

[0060] At each training step, the training system may receive a set of one or more network inputs for the training step (step 302). For example, the system may receive mini-batches of network inputs from a set of training data.

[0061] The system may process the network inputs for the training step using the target neural network to generate corresponding network outputs (step 304). In particular, the target neural network may process each of the network inputs to generate corresponding output samples for performing a conditional generation task.

[0062] The system may process the network inputs for the training step using an alternative neural network for the training step to generate corresponding network outputs (step 306). In particular, the alternative neural network may process each of the network inputs to generate corresponding output samples for performing the conditional generation task.

[0063] For each network input, the system may apply a preference function to the corresponding network outputs from the target neural network and the alternative neural networks to determine a preference score comparing the network outputs (step 308). For each pair of network outputs from the target neural network and the alternative neural network, the preference score indicates the likelihood that the network output from the target neural network is a higher quality output than the network output from the alternative neural network.

[0064] In some implementations, the preference function is a preference model neural network. The preference model neural network can be trained to process pairs of neural network outputs and generate a corresponding preference network output for each processed pair of neural network outputs that indicates the likelihood that one member of the processed pair is a higher quality output than the other member of the processed pair. An exemplary process for training a preference model neural network is described in more detail below with reference to FIG. 4.

[0065] The system may determine whether the training step is complete (step 310). For example, the system may determine that the training step is complete when it has processed all network inputs in the set of network inputs. If the training step is not complete, the system may process the next network input for the training step.

[0066] Once the training step is complete, the system may update the weights of the target neural network based on the preference scores determined for the training step (step 312). In particular, the system may update the weights of the target neural network using an objective function for the training step that includes a term that encourages the system to favor network outputs from the target neural network over corresponding network outputs from alternative neural networks according to the preference scores. An exemplary objective function for the training step is described in more detail below with reference to FIG. 5.

[0067] 4 is a flow diagram of an exemplary process for training a preference model neural network. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a training system such as training system 100 of FIG. 1 suitably programmed in accordance with this specification can perform process 400.

[0068] The system receives a set of training data (step 402). The training data includes example network inputs for training a target neural network to perform a conditional generation task. The training data may include data specifying a preference function for the network outputs. For example, the training data may include, for each example network input, a pair of example network outputs and a corresponding indication that one network output of the pair is of higher quality than the other network output. As a particular example, the training data may include, for each pair of example network outputs, a target likelihood that the network output of the pair is of higher quality than the other output of the pair. As another particular example, the training data may include, for each pair of example network outputs, a target classification of which network output of the pair is the higher quality output.

[0069] The system trains a preference model neural network to model the preference function specified by the training data (step 404). As described above, the system can process pairs of neural network outputs and train the preference model neural network to generate corresponding preference network outputs for each processed pair of neural network outputs that indicate the likelihood that one member of the processed pair is a higher quality output than the other member of the processed pair. As one example, the system can train the preference model neural network to replicate target likelihoods from the training dataset by optimizing a regression loss function (e.g., L2 loss) between the preference network outputs and the corresponding target likelihoods. As another example, the system can train the preference model neural network to replicate target classifications for pairs of network outputs by optimizing a classification loss function (e.g., cross-entropy loss) between the preference network outputs and the corresponding target classifications.

[0070] The preference model neural network can have any of a variety of network architectures suitable for processing pairs of network outputs. For example, if the network outputs are images, the preference model neural network can include a neural network suitable for processing image data (e.g., a convolutional neural network, a visual transformer, etc.). As another example, if the network outputs are sequences of values ​​(e.g., time series data, text sequences, video sequences, audio sequences, etc.), the preference model neural network can include a neural network suitable for processing sequences (e.g., a recurrent neural network, a transformer network, etc.). As a particular example, the preference model neural network can be a large-scale language model.

[0071] The system can train the preference model neural network to model any of a variety of preference functions for the network outputs. For example, for each pair of network outputs, the preference function can indicate the likelihood that one network output of the pair is of higher quality than the other network output, as determined by some quality measure. As a further example, the preference function can indicate the likelihood that one output of the pair has lower, e.g., distortion, perceptual distortion, etc. As another further example, when the network outputs are actions of an agent interacting with an environment, the preference function can indicate the likelihood that one output of the pair will receive a greater reward.

[0072] The preference function can represent the preferences of a human or group of humans. For example, for each pair of network outputs, the preference function can indicate the human's preference for one of the network outputs as the higher quality output of the pair. As another example, the preference function can indicate the human's belief or confidence that one of the network outputs is more likely to be the higher quality output of the pair.

[0073] In particular, the preference function can be a non-transitive preference function. For example, if y1, y2, and y3 are network outputs, the preference function can indicate that humans prefer y1 to y2, and prefer y2 to y3, but also prefer y3 to y1. As another example, if the network outputs represent actions performed by agents interacting in an environment, the preference function can indicate the non-transitive likelihood of success of the output actions when performed by competing agents.

[0074] As described above, the system can then use the training dataset and the trained preference model neural network to train a target neural network to perform a conditional generation task (step 406).

[0075] 5 is a flow diagram of an exemplary process for determining an objective function for training a target neural network to perform a conditional generative task. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a training system such as training system 100 of FIG. 1 suitably programmed in accordance with this specification can perform process 500.

[0076] As described above, the system trains the target neural network over a series of training steps, in each of which the system receives a set of network inputs.

[0077] The system processes a set of network inputs to generate a set of network outputs from a target neural network and a corresponding set of network outputs from an alternative neural network for the training step (step 502).

[0078] The alternative neural network for each time step can be determined based on the target neural network. As described above, the alternative neural network can be any of a variety of networks that can provide alternative output samples (relative to the target neural network) for the conditional generation task.

[0079] For example, the alternative neural network may be a network trained to perform a conditional generation task with a different network architecture compared to the target neural network. As another example, the alternative neural network may have the same network architecture as the target neural network. As a further example, the alternative neural network may have the same architecture as the target neural network with different hyperparameters (e.g., trained with different hyperparameters, performed inference using different hyperparameters, etc.), such as network size, network depth, number of inputs, learning rate, etc. As a further example, the alternative neural network may be a reparameterization of the target neural network with different network weights, or may be a copy of the target neural network fine-tuned on a set of training data specific to the alternative neural network. As a further example, the alternative neural network may process modified network inputs compared to the target neural network (e.g., by conditionally generating network outputs based on modified neural network inputs that specify generating outputs in a different style than the target neural network, by processing modified input prompts compared to the target neural network, etc.).

[0080] In particular, the alternative neural network can have the same network architecture as the target neural network and can have alternative neural network weights determined based on the target neural network weights in the current and previous training steps. As an example, the alternative neural network weights can be the target neural network weights from a previous time step (e.g., the alternative neural network weights in the Tth training step,

[0081]

number

[0082] is θ T-K , which are the weights of the target neural network at the TK-th time step).

[0083] As another example, the weights of the replacement neural network can be determined by a moving average of the weights of the target neural network in the current and previous training steps. That is, the replacement neural network is determined by the conditional distribution

[0084]

number

[0085] can be modeled as the weights of the alternative neural network at the T-th training step.

[0086]

number

[0087] is determined by a moving average.

[0088]

number

[0089] K>0 is the window length, and θ i is the weight of the target neural network at the ith training step.

[0090] As another example, the weights of the replacement neural network can be determined by an exponential moving average of the weights of the target neural network in the current and previous training steps. That is, the replacement neural network is determined by the conditional distribution

[0091]

number

[0092] can be modeled as the weights of the alternative neural network at the T-th training step.

[0093]

number

[0094] is determined by an exponential moving average.

[0095]

number

[0096] α∈[0,1] is a weighting parameter, and θ i is the weight of the target neural network at the ith training step.

[0097] The system processes each network output from the target neural network and the corresponding network output from the alternative neural network at the time step using a pairwise preference function to determine a preference score for the pair (step 504).

[0098] In some implementations, the system can determine a likelihood score for each network output according to the target neural network (step 506). The likelihood score for a particular network output by the target neural network indicates the likelihood of the network output under the conditional distribution modeled by the target neural network, given the network input corresponding to the particular network output. For example, if the target neural network is modeled according to the conditional distribution p θwhere θ is the weight of the target neural network (e.g., y~p θ (·|x)), the likelihood score of y according to the target neural network can be: p θ (y|x)

[0099] In some implementations, the network output can be an output sequence. For example, a particular network output, y, can be a sequence of N elements, y=(y,...,y N ) The target neural network can generate an output sequence autoregressively, with outputs y = (y1, ..., y N ) may be the following likelihood score by the target neural network:

[0100]

number

[0101] In the formula, p θ (y i |x,y j<i ) is the network input x and the preceding element y of the output sequence j<i The i-th element y of the target neural network given i is the likelihood score of

[0102] In some implementations, the system can determine a likelihood score for each network output according to a reference neural network for the training step (step 508). The reference neural network can be any generative model suitable for sampling network outputs conditional on network inputs.

[0103] For example, the reference neural network may be a network trained to perform a conditional generation task with a different network architecture compared to the target neural network. As another example, the reference neural network may have the same network architecture as the target neural network. As a further example, the reference neural network may have the same architecture as the target neural network using different hyperparameters (e.g., trained with different hyperparameters, performed inference using different hyperparameters, etc.), such as, for example, network size, network depth, number of inputs, learning rate, etc. As a further example, the reference neural network may be a reparameterization of the target neural network with different network weights, or may be a copy of the target neural network fine-tuned on a set of training data specific to the reference neural network. As a further example, the reference neural network may process modified network inputs compared to the target neural network (e.g., by conditionally generating network outputs based on modified neural network inputs that specify generating outputs in a different style than the target neural network, by processing modified input prompts compared to the target neural network, etc.).

[0104] The likelihood score for a particular network output according to a reference neural network at a time step indicates the likelihood of the network output under the conditional distribution modeled by the reference neural network, given the network input corresponding to the particular network output. For example, if q denotes the conditional distribution modeled by the reference neural network, the likelihood score of network output y given network input x according to the reference neural network may be: q(y|x)

[0105] The reference neural network for each time step can be determined based on the target neural network. In particular, the reference neural network can have the same network architecture as the target neural network and can have weights of an alternative neural network determined based on the weights of the target neural network in the current and previous training steps. As an example, the weights of the reference neural network can be the weights of the target neural network from the previous time step (e.g., the weights of the reference neural network in the Tth training step,

[0106]

number

[0107] is θ T-K , which are the weights of the target neural network at the TK-th time step).

[0108] As another example, the weights of the reference neural network can be determined by a moving average of the weights of the target neural network in the current and previous training steps. That is, the reference neural network is determined by the conditional distribution

[0109]

number

[0110] can be modeled as the weights of the reference neural network at the T-th training step.

[0111]

number

[0112] is determined by a moving average.

[0113]

number

[0114] K>0 is the window length, and θ i is the weight of the target neural network at the ith training step.

[0115] As another example, the weights of the reference neural network can be determined by an exponential moving average of the weights of the target neural network in the current and previous training steps. That is, the reference neural network is determined by the conditional distribution

[0116]

number

[0117] can be modeled as the weights of the reference neural network at the T-th training step.

[0118]

number

[0119] is determined by an exponential moving average.

[0120]

number

[0121] α∈[0,1] is a weighting parameter, and θ i is the weight of the target neural network at the ith training step.

[0122] The reference neural network can model a distribution determined by a fixed reference distribution μ(y|x). When the reference neural network is determined by a fixed reference distribution μ(y|x), the system can use the reference neural network to regularize the training of the target neural network. For example, the fixed reference distribution can be a previously trained neural network that has been verified to produce safe or desirable network outputs.

[0123] The reference neural network can model a distribution based on a mixture of the target neural network and a fixed reference distribution μ(y|x).

[0124] As an example, the reference neural network can be determined as an additive mixture between the target neural network and a fixed reference distribution μ(y|x). For example, using a weighting parameter A∈[0,1], the reference neural network is determined as follows: q(y|x)=Ap θ (y|x)+(1-A)μ(y|x)

[0125] As another example, the reference neural network can be determined as a geometric mixture between the target neural network and a fixed reference distribution μ(y|x). For example, using a weighting parameter β∈[0,1], the reference neural network can be determined as follows:

[0126]

number

[0127] where c(x) is a normalization constant based on the network input x. In particular, c(x) can be defined as:

[0128]

number

[0129] When the network output is an output sequence, the reference neural network can autoregressively process the output sequence, generating the output y=(y1,…,y N ) may be the likelihood score by the reference neural network:

[0130]

number

[0131] In the formula, q(y i |x,y j<i ) is the network input x and the preceding element y of the output sequence j<i The i-th element y given by the reference neural network i is the likelihood score of

[0132] For example, when the reference neural network is a geometric distribution between the target neural network and a fixed reference distribution μ(x|y) using weighting parameters β∈[0,1], the i-th element y i The likelihood score of can be determined as follows:

[0133]

number

[0134] In the formula, c(x,y j<i ) is the network input x and the predecessor element y j<i In particular, c(x) can be defined as follows:

[0135]

number

[0136] In some implementations, the reference neural network is an alternative neural network.

[0137] The system then updates the target neural network using the objective function to perform the conditional sampling task (step 510).

[0138] The objective function includes a first term that encourages preferring network outputs from the target neural network over corresponding network outputs from alternative neural networks according to preference scores.

[0139] In some implementations, the objective function can include a second term that penalizes the target neural network for producing network outputs having likelihood scores from the target neural network that deviate from corresponding likelihood scores from the reference neural network.

[0140] For example, using a weighting parameter η∈[0,1], the objective function may be:

[0141]

number

[0142] where x is the network input, p θ is the conditional distribution modeled by the target neural network, p' is the conditional distribution modeled by the alternative neural network, q is the conditional distribution modeled by the reference neural network,

[0143]

number

[0144] is the preference score determined for the target network output y relative to the alternative network output y', and KL is the Kullback-Leibler divergence.

[0145] The system can determine the objective function as the sum of the loss terms for each network input x. As an example, the system can determine the loss terms for the target neural network output y and the alternative neural network output y' given the network input x as follows:

[0146]

number

[0147] loss term

[0148]

number

[0149] encourages the target neural network to produce outputs that are preferred over those produced by the alternative neural network.

[0150]

number

[0151] penalizes the target neural network for producing network outputs having likelihood scores from the target neural network that deviate from the corresponding likelihood scores from the reference neural network.

[0152] In some implementations, the system can update the target neural network by calculating the gradient of the objective function with respect to the target neural network weights θ. For example, the system can determine the gradient of the objective function as the sum of the gradients of the loss terms for each network input, as follows:

[0153]

number

[0154] As a specific example, preference scores

[0155]

number

[0156] is a value between 0 and 1 indicating the likelihood that y is a higher quality output than y', and the reference neural network is a geometric mixture between the target neural network with weighting parameter β and a fixed reference distribution μ(y|x), the system can determine the gradient of the loss term for each network input as follows:

[0157]

number

[0158] The system may update the weights of the alternate and reference neural networks (step 512). The system may determine whether to update the weights of the alternate and reference neural networks based on any of a variety of criteria. For example, the system may update the weights of the alternate and reference neural networks after each training step.

[0159] In particular, the system can use the updated target neural network weights for the training step to update the weights of the alternative and reference neural networks. For example, if the alternative neural network is determined based on an average of the target neural network weights over multiple time steps, the system can update the weights of the alternative neural network to include the updated target neural network weights. As another example, if the reference neural network is determined based on an average of the target neural network weights over multiple time steps, the system can update the weights of the reference neural network to include the updated target neural network weights. As another example, if the reference neural network is determined based on a mixture distribution between the target neural network and a fixed reference distribution, the system can update the mixture distribution of the reference neural network to include the updated target neural network.

[0160] This specification uses the term "configured" with respect to systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed on it software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0161] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver apparatus for execution by a data processing apparatus.

[0162] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0163] A computer program, also referred to or sometimes written as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative, or procedural, and can be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not necessarily, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a file dedicated to the program, or in multiple coordinated files, such as a file storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0164] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and executed on the same one or more computers.

[0165] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0166] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from, transfer data to, or both of, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few.

[0167] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0168] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending and receiving documents from a device used by the user, such as by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a reply message from the user in return.

[0169] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of the machine learning training or production, i.e., inference, workload, for example.

[0170] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0171] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component such as an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0172] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., the result of a user interaction, can be received at the server from the device.

[0173] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features that are described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features are described above as acting in some combinations and are initially claimed as such, in some cases, one or more features from a claimed combination can be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0174] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence shown, or that all of the illustrated operations be performed to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be incorporated together in a single software product or packaged in multiple software products.

[0175] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0176] 100 Training System 102 Target Neural Network 106 Weight Update System 108 Network Input 109 Pairwise Preference Functions 110 Alternative Neural Networks 111 Preference Score 112 Target Network Output 114 Alternate Network Output 116 Weight Update 202 Generative Model 204 Output Samples 206 Sample Likelihood System 208 Network Output 210 Likelihood Score 300 processes 400 processes 500 processes

Claims

1. 1. A method, executed by one or more computers, for training a target neural network, the target neural network having a plurality of target neural network weights and configured to process network inputs according to the target neural network weights to generate network outputs, wherein in each of a plurality of training steps: receiving one or more network inputs; for each of the one or more network inputs: processing the network inputs with the target neural network to generate a first network output; processing the network inputs using an alternative neural network for the training step to generate a second network output; applying a preference function to the first network output and the second network output to generate a preference score indicating the likelihood that the first network output is a higher quality output than the second network output; updating weights of the target neural network using an objective function including a first term that encourages preference of the first network output over the corresponding second network output according to the preference score; A method comprising:

2. In each of the plurality of training steps, and for each of the one or more network inputs, determining a first likelihood score to be assigned by the target neural network to the first network output given the network input; determining a second likelihood score to be assigned to the first network output by a reference neural network for the training step given the network input; further comprising the objective function includes a second term that penalizes the target neural network for producing a first likelihood score that deviates from a corresponding second likelihood score; The method of claim 1.

3. 3. The method of claim 2, wherein in each of the plurality of training steps, the reference neural network for the training step is the alternative neural network for the training step.

4. applying the preference function to the first network output and the second network output to generate the preference score indicating the likelihood that the first network output is a higher quality output than the second network output; processing the first network output and the second network output using a preference model neural network to generate preference network outputs, the preference model neural network being trained to process pairs of neural network outputs and to generate corresponding preference network outputs for each processed pair of neural network outputs that indicate a likelihood that one member of the processed pair is a higher quality output than the other member of the processed pair; generating the preference score based on the preference network outputs, the preference score indicating the likelihood that the first network output is a higher quality output than the second network output; 2. The method of claim 1, comprising:

5. In each of the plurality of training steps, and for each of the one or more network inputs, determining the second likelihood score to be assigned to the first network output by a reference neural network for the training step given the network input, comprises: determining the second likelihood score to be assigned to the first network output based on a geometric mixture of the first likelihood score of the first network output and a likelihood score of the first network output under a fixed reference distribution and based on a normalization constant for the training step; 3. The method of claim 2, comprising:

6. In each of the plurality of training steps, the alternative neural network for the training step includes a plurality of alternative neural network weights for the training step and has the same architecture as the target neural network; the weights of the alternative neural network for the training step are determined by an exponential moving average of the weights of the target neural network over the current and previous training steps; The method of claim 1.

7. The objective function is, for each network input and corresponding first and second network outputs: a preference score determined by the preference function that indicates the likelihood that the first network output is a higher quality output than the second network output; a first likelihood score of the first network output determined by the target neural network; and a second likelihood score of the first network output determined by a fixed reference distribution; and The method of claim 1, based on

8. updating weights of the current target neural network using the objective function including a first term that encourages a preference for a first network input over a corresponding second network input according to the preference score; determining the gradient of the objective function with respect to the weights of the target neural network; updating the weights of the target neural network based on the gradient of the objective function; 2. The method of claim 1, comprising:

9. The step of determining the gradient of the objective function with respect to the current target neural network weights comprises, for each network input and corresponding first and second network outputs: a gradient of a first likelihood score of the first network output as determined by the target neural network with respect to the current target neural network weights; a preference score determined by the preference function that indicates the likelihood that the first network output is a higher quality output than the second network output; the first likelihood score of the first network output determined by the target neural network; and a likelihood score of the first network output determined by a fixed reference distribution; and The method of claim 8, comprising the step of calculating:

10. The method of claim 1 , wherein the target neural network is a large language model.

11. The method of claim 1 , wherein the preference neural network is a large-scale language model.

12. the first network output comprises a first network output sequence; the second network output comprises a second network output sequence; processing the network inputs using the alternative neural network to generate the second network outputs comprises using the alternative neural network to autoregressively generate the second network output sequence; The method of claim 2.

13. In each of the plurality of training steps, and for each of the one or more network inputs, determining the second likelihood score to be assigned to the first network output by a reference neural network for the training step given the network input, comprises: For each output element of the first network output sequence: determining a first likelihood score for the output elements using the target neural network given the network inputs; determining second likelihood scores for the output elements based on a geometric mixture of the first likelihood scores for the output elements and likelihood scores for the output elements under a fixed reference distribution and based on a normalization constant for the training step; determining the second likelihood score to be assigned to the first network output based on the second likelihood scores of the output elements in the first network output sequence; 13. The method of claim 12, comprising:

14. The method of claim 1 , wherein the preference function is capable of modeling a non-transitive preference relationship.

15. one or more computers; and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a target neural network, the target neural network having a plurality of target neural network weights and configured to process network inputs according to the target neural network weights to generate network outputs, the operations comprising, in each of a plurality of training steps: receiving one or more network inputs; for each of the one or more network inputs: processing the network inputs using the target neural network to generate a first network output; processing the network inputs using an alternative neural network for the training step to generate a second network output; applying a preference function to the first network output and the second network output to generate a preference score indicating the likelihood that the first network output is a higher quality output than the second network output; updating weights of the target neural network using an objective function including a first term that encourages preference of the first network output over the corresponding second network output according to the preference score; system.

16. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a target neural network, the target neural network having a plurality of target neural network weights and configured to process network inputs according to the target neural network weights to generate network outputs, the operations comprising, in each of a plurality of training steps: receiving one or more network inputs; for each of the one or more network inputs: processing the network inputs using the target neural network to generate a first network output; processing the network inputs using an alternative neural network for the training step to generate a second network output; applying a preference function to the first network output and the second network output to generate a preference score indicating the likelihood that the first network output is a higher quality output than the second network output; updating weights of the target neural network using an objective function including a first term that encourages preference of the first network output over the corresponding second network output according to the preference score; one or more non-transitory computer storage media,

17. said operation comprising: in each of said plurality of training steps, and for each of said one or more network inputs: determining a first likelihood score to be assigned by the target neural network to the first network output given the network input; determining a second likelihood score to be assigned to the first network output by a reference neural network for the training step given the network input; the objective function includes a second term that penalizes the target neural network for producing a first likelihood score that deviates from a corresponding second likelihood score; 16. One or more non-transitory computer storage media according to claim 15.

18. applying the preference function to the first network output and the second network output to generate the preference score indicating the likelihood that the first network output is a higher quality output than the second network output; processing the first network output and the second network output using a preference model neural network to generate preference network outputs, the preference model neural network being trained to process pairs of neural network outputs and generate a corresponding preference network output for each processed pair of neural network outputs that indicates a likelihood that one member of the processed pair is a higher quality output than the other member of the processed pair; generating the preference score based on the preference network outputs, the preference score indicating the likelihood that the first network output is a higher quality output than the second network output; 16. One or more non-transitory computer storage media according to claim 15, comprising:

19. The objective function is, for each network input and corresponding first and second network outputs: a preference score determined by the preference function that indicates the likelihood that the first network output is a higher quality output than the second network output; a first likelihood score of the first network output determined by the target neural network; and a second likelihood score of the first network output determined by a fixed reference distribution; and 16. One or more non-transitory computer storage media according to claim 15,

20. 16. The one or more non-transitory computer storage media of claim 15, wherein the preference functions are capable of modeling non-transitive preference relationships.

Images