Training neural network systems using negative examples

Abstract

Methods, systems, and computer program code for training a neural network system. The neural network system is configured to process a policy input to generate a policy output, according to a policy that represents, or is defined by, a probability distribution of the policy output given the policy input. The method involves obtaining a reference policy representing a reference probability distribution, and training data items, each comprising an example policy input and a corresponding example policy output. At least one of the training data items is identified as a negative example. The neural network system is trained using the training data items to update the policy towards an improved policy that minimizes a likelihood of the training data item(s) identified as negative examples. The training includes penalizing a divergence from the reference policy.

Description

ATTORNEY DOCKET NO.45288-0489WO1 TRAINING NEURAL NETWORK SYSTEMS USING NEGATIVE EXAMPLES CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 684,239, filed on August 16, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to generating data using machine learning models.

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. SUMMARY

[0005] This specification describes systems and methods implemented as computer programs on one or more computers in one or more locations that can train, e.g. fine-tune, a neural network system using one or more negative examples, i.e. examples of undesirable outcomes. The described techniques do not require a balanced dataset of positive and negative examples and, surprisingly, can train using just negative examples. The neural network system can be, e.g., a generative neural network system or a reinforcement learning system. The specification further describes the use neural network systems trained in this way.

[0006] According to a first aspect there is provided a computer-implemented method of training a neural network system. The neural network system is configured to process a policy input to generate a policy output, according to a policy that represents, or is defined by, a probability distribution of the policy output given the policy input.

[0007] The method involves obtaining a reference policy representing a reference probability distribution, and training data items, each comprising an example policy input and a corresponding example policy output. At least one of the training data items is identified as a negative example. The neural network system is trained using the training data items to update the policy towards an improved policy that minimizes a likelihood of the training data item(s)ATTORNEY DOCKET NO.45288-0489WO1 identified as negative examples. The training includes penalizing a divergence from the reference policy.

[0008] There is also described a computer-implemented method of generating a data item, comprising obtaining a generative neural network system that has been trained as described above, obtaining a conditioning input, e.g. an input token sequence including the conditioning input, and processing the conditioning input using the trained generative neural network system to generate the data item, e.g. an output token sequence representing the data item, based on the conditioning input.

[0009] There is further described a computer-implemented method of generating a data item that involves obtaining a generative neural network system that has been trained to update the data item generation policy towards an improved policy that minimizes a likelihood of one or more training data items identified as negative examples whilst penalizing a divergence from a reference policy. The method involves obtaining the conditioning input for a data item to be generated, and processing the conditioning input using the trained generative neural network system to generate the data item.

[0010] In some applications the neural network system comprises an action selection policy neural network for selecting actions to be performed by an agent in an environment to perform a task. The policy input can then comprise an observation characterizing a state of the environment, and the policy output can comprise an action selection output for selecting an action to control the agent to perform the task.

[0011] There is also described a computer-implemented method of using an action selection policy neural network to select actions to be performed by an agent in an environment to perform a task, where the action selection policy neural network has been trained using a method as described above. The method involves receiving an observation characterizing a state of the environment, processing the observation using the trained action selection policy neural network to generate an action selection output, and controlling the agent using the action selection output.

[0012] According to another aspect, there is provided a system that includes one or more computers and one or more storage devices communicatively coupled to the one or more computers and storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the previously described method.ATTORNEY DOCKET NO.45288-0489WO1

[0013] According to another aspect, there is provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the previously described method.

[0014] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0015] The described techniques can be used where only one type of example is present, in particular only negative examples. They do not require paired positive and negative example policy outputs for a particular example policy input, although they can use paired examples. Implementations of the described techniques can directly increase the likelihood of positive examples compared to the reference policy prior, resulting in policy improvement. In implementations even training using just negative examples is enough to drive optimization, particularly when using a KL or other constraint to keep close to the reference policy. There is no restriction on the number of negative (or positive) examples that may be taken into account per example policy input.

[0016] As the described techniques can make use of examples of positive and negative outcomes, but do not require paired outcomes, they can make use of data for which, say, it is only known whether an example is either good or bad, without knowing about its relative preference to other examples. This also allows the techniques to naturally utilize unbalanced datasets, e.g. where, based on a particular example policy input, there are multiple positive or preferred examples for each negative or “dis-preferred” example, or vice-versa. In implementations there is a parameter (^) that can be used to control a contribution of the negative and positive, dis-preferred and preferred examples when both are available.

[0017] The described techniques are general and can be used, for example, with generative neural network systems such language models and diffusion models (e.g. the policy output can comprise tokens that define a generated data item); and with reinforcement learning systems, including in behavioral cloning and inverse reinforcement learning settings. They can improve training speed, and the ability to learn from negative examples can, e.g., help to improve exploration.

[0018] The described techniques can be particularly useful when training neural network systems to perform challenging tasks, e.g. where all generations are failures (negative examples). Such tasks can include, e.g., generating code in a computer language to perform a particular function (described by a prompt), solving math problems, and aspects of safety assurance.ATTORNEY DOCKET NO.45288-0489WO1

[0019] The techniques can be extended to multi-objective optimization, and other optimization settings. The described techniques have a strong theoretical basis and in implementations come with convergence guarantees.

[0020] Some implementations of the described techniques are adapted to use in a parallel computing system, which can further facilitate faster and more efficient training.

[0021] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIGS. 1A-1C show an example system for training neural network system, and examples of the neural network system.

[0023] FIG.2 is a flow diagram of an example process for training a neural network system.

[0024] FIG. 3 shows an example parallel processing computer system for training a neural network system.

[0025] FIG.4 is a flow diagram of an example process for generating a data item.

[0026] FIG. 5 is a flow diagram of an example process for selecting actions to be performed by an agent.

[0027] FIGS. 6A and 6B illustrate learning to control an agent using implementations of the described technique.

[0028] FIG.7 illustrates fine-tuning a language model using implementations of the described technique.

[0029] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0030] FIG. 1A shows a system 100 implemented as computer programs on one or more computers in one or more locations for training a neural network system 100. The system of FIG.1A can be implemented as one or more computer programs on one or more computers in one or more locations.

[0031] In general the neural network system 110 is configured to process a policy input 112 to generate a policy output 114, according to a policy that represents a probability distribution of the policy output given the policy input.ATTORNEY DOCKET NO.45288-0489WO1

[0032] As one example, illustrated in FIG. 1B, the neural network system 110 may comprise a generative neural network system 110. The policy can then comprise a data item generation policy, the policy input can comprise a conditioning input 112 for the generative neural network system, and the policy output can define a data item 114.

[0033] In this example the data item generation policy can refer to a (learned) probabilistic mapping from the conditioning input to the generated data item, encoded within the parameters, e.g. weights, of the generative neural network system. The generative neural network system 110 may comprise a data item generator neural network such as sequence-to-sequence neural network or a diffusion model neural network. For example the generative neural network system 110 can generate text in a natural language, or still or moving images, or both; e.g. the generative neural network system 110 can be a LLM (Large Language Model) or VLM (Vision Language Model).

[0034] As another example, illustrated in FIG. 1C, the neural network system 110 may comprise an action selection policy neural network 110. The action selection policy neural network 110 can be used to select actions to be performed by an agent 162 in an environment 160 to perform a task, in response to observations 112 characterizing a state of the environment. The policy input then comprises an observation 112 characterizing a state of the environment, and the policy output comprises an action selection output 114 for selecting an action to control the agent to perform the task.

[0035] In general the neural network system 110 can have any suitable architecture and can include, e.g., one or more feed forward neural network layers, one or more recurrent neural network layers, one or more convolutional neural network layers, one or more attention neural network layers, one or more normalization layers, and so forth. Merely as an example, the neural network system may comprise a Transformer neural network that is configured to process the policy input, which may comprise an input token sequence, to generate the policy output, which may comprise an output token sequence.

[0036] A Transformer-based architecture can be one in which is characterized by having a succession of self-attention neural network layers. A self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism over the attention layer input to generate an attention layer output for each element of the input. There are many different attention mechanisms that may be used.

[0037] The described techniques involve obtaining a reference policy representing a reference probability distribution.ATTORNEY DOCKET NO.45288-0489WO1

[0038] In some implementations obtaining the reference policy involves obtaining a reference neural network system 120 configured to process a reference policy input 122 to generate a reference policy output 124 according to the reference policy, which represents a probability distribution of the reference policy output given the reference policy input. In these implementations any (trained) reference neural network system 120 can be used; the reference neural network system 120 may, but need not, have the same neural network architecture as the neural network system 110 being trained. Obtaining the reference policy / reference neural network system is described further later.

[0039] The described techniques involve obtaining one or more training data items 132, e.g. from a training data store 130. Each training data item 132 comprises an example policy input and a corresponding example policy output. In implementations at least one of the training data items is identified as a negative example. Some, a majority of, or all of, the training data items 132 can be negative examples.

[0040] In this context a negative example can be an example policy output that has been labelled as one that should not be generated from the example policy input, or that is relatively less preferred to other policy outputs generated from the example policy input, e.g. where there should be a bias against generating the example policy output from the example policy input. A negative example is also referred to as a dis-preferred example.

[0041] Some implementations of the described techniques use both positive and negative examples, although this is not essential. That is, the neural network system 110 can be trained using just negative examples.

[0042] A positive example, where used, can be an example policy output that has been labelled as one that should be generated from the example policy input, or that is relatively more preferred to other policy outputs generated from the example policy input. A positive example is also referred to as a preferred example.

[0043] One way to obtain the training data items is for a human to label example policy outputs as negative or positive, or as examples of failure and success. Such an approach can be used, e.g., where the neural network system 110 is a generative system such as an LLM or VLM, and the example policy outputs are generated in response to prompt inputs. This can be referred to obtaining human feedback. Alternatively the labelling can be performed by an AI system such as the or another LLM or VLM, i.e. by obtaining AI feedback.

[0044] The training data items can be obtained by processing the example policy outputs to label the example policy outputs as either negative or positive (dis-preferred or preferred) using a trained reward model or classifier. For example, the use of reward models to identifyATTORNEY DOCKET NO.45288-0489WO1 preferred example outputs is known in the context of LLMs or VLMs, and such a reward model can be used to label generated data items as ither dis-preferred or preferred; or an LLM or VLM judge can be used.

[0045] More generally the training data items can be obtained using an evaluation function to evaluate the example policy outputs and generate a score, e.g. a scalar value. An example may be then labelled as negative by comparing the score against a relative baseline value or absolute baseline value. For example, when an example policy output is better than the baseline value it is labelled as preferred, otherwise it is labelled as dispreferred. An absolute baseline value may, e.g., be directly defined by a human user. A relative baseline value may be based on the scores of one or more other example policy outputs, e.g. on a moving average or in some other way.

[0046] Where the neural network system 110 comprises an action selection policy neural network the evaluation function may comprise a (learned) action value function, such as a Q- function. For example, in case of a Q-function two actions can be sampled, and the action with the higher Q-value accepted and the action with the lower Q-value is rejected.

[0047] The neural network system 110 is trained using a training engine 150, as described below.

[0048] FIG. 2 is a flow diagram of an example process for training a neural network system, e.g. the neural network system 110. The process of FIG. 2 may be implemented by one or more computers in one or more locations.

[0049] At step 200, the process obtains a reference policy representing a reference probability distribution, i.e. a prior.

[0050] As one example, the reference probability distribution can be that of the neural network system 110 at an earlier point in time. For example, the reference probability distribution can be defined by an earlier training iteration of neural network system 110, that is to be improved; or it can be a version of the neural network system 110 prior to a training (fine-tuning) step or process. That is, where the reference probability distribution is obtained from a reference neural network system, the reference neural network system can be the neural network system 110 at an earlier point in time. Such an approach can be used, e.g., in a setting that uses RLHF (Reinforcement Learning from Human Feedback) to improve an LLM or VLM.

[0051] As another example, in an agent control reinforcement learning setting, the reference probability distribution can be obtained from observations of the behaviour of other “expert” humans or other agents (behavioural cloning).ATTORNEY DOCKET NO.45288-0489WO1

[0052] Generally, the reference probability distribution can be any probability distribution that it is desired to improve. For example, the reference probability distribution can be one defined by any reference neural network system 120.

[0053] The method also obtains one or more training data items (step 202). As previously described, each training data item comprises an example policy input and a corresponding example policy output. At least one of the training data items is identified as a negative (dis- preferred) example.

[0054] The method trains the neural network system, using the training data items 132, to update the policy (step 204). In particular, the policy is updated towards an improved policy that minimizes a likelihood of the training data items identified as negative examples. In some implementations the training also includes penalizing a divergence from the reference policy.

[0055] Optionally the training can be repetitive, in particular iterative, in which case there may be only one, two, or a few training data items used per repetition or iteration i.e. per training step.

[0056] In some implementations, but not necessarily, the method can initialize the parameters the neural network system with the values of the corresponding parameters of the reference neural network system, e.g. when the reference neural network system is the neural network system 110 at an earlier time (training step). The neural network system 110 can be trained iteratively by, at each of a succession of training iterations, using the policy of the trained neural network system as the reference policy for a subsequent training iteration.

[0057] In some implementations, e.g. where negative and / or positive examples are derived from a Q-function, or from another model that scores policy input - policy output pairs, the improved policy can be used to update the Q-function (or other model) and this can then be used to once again improve the policy. That is, in a reinforcement learning setting, the Q- function can be learned in parallel with the policy optimization. Any suitable reinforcement learning technique can be used; as an illustration, MPO (Maximum a Posteriori Policy Optimization, Abdolmaleki et al.2018, arXiv:1806.06920).

[0058] In some implementations of the method there can also be one or more positive training examples. As previously described, a positive example can be an example policy output has been labelled as one that should be generated from the example policy input, or is relatively more preferred to other policy outputs generated from the (same) example policy input, e.g. where there should be a bias towards generating the example policy output from the example policy input.ATTORNEY DOCKET NO.45288-0489WO1

[0059] Implementations of the described techniques can be used to train the neural network system 110 using only negative, dis-preferred examples, or the neural network system 110 can be trained using a mixture of positive and negative examples, or the neural network system 110 can be trained using only positive examples. A mixture of positive and negative examples used to train the system can be unbalanced, i.e. can have different numbers of positive and negative examples.

[0060] The described techniques may be used to train a neural network system from scratch, i.e. from initial random initialization of the learnable parameters, or they can be to fine-tune a pre-trained neural network system, to improve a quality of the policy outputs generated. For example, the policy of the neural network system 110 can be initialized to the reference policy, e.g. by initializing learnable parameters of the neural network system to the values of corresponding parameters of the reference neural network system 120.

[0061] As previously described, in some implementations the negative examples and / or positive examples used to train the system can be labelled as negative examples and / or as positive examples by one or more humans or machines. Where the neural network system 110 comprises a generative neural network system the generated data items may be labelled as negative or positive. Where the neural network system comprises an action selection policy neural network an action, or a trajectory comprising multiple actions, may be labelled as negative or positive, e.g. in a behaviour cloning or other setting.

[0062] In some implementations, a training data item 132 may be labelled as positive or negative based on a model such as a preference model, reward model, or Q-function that, e.g., takes as input a policy input and a corresponding policy output and generates a score or other output that can be used to identify a training comprising the policy input and corresponding policy output as positive or negative.

[0063] In general training the neural network system 110 comprises updating values of learnable parameters, e.g. weights, of the neural network system using an objective function. This can involve backpropagating gradients of the objective function to update the learnable parameters using any appropriate gradient descent optimization algorithm, e.g. Adam or another optimization algorithm.

[0064] The objective function can have a first term that depends on a likelihood of the example policy output according to the policy given the example policy input. The likelihood can, e.g., be evaluated for each of the training data items 132 identified as negative examples, e.g. by summing a log likelihood of each of the training examples according to the policy, moreATTORNEY DOCKET NO.45288-0489WO1 particularly by summing, for each of the negative examples, a log likelihood of the respective example policy output given the example policy input.

[0065] The objective function can have a second term that measures a difference between the probability distribution and the reference probability distribution.

[0066] In some implementations the first term and the second term have the same sign. For example, the objective function may comprise a loss function and the first term and the second term can both be positive; or the objective function can define an objective to be maximized and the first term and the second term can both be negative.

[0067] In some implementations the second term is evaluated using each of (all of) the training data items 132, i.e. not just for the negative examples, but also the positive examples (where present).

[0068] For each of the training data items 132 identified as negative examples the example policy input can be processed using the neural network system to determine the likelihood of the respective example policy output given the example policy input.

[0069] A likelihood value for a policy output ^ given a policy input ^ may be written as^^^|^^, where ^^∙ | ∙^ represents the policy of the neural network system. In someimplementations, e.g. where the neural network system 110 comprises a generative neural network system 110 such as an LLM or VLM, the neural network system 110 can generate a sequence of outputs. For example a sequence of output tokens ^^, for the policy output ^, canbe generated, e.g. autoregressively, as ^^^|^^ = ∏^^^^ ^^^^^^, ^^^^^ for a policy output ^ =^^^, … , ^^^ given the policy input, ^, wherescore of the ^-thelement, ^^according to the neural network system. In this case the likelihood of a particular output token ^^can be determined by processing ^ and the previous tokens, ^^^^, of ^, todetermine ^^^^^^, ^^^^^. The likelihood of the sequence of tokens of the example policy outputunder the policy can be determined from the likelihood of each particular output token, ^^, ofthe example policy output ^ according to ^^^^^^, ^^^^^, e.g. by summing the log likelihood ofeach particular output token, ^^.

[0070] As an example, the second term of the objective function can be determined by processing the example policy input of each of the training data items 132 using the neural network system 110 to generate a respective example policy output, and processing the example policy input of each of the training data items using the reference neural network system 120 to generate a respective reference example policy output. The second term can then be determined from a measure of a difference, e.g. a KL divergence, between theATTORNEY DOCKET NO.45288-0489WO1 probability distribution obtained from the neural network system and the reference probability distribution obtained from the reference neural network system, i.e. from the example policy outputs and from the reference example policy outputs obtained by the processing.

[0071] In some implementations, e.g. where the (reference) neural network system 120 comprises a (reference) policy neural network that generates a (reference) policy output defining a categorical distribution, this can involve comparing the categorical distributions.

[0072] In some implementations the measure of the difference, e.g. the KL divergence, may be determined from a Monte Carlo estimate. Such a Monte Carlo estimate can be based on the log likelihood of a single (reference) policy output, or on an average of multiple samples.

[0073] In some implementations the policy input 112, the policy output 114, the reference policy input 122, and the reference policy output each comprise a respective sequence of tokens. This can be the case where, e.g., the neural network system 110 comprises a generative neural network system such as an LLM or VLM. Determining the measure of the difference between the probability distribution and the reference probability distribution can then be performed as follows.

[0074] A reference policy sequence of tokens for the reference example policy output can be generated using the reference neural network system by iteratively, e.g. autoregressively, for each token of the reference policy sequence of tokens, processing i) an input sequence of tokens representing the example policy input and ii) previously generated tokens of the reference policy sequence of tokens, to generate the, e.g. categorical, reference probability distribution for a current token of the reference policy sequence of tokens. The current token of the reference policy sequence of tokens can then be selected using the reference probability distribution, e.g. by sampling from the distribution or by choosing a token with a maximum likelihood.

[0075] For each token of the reference policy sequence of tokens, the input sequence of tokens and the previously generated tokens of the reference policy sequence of tokens can be processed using the neural network system to generate the, e.g. categorical, probability distribution (of the policy) corresponding to current token of the reference policy sequence of tokens.

[0076] The measure of the difference between the probability distribution and the reference probability distribution for the second term of the objective function can be determined by summing, for each token of the reference policy sequence of tokens, a measure of a difference between a) the reference probability distribution for the current token of the reference policyATTORNEY DOCKET NO.45288-0489WO1 sequence of tokens and b) the probability distribution corresponding to current token of the reference policy sequence of tokens.

[0077] As previously described, at least one of the training data items 132 can be identified as a positive example. The neural network system can then be trained using the training data items 132 to update the policy towards an improved policy that maximizes a likelihood of the training data items identified as positive examples.

[0078] The objective function can then include a third term that depends on a likelihood of the example policy output according to the policy, given the example policy input, evaluated for each of the training data items identified as positive examples. The third term has an opposite sign to the sign of the first and second terms. Where there are multiple such positive examples, their log likelihoods may be summed.

[0079] In some implementations each of the training data items 132 is identified as either a negative example or a positive example. The method can then involve further setting a thirdrelative weight, ^, of the third term and setting a first relative weight, ^1 − ^^, of the first term.This facilitates controlling a relative contribution of the negative and positive examples toimproving the policy. In general 0 ≤ ^ ≤ 1. Some particular example options are ^ = 0(learning solely from dis-preferences), ^ = 1 (learning solely from preferences), ^ < 0.5, ^ >0.5, or ^ = 0.5. In some implementations a value of ^ can be varied during the trainingprocess, e.g. to increase or decrease ^ during training.

[0080] In some implementations the objective function (to be maximized; or minimized with signs reversed) is given by: ^^ log ^^^^# ^^^^ − ^1 − ^^ ^ log ^^^^ ^$^^ ^ − %&'^^$()||^^where ^corresponding example policy output for a positive example, ^$the corresponding example policy output for a negative example, ^^^|^^ denotes a likelihood of the corresponding example policy output given the example policy input according to the policy of the neural network system, ^ (^^^|^^) represents the probability distribution of the policy of the neural network system, ^$()(^$()^^|^^) represents the reference probability distribution, &'^∙^ refers to Kullback Leibler% is a weight (here referred to as a second relative weight), and ^ is also a weight (in general a different weight, here referred to as the third relative weight). In particular ^ canbe a relative weight of the third term, and ^1 − ^^ can be a relative weight of the first term.ATTORNEY DOCKET NO.45288-0489WO1

[0081] In some implementations &'^^$()|^^may be determined as a Monte Carlo estimate based on one, or a few, as &'^^$()||^^ = &' *^$()^^|^^||^+^^|^^, =-.log ^$()^^|^^ − log ^+^^|^^ / where -^ and()^∙ |^^. Where ^ comprises a sequence of tokens a value oflog ^+^^|^^ can be determined as described above (summing log likelihoods to score asequence generated by the reference policy).

[0082] The above example objective function can alternatively be expressed as:network system 110, which has learnable parameters 3, ^~4 denotes sampling from the training data items, ^, from the training data store 130, 4, where 4#denotes accepted (preferred) examples and 4$denotes rejected (dis-preferred) examples; and the remaining terms are as previously described. As previously described ^ can take any value between 0 and 1, i.e. the training can be performed using only positive, preferred examples (training data items) or using inly negative, dis-preferred examples, or using both.

[0083] The above described objective functions have no limitation on the number of positive and / or negative examples per policy input, ^, e.g. per query in the case of an LLM or VLM.

[0084] In some implementations the value of the second relative weight %, is set to the value of ^. In some implementations the second relative weight, %, of the second term can be sufficiently high to, on average, omit just the negative examples from the probability distribution of the policy output of neural network system 110 after training, substantially removing the dis-preferred outcomes from the prior.

[0085] In general the value of % can be chosen empirically. As one example a value of % ≥ 1can assist learning when training using predominantly negative, dis-preferred examples. As another particular example a value of % > ^1 − ^^ can be chosen. As a further example thevalue of % can be chosen using a parameter sweep over a range; e.g. from 0; e.g. to 1, 2, 5, or 10.

[0086] The described techniques lower the contribution of, and can substantially remove, negative examples from the probability distribution of the policy of the (trained) neural network system 110, whilst retaining useful policy outputs (or positive examples). The objective canATTORNEY DOCKET NO.45288-0489WO1 be seen as a subtraction operation in distributional space, removing negative, dis-preferred examples from the reference policy prior.

[0087] In the above example objective functions, the &'^∙^ term penalizes divergence from the reference policy. Other measures of a difference between the policy and reference probability distributions can be used. Penalizing divergence from the reference policy facilitates learning from negative, dis-preferred examples as it can help avoid arbitrary solutions where probability could otherwise be assigned to random out-of-distribution responses. In implementations where the neural network system 110 is trained iteratively, the policy of the neural network system 110 is constrained towards the slowly changing reference policy, which also facilitates learning from negative, dis-preferred examples.

[0088] In general the training can be performed online or offline. As previously described, in some implementations a training data item 132 is obtained by processing the example policy input, ^, using the reference neural network system 120 to generate the corresponding example policy output. In some implementations training data items may be obtained independently, e.g. from a corpus of labelled training data items.

[0089] In a generative model setting, e.g. an LLM or VLM setting, a prompt can be provided to the reference neural network system 120 and one or more generated outputs (reference policy outputs 124) can be sampled from the generative model. These can be evaluated using a reward model to label them as (dis)preferred. Merely as an illustration, four generated outputs can be sampled from the generative model, ranked using a reward model, and then the top two labelled as preferred, and the bottom two as dispreferred.

[0090] In some implementations a training data item can be identified as a negative example by labelling the training data item dependent on the corresponding example policy output. As previously described, such labelling may be performed by a human and / or by a machine, e.g. by a trained model neural network, in some implementations by a trained language model.

[0091] For example, an example policy input can be processed using the reference neural network system 120 to generate (at least) first and second corresponding example policy outputs. The example policy input can be paired with one of the first and second corresponding example policy outputs to obtain a training data item, or with each of the first and second corresponding example policy outputs to obtain two training data items. A relative preference of the first and second corresponding example policy outputs can then be determined, e.g. by determining a relative preference of the one of the first and second corresponding example policy outputs in the pairing compared with the other of first and second corresponding example policy outputs, e.g. by human and / or machine. The training data item can then beATTORNEY DOCKET NO.45288-0489WO1 identified as a negative example or as a positive example, e.g. the training data item can then be identified as either a negative example or a positive example, by labelling the training data item according to the determined relative preference. Such an approach can be used to obtain negative examples, or positive examples, or both.

[0092] As an particular example, one way of using an evaluation function or “scoring model” to obtain preference information is given by the process below: This process can be used in various settings including an agent control reinforcement learning (RL) setting, and for optimization of an LLM or VLM using RLHF. The evaluation function may be a learned evaluation function such as a preference or reward model, or Q-value function. The baseline can be absolute or relative (e.g. a learned mean or median value for the input ^).

[0093] As another example, a training data item can be identified as a positive or as a negative example by labelling the training data item, manually or automatically, dependent on a process performed using the corresponding example policy output; or where the corresponding example policy output defines the result of a process performed on the example policy input, on that result.

[0094] A few particular examples follow; these examples can be used with manual or automatic labelling of responses or actions.

[0095] As one example, the policy output can define a data item comprising code in a computer language, such as a programming language, e.g. generated by a language model or VLM. In general the code may be code that, when executed by a computer, causes the computer to carry out a particular process or task. The particular process, or task, may be specified by the policy input, e.g. a prompt in the case of an LLM or VLM.

[0096] The code may be complied, or executed (which may involve a compiling step), to determine a label or a score for determining a label for the policy output defining the data item. For example, if the code cannot be compiled and / or executed, or if it performs the intended process or task poorly or not at all, it may be labelled as a negative example. Optionally if the code is executable and performs, the task it may be labelled as a positive example.ATTORNEY DOCKET NO.45288-0489WO1

[0097] An example (output) may be labelled as positive or negative according to whether the code (process or task) was successfully compiled or executed, or produced a correct result; or it may be labelled as positive or negative according to one or more metrics relating to execution of the process, such as memory usage or execution speed.

[0098] As another example, the policy output can define a data item that, in turn, defines an action or sequence of actions for an agent, e.g. a mechanical agent such as a robot. Then a label or a score for determining a label for the policy output defining the data item may be determined from whether or not the action(s) were successful in performing a task, or in achieving progress towards performing a task. Such an evaluation may be performed in simulation or in the real-world.

[0099] Also or instead, when the policy output defines a sequence of actions for controlling an agent, in particular a mechanical agent such as a robot, a label for the policy output may be determined according to whether or not one or more of the actions is safe. More specifically, the policy output can be labelled as negative when one or more actions it defines is outside a set of parameters that define a region of safe operation of the e.g. mechanical agent, e.g. robot, such as moving of a part of the robot to outside a defined (safe) region, or the robot applying greater than a defined (safe) force to an object. Optionally, if all the selected actions are safe the policy output can be labelled as positive.

[0100] In this way the agent, e.g. mechanical agent, e.g. robot, can be trained or fine-tuned to operate safely. More specifically the neural network system 110 can be trained to generate safe actions (according to a set of parameters that define a region of safe operation).

[0101] More generally, the described techniques can be used to train an action selection policy of the neural network system 110 to perform one or more tasks, e.g. by labelling actions as preferred or dis-preferred, e.g. using a (learned) Q-value neural network.

[0102] As a further example, where the corresponding example policy output defines the result of a mathematical process performed on the example policy input, the example may be labelled as negative (or positive) according to whether the result is incorrect (or correct).

[0103] As a further example, the corresponding example policy output for an example policy input can define the result of determining the structure of a chemical or biological entity defined by the example policy input, e.g. having a particular property or function specified by the example policy input. The or another chemical or biological entity, with a structure determined by the neural network system, may be synthesised according to the determined structure. The example may be labelled as positive or negative according to whether or not the structure ofATTORNEY DOCKET NO.45288-0489WO1 the chemical or biological entity is physically feasible, e.g. according to bond lengths, steric hindrances, and so forth, or physically synthesisable.

[0104] Parallel processing

[0105] Implementations of the method can be adapted to run on parallel processing computer system. FIG. 3 shows an example parallel processing computer system 300 configured to implement a version of the above-described method.

[0106] The parallel processing system computer 300 comprises a plurality of hardware computing devices 170A,B configured to operate in parallel. Each hardware computing device 170A,B may comprise a neural network accelerator, i.e. specialized hardware that is used to accelerate neural network computations, such as a GPU (Graphics Processing Unit) or TPU (Tensor Processing Unit). A typical neural network accelerator is configured to perform hardware matrix multiplications; it can include a set of one or more multiply accumulate units (MACs).

[0107] First and second respective instances of the reference neural network system 120A,B are maintained on the respective first and second respective hardware devices 170A,B. Each instance of the reference neural network system 120A,B has a respective reference policy input 122A,B and a respective reference policy output 124A,B.

[0108] The example policy input, ^, 122A,B, is processed using the first instance of the reference neural network system 120A on the first hardware device 170A to generate the first corresponding example policy output 124A and, in parallel, the example policy input is processed using the second instance of the reference neural network system 120B on the second hardware device 170B to generate the second corresponding example policy output 124B.

[0109] In such an environment two training data items 132 can be obtained, a negative example and a positive example, by labelling the training data item according to the determined relative preference of the first and second corresponding example policy outputs.

[0110] In some implementations the example policy input of, and the corresponding example policy output of, each of one or more of the training data items can be processed using a scoring model to generate a score for the training data item. The training data item can then be identified (labelled) as a negative (or positive) example dependent on the generated score, e.g. on an absolute score or on a relative score of the training data item with respect to another training data item. As previously described, the (trained) scoring model may comprise, e.g. a preference model, a reward model, or a Q-function, e.g. a Q-value neural network, that generates scores that are Q-values.ATTORNEY DOCKET NO.45288-0489WO1

[0111] In implementations this can involve processing the example policy input using the reference neural network system to generate at least two corresponding example policy outputs, the example policy input and each respective corresponding example policy output defining a respective training data item. The example policy input and the corresponding example policy output of each respective training data item can be processed using the scoring model to generate the score for the respective training data item. The respective training data item can then be labelled as either a negative example or a positive example dependent on the relative scores of the respective training data items.

[0112] Merely as an example, the top n scoring training data items (of a group of training data items) can be labelled as positive examples and / or the bottom m scoring training data items (ofa group of training data items) can be labelled as negative examples. Thus 6 + 8 instances ofthe reference neural network system can be run in parallel across a corresponding number of processors, e.g. neural network accelerators, to efficiently generate a large training dataset.

[0113] Such a parallel implementation facilitates efficient operation and scaling of a training pipeline in which multiple training data items are generated in parallel, and optionally also scored in parallel (by parallel instances of the scoring model), to generate training data for training the neural network system 110 as described above. Such an approach also reduces computational bottlenecks in the training process. Potentially multiple instances of the neural network system 110 can be implemented and trained in parallel, and afterwards their weights merged by averaging or in some other way.

[0114] FIG.4 is a flow diagram of an example process for generating a data item. The process of FIG.4 may be implemented by one or more computers in one or more locations.

[0115] A generative neural network system 110 (i.e. neural network system 110) is obtained, that is configured to process a conditioning input to generate a data item according to a data item generation policy. The generative neural network system 110 has been trained as described above, i.e. to update a data item generation policy of the system towards an improved policy that minimizes a likelihood of training data items identified as negative examples whilst penalizing a divergence from a reference policy. The process involves obtaining a conditioning input for a data item to be generated (step 400), and processing the conditioning input using the trained generative neural network system 110, to generate the data item (step 402).

[0116] FIG. 5 is a flow diagram of an example process for using an action selection policy neural network 110 (i.e. neural network system 110) to select actions to be performed by anATTORNEY DOCKET NO.45288-0489WO1 agent in an environment to perform a task. The process of FIG.5 may be implemented by one or more computers in one or more locations.

[0117] The action selection policy neural network 110 has been trained using a method as described above. The method involves obtaining an observation characterizing a state of the environment (step 500), e.g. from one or more sensors of the environment such as an image sensor. The observation is processed using the trained action selection policy neural network 110 to generate an action selection output (step 502). The agent is controlled using the action selection output (step 504), by selecting an action for the agent to perform using the action selection output, in particular in order to accomplish a specific task. The action may have multiple dimensions, e.g. defining multiple different movements for different parts of a mechanical robot. The environment can be a real-world environment and the agent can act in the real-world environment to perform the task. The action selection policy neural network 110 can have been trained in the real-world environment, or it can have been trained in a simulation of the real-world environment prior to being used in the real-world environment to perform the task.

[0118] FIGS. 6A and 6B illustrate the performance of implementations of the technique compared with some other approaches. FIGS.6A and 6B illustrate an implementation of neural network system 110 that is an action selection policy neural network, being trained to control a simulated humanoid agent to walk (a high-dimensional control task). In FIG.6A the actions are selected according to a categorical action selection policy (in which the policy output 114 comprises categorical logits over discrete actions). In FIG. 6B the actions are selected according to a continuous, Gaussian action selection policy (in which the policy output 114 defines the means and standard deviations of a multivariate Gaussian). In FIGS. 6A and 6B the x-axis shows the number of training steps, and the y-axis shows the rewards received when performing the selected actions.

[0119] Curve 600 illustrates an implementation of the described technique that learns from both positive and negative examples; curve 602 illustrates an implementation of the described technique that learns from just positive examples; curve 604 illustrates an implementation of the described technique that learns from just negative examples. Curve 606 is for DPO (Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, arXiv:2305.18290); dashed line 608 is for MPO (ibid), as a baseline reinforcement learning technique. It can be seen that the performance of the described techniques matches orATTORNEY DOCKET NO.45288-0489WO1 exceeds the strong MPO baseline, and that good performance can be achieved even when onlynegative examples are used for training (here with % > 1 for more effective learning).

[0120] FIG. 7 illustrates the performance of implementations of the technique in which the neural network system 110 is a language model (Google DeepMind “Gemma”) fine-tuned on a language alignment task. The x-axis shows the number of training steps, and the y-axis shows the win rate. The win rate is determined by performing comparisons against a baseline version of the model using held-out prompts (where the preferred output is judged using GPT-4). Curve 700 illustrates learning from both positive and negative examples; curve 702 illustrates learning from just positive examples; and curve 704 illustrates learning from just negative examples. It can be seen that the described technique improves on the baseline model, but is not restricted to balanced numbers of positive and negative examples. The eventual drop in curve 700 is due to reward hacking.

[0121] Example reward models for LLMs, VLMs, and other applications

[0122] Generally, techniques for obtaining reward models are known. A reward model can define a reward value for a combination of a policy input and a corresponding policy output; these may be defined by an input sequence of tokens and a corresponding output sequence of tokens. A reward model can process a combination of an input sequence and an output sequence to generate the reward value. The reward value can be e.g. a scalar number or score, or the reward model can have a categorical output that defines a category, e.g. dis-preferred or preferred. The reward model can comprise a reward model neural network. A reward model neural network can have any suitable architecture; merely as an illustration it can have a sequence processing architecture comprising a stack of transformer neural network blocks.

[0123] Some implementations of the described techniques use a reward model. As an example, one way in which this can be obtained is by training the reward model using a dataset of training examples each comprising a conditioning input, first and second training examples of a data item based on the conditioning input, and an indication of which of the first and second training examples is preferred (or a score for each). The preference, or score, can be according to one or more predetermined criteria such as accuracy (for text or, say, a regional accent for audio), realism (of an image), length, level of detail (of an image), helpfulness, safety, and so forth. In implementations where the neural network system 110 comprises a generative model the reward model can define, e.g., a score for a combination of a conditioning input and a data item that has been generated by the neural network system based on the conditioning input. As anATTORNEY DOCKET NO.45288-0489WO1 example, where a training example comprises text or an image the reward model may comprise an LLM or a VLM with an extra linear layer, e.g. a neural network head comprising a single neuron. The reward model can be trained using any suitable objective function to align the model output with the human (or AI) rankings, e.g. a variation of a cross-entropy loss, such as a Bradley-Terry loss, or more generally using a form of maximum likelihood estimation (MLE).

[0124] In principle an approach in which a training data item is labelled as a negative (or positive) example dependent on the (relative) score generated by a scoring model can be extended to generate a score that is dependent on multiple scoring models. Each of these scoring models can correspond to a different set of rewards or preferences, e.g. where optimization for multiple different objectives is desired. For example where the neural network system 110 comprises a generative neural network system, different scoring models may generate respective scores that depend on different characteristics of a generated data item. Where the neural network system comprises an action selection policy neural network for controlling an agent to perform a task different scoring models may generate respective scores that depend on, e.g. a degree of manner of achieving success and an energy used. The scores from the different scoring models can be combined to obtain a combined score for labelling a particular training data item as a negative example or as a positive example.

[0125] Example neural network systems.

[0126] As previously described, in some implementations the neural network system 110 comprises a generative neural network system, e.g. a data item generator neural network. The policy can then comprise a data item generation policy. The policy input can comprise a conditioning input for the generative neural network system, e.g. obtained from a user. The policy output can define a data item generated according to the data item generation policy as a response to the conditioning input, e.g. a text, audio, and / or still or moving image data item. The conditioning input may comprise a “prompt” for the data item generator neural network that, e.g., characterizes one or more desired properties for the generated data item. The conditioning input may comprise one or more of text, audio, and (still or moving) image data.

[0127] In some implementations the neural network system 110, e.g. the generative neural network system, is configured to process an input token sequence including the conditioning input, using the generative neural network system, e.g. autoregressively, to generate an output token sequence that defines the data item. This can involve using a policy neural network of the neural network system to generate, for each position in the output token sequence, a respective score for each token in a vocabulary of output tokens. The output token at theATTORNEY DOCKET NO.45288-0489WO1 position in the output token sequence can then be selected using the generated scores, e.g. by selecting based on a distribution of the scores over the vocabulary of tokens.

[0128] In some implementations the tokens can represent text, e.g., words, wordpieces or characters, in a natural or computer language. For example text may be received, e.g., as a series of encoded characters, e.g. UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like. A text encoder, i.e. a tokenizer, can process a sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g. that each represent words, wordpieces or characters in a natural or computer language. The computer language may be any formal language used to communicate with a computer, e.g. a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language. The tokenizer can, e.g., implement BPE (Byte Pair Encoding) or Wordpiece tokenization. Optionally the text can be obtained from audio data representing speech; the output tokens may be converted into audio data that represent speech corresponding to the text.

[0129] Also or instead the tokens may represent an image. For example a set (sequence) of input or output tokens can represent an image. Each image token may comprise a block encoding of values of the pixels in a different region of an image that maps a set of values of the pixels to a respective image token. The block encoder may comprise a neural network, e.g. having one or more (self-)attention layers, such as a Transformer neural network.

[0130] Also or instead the tokens may represent an audio waveform. For example a set (sequence) of input or output tokens can represent audio data representing an waveform e.g. instantaneous audio amplitude values or time-frequency audio data. Each image token may comprise a block encoding of the audio waveform in a different time segment of the audio that maps a set of values representing the audio waveform to a respective image token. The block encoder may comprise a neural network, e.g. having one or more (self-)attention layers, such as a Transformer neural network.

[0131] In a multimodal system audio data or an image may be flagged by a start-of-audio token or start-of-image token.

[0132] In some implementations such a generative neural network system can also or instead comprise a data item generator neural network that is a diffusion model neural network. In general a diffusion model neural network can be a neural network that has been trained to process a diffusion input comprising a current noisy data item and data specifying a current time to generate a diffusion output that defines an estimate (given the current time) of either a noise component of the current noisy data item, i.e. an estimate of the noise that has been addedATTORNEY DOCKET NO.45288-0489WO1 to an original data item to generate the current noisy data item; or of a de-noised version of the current noisy data item.

[0133] In some implementations the generative neural network system is a multimodal system that is configured to process a conditioning input comprising one or more of text data, audio data defining an audio signal (e.g. as amplitude values of the audio signal or as a time-frequency representation of the audio signal), or a still or moving image (e.g. as image pixel values), to generate a data item that can similarly comprise text data, audio data, or a still or moving image.

[0134] For example the conditioning input may comprise text and the data item may comprise an image or an audio signal that represents speech an image generated in response to the text, e.g. described by the text. Also or instead the conditioning input may comprise an audio signal that represents speech, or an image, and the data item may comprise text, e.g. that describes the conditioning input.

[0135] As another example the conditioning input may comprise an observation, e.g. of a real world environment, e.g. from sensor such as a camera or other image sensor; and optionally additional information such as information defining a particular task to be deformed. The output data item may comprise agent control data that defines one or more actions to be performed by an agent, e.g. by a mechanical agent such as a robot or autonomous vehicle, to perform a task. The reward model(s) may, e.g., define a preferred trajectory of motion of the mechanical agent in the (real-world) environment.

[0136] In some implementations the generative neural network system may comprise a language and / or image generation neural network system, e.g. an LLM or VLM, that may have been trained before being fine-tuned by the above described method.

[0137] The conditioning input may comprise a prompt, e.g. a natural or computer language prompt (such as text) for the generative neural network system. The generated data item may comprise a natural or computer language and / or (still or moving) image, and / or audio (audio waveform) response to the prompt. For example the generated data item may comprise a still or moving image, or audio, that has one or more one or more desired properties defined or characterized by the conditioning input. For example, the conditioning input can be or include a sequence of text and the generated data item can be a still or moving image (video) that is described by the text, i.e. that has one or more characteristics that are specified by the text. For example the conditioning input can be a caption for a generated (still or moving) image. The image may represent a real-world environment. As another example the conditioning input can comprise text and the generated data item may comprise audio that represents corresponding speech (text-to-speech); or vice-versa. As another example the conditioningATTORNEY DOCKET NO.45288-0489WO1 input can comprise an image (or audio) and the generated data item may comprise text that characterizes, or describes, the image (or audio) or part of the image (or audio).

[0138] In general, and as previously described, the generative neural network system can have any appropriate architecture for processing the conditioning input to generate the data item.

[0139] As one particular example, the generative neural network system may comprise an auto- regressive generative model (e.g., a Transformer, a recurrent neural network, etc.) that can auto-regressively generate an output sequence as the data item based on the conditioning input. The generative model can, for example, comprise a large language model (LLM) that can auto- regressively generate tokenized representations of text data, a vision-language model (VLM) that can auto-regressively generate tokenized representations of image or video data, e.g. in response to a text conditioning input or that can auto-regressively generate tokenized representations of text, e.g. in response to an image conditioning input, an audio language model that can auto-regressively generate tokenized representations of text data, or a multimodal model that can that can generate tokens representing any of text, image or audio, e.g. in response to a conditioning input comprising any of text, image or audio, and so forth.

[0140] As another particular example, the generative neural network system may comprise a diffusion model (e.g., a denoising diffusion model, a score-based diffusion model, a latent diffusion model, etc.) that can generate the data item by repeatedly transforming samples from a noise distribution (e.g., a Gaussian distribution) based on the conditioning input over a sequence of iterations. For example, the generative neural network system may comprise a diffusion model that transforms samples from the noise distribution using a denoising neural network with any appropriate architecture (e.g., a convolutional neural network, a recurrent neural network, etc.). Such a diffusion model may be used to generate, e.g., a still or moving (video) image.

[0141] As another particular example, the generative neural network system may comprise a neural network that can generate the data item by transforming samples from a noise distribution (e.g., a Gaussian distribution). The generative neural network system may comprise, e.g., a generator network of a generative adversarial network, a decoder of a variational auto-encoder, a normalizing flow, and so forth.

[0142] As used herein an image may be any still or moving image, i.e. the image may be part of a video, in 2D or 3D, and may be a monochrome, color or hyperspectral image, i.e. comprising monochrome or color pixels. As defined herein an “image” includes a point cloud e.g. from a LIDAR system, and a “pixel” includes a point of the point cloud. An image mayATTORNEY DOCKET NO.45288-0489WO1 have been captured by a camera or other image sensor from the real world; and objects in the image may comprise physical objects, represented by the image.

[0143] Example hardware implementations

[0144] In some implementations the generative neural network system, e.g. a language model or a visual language model, is stored on a user computing device, i.e. a device local to the user, such as a mobile device e.g. a mobile phone, or a smart speaker.

[0145] In some implementations the generative neural network system is implemented on a remove server in communication with a user computing device over a wired or wireless network communications link between the user computing device and the server.

[0146] The user computing device may be provided with an input mechanism, such as a text or voice interface, that enables user input from the user in a natural language. The user computing device may be provided with an output mechanism that provides a system output for the user in the or another natural language e.g. as speech or text; or in some other way, e.g. by displaying an image. The input and output mechanism may comprise, e.g., a keyboard, microphone, speaker, display, and / or camera.

[0147] As an example the input mechanism may comprise a system configured to input audio data characterizing a speech waveform of speech representing the input from the user in a natural language, and configured to convert the audio data into tokens representing the speech in the natural language, e.g. representing a transcription of the spoken input. The output mechanism may comprise a system configured to receive tokens representing the output for the user in the or another natural language and a system configured to convert the received tokens into audio data representing a waveform of speech representing the output to the user in the natural language, i.e. representing spoken words.

[0148] As a further example, the trained system can be deployed in an environment that enables a user to provide a request for the system, e.g. to process a multimodal conditioning input to generate a corresponding data item output. A users can provide the request, e.g., by way of a user interface or through an application programming interface (API). The request can be transmitted from a user device, e.g., over a data communications network such as the internet, to one or more computers implementing the system, e.g., in a data center. The system can generate a data item and then transmit the data item to a user device over a data communications network.

[0149] Some example applicationsATTORNEY DOCKET NO.45288-0489WO1

[0150] A generative neural network system, trained as described above, can be used for diagnosing a fault, or for correcting undesired behavior, in a mechanical or computing system operating in the real world environment. The conditioning input may comprise a description and / or image of one or more observations of the mechanical or computing system, e.g. of operation of the system, optionally obtained from one or more sensors sensing a condition or operation of the system. An image observation may be converted into a text description e.g. using an image captioning system or in other ways. The generated data item may comprise an image, audio, or text that identifies (described) a likely cause of the fault or undesired behavior. This may be used to repair the fault or correct the behavior. The reward model can define relatively more useful types of output for repairing the fault or correcting the behavior

[0151] The (trained) generative neural network system can be used for controlling a mechanical agent such as a robot or vehicle. For example the conditioning input may comprise a description of a task to be performed, and the generated data item may comprise a list of sub- tasks to be performed by the mechanical agent (trained to perform such sub-tasks), in order to perform the task. The reward model can define relatively more preferable or useful types of sub-task.

[0152] Example multimodal applications

[0153] The generative neural network system may comprise a multimodal machine learning system such as a visual language model (VLM). That is implementations of the generative neural network system can perform a multimodal task in which the conditioning input and data item, collectively, comprise data of multiple different types. As used herein text can include numbers, punctuation, special symbols, and so forth.

[0154] In some implementations, after training, a particular task that is to be performed by the generative neural network system can be described by part or all of a sequence of text in the conditioning input to the system. For example in a conditioning input that includes an image such a prompt might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image or video]”, or “Detect a person”. Where the system is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. Also or instead such a prompt may give one or more examples of a task to be performed. The generative neural network system can be trained on multiple natural and / or computer languages and the prompt may then specify a language to use.

[0155] A few further examples of some machine learning tasks that can be performed by a system trained as described herein follow. The tasks described below may be tasks that requireATTORNEY DOCKET NO.45288-0489WO1 spatial awareness or other context from the image or video. For example, a prompt may ask “What is the object in the top left corner?”.

[0156] In general for the tasks below the system can have been trained or fine-tuned on examples of the input and output for the task. For example the system can have been trained using still or moving images containing one or more objects or actions, and corresponding sequences of text or other data e.g. describing or classifying the images. However large, “foundation” models can, in general, perform some tasks zero-shot, i.e. without having been specifically trained on those tasks.

[0157] As one example the task may comprise an object or action detection task. For example the generated data item may comprise or represent text that describes or otherwise labels detected object(s) or action(s) in a conditioning input comprising an image or audio, and may include coordinates such as bounding-box coordinates for the detected object(s) or action(s), e.g. "102090100 cat 2030100100 dog”.

[0158] As another example the task may comprise a classification task, e.g. an object or action classification task. The generated data item may comprise data, e.g. text, that classifies the object(s) or action(s) in represented in the conditioning data, e.g. in an image or audio, into one of a plurality of classes, or that otherwise classify object(s) or action(s) represented in the conditioning data.

[0159] As another example the task may comprise a still or moving image describing task, e.g. a captioning task (which, as used here, includes an audio description task to explain what is happening in an image that is input to the system). The generated data item may comprise data, e.g. text, describing an image or video in the conditioning data. For example the generated data item may provide a caption or description or it may count objects in the image or video, or it may provide some other form of description.

[0160] As another example the task may comprise a still or moving image question-answering task. The generated data item may comprise data, e.g. text, that answers a question about the conditioning input, e.g. an image or audio, where the question is also specified in the conditioning input, e.g. as sequence of text. This may be used, e.g., to answer questions about visual plots and charts or about sounds.

[0161] As another example the task may comprise a character or word recognition task, e.g. an OCR (optical character recognition) task. The conditioning input may comprise a still or moving image and the generated data item may comprise text that represents characters or words in the conditioning input, e.g. in a natural language.ATTORNEY DOCKET NO.45288-0489WO1

[0162] As another example the task may comprise a still or moving image generation task. The generated data item may comprise image data defining values for pixels of a still or moving image, and the conditioning input, e.g. a sequence of text, may describe or characterize the image to be generated. Merely as an example, an image of a plot or chart may be generated to represent the conditioning input, e.g. comprising text.

[0163] As another example the task may comprise a computer language text generation task. The conditioning data may comprise a natural language description of a task to be performed, and optionally an image (if the task is to be performed on or in relation to an image), and the generated data item may comprise text in a computer language to perform the task, e.g. a task of analyzing the content of the image to provide a result of the analysis or to search for information relating to the content of the image.

[0164] As a particular example the computer language in the generated data item may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such a data item may comprise data formatted as a JSON object. As previously, the conditioning input may define the task to be performed and may also include an image in relation to which the task is to be performed. In general the task can involves manipulation of particular types of data that may benefit from access to an API such as mathematical data, date / time related data, scientific data, recent data that may post-date training of the system (that may be accessed by a search function or API), and so forth; and the generated data item may comprise text in a computer language for performing the task. The method may then include using the text in the computer language to perform the task.

[0165] In general where the generated data item comprises text this may be converted to speech representing the text, and an audio (speech) output provided.

[0166] In some implementations the task comprises an agent control task in which the agent interacts with an environment to perform the agent control task. In these implementations the conditioning input can include an observation characterizing the environment. For example the conditioning input can include a sequence of text that defines the task to be performed by the agent and the image can represent an observation of the environment, e.g. captured by a camera or other imaging device from a real-world environment. The generated data item can comprise an action selection output, e.g. including text, that is used to select one or more actions to be performed by the agent in the environment in response to the observation. As an illustration the generated data item may define an action as text such as “A: 132114128525 156”, that can be converted into a control signal for a mechanical agent, such as a robot, e.g.“Δ: = ;0.1, −0.2,0= Δ? = ;10@, 25@, −7@=”. The action selection output may also or insteadATTORNEY DOCKET NO.45288-0489WO1 define one or more low-level skills, e.g. from a vocabulary of previously learnt skills. As before, the sequence of text in the conditioning input to the system may describe the task to be performed, e.g. “What action should the robot take to [perform task]”. Examples of systems for controlling an agent that may be fine tuned as described herein can include PaLM-E (Driess et al. arXiv:2303.03378), RT-1 (Brohan et al. arXiv:2212.06817), and RT-2 (Brohan et al. arXiv:2307.15818).

[0167] In some agent control implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real- world environment to perform the task. For example, the agent may be a robot or other mechanical agent interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment. The actions may define control signals to control the robot or other mechanical agent, e.g., positions, torques, or other control signals for the parts of the mechanical agent, or higher-level control commands.

[0168] In some agent control implementations the agent may be a human agent and the environment may be a real-world environment. For example the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real-world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g. a monitoring system such as a video camera or sound capture system, to capture visual observations of the user performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task.

[0169] Example reinforcement learning implementations

[0170] At each time step that an agent interacts with an environment to perform a task, the agent can receive a reward based on the current state of the environment and the action performed by the agent at the time step. Generally, the reward may be represented as a numerical value. The reward can be based on any event in or aspect of the environment. For example, the reward may indicate whether the agent has accomplished a task (e.g., navigatingATTORNEY DOCKET NO.45288-0489WO1 to a target location in the environment) or the progress of the agent toward accomplishing the task.

[0171] A Q-value neural network can be trained using any Q-learning algorithm to generate a Q-value for a combination of an observation and a possible action. The Q-value for an action is an estimate of a “return” that would result from the agent performing the action in response to the observation and thereafter selecting future actions performed by the agent in accordance the current action selection policy. A return comprises a cumulative measure of rewards received by the agent, for example, a time-discounted sum of rewards. The Q-value neural network may, for example, define a Q-value or it may define one or more parameters of a distribution from which a Q-value may be sampled.

[0172] The action selection policy neural network 110 can process the observation for a current time step, in accordance with current values of parameters of the action selection policy neural network to generate the action selection output, and the action may be selected using the action selection output. The action selection output may comprise action selection scores according to which the action is selected, or the action selection output may determine the action directly, or the action selection output may parameterize a distribution e.g. a Gaussian distribution, according to which the action may be selected stochastically.

[0173] In some implementations, the environment is a real-world environment, the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the mechanical agent, e.g. robot, may be interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0174] In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. TheATTORNEY DOCKET NO.45288-0489WO1 observations may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and / or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0175] In these implementations, the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher- level control commands. The control signals can include for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the control signals may define actions to control navigation e.g. steering, and movement e.g., braking and / or acceleration of the vehicle.

[0176] In some implementations the environment is a simulation of the above-described real- world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world.

[0177] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material e.g. to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g. robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g. via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot.

[0178] The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent mayATTORNEY DOCKET NO.45288-0489WO1 comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines.

[0179] As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g. minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process.

[0180] The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment e.g. between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine.

[0181] The rewards or return may relate to a metric of performance of the task. For example in the case of a task that is to manufacture a product the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g. a metric of a quantity of energy, materials, or other resources, used to perform the task. In the case of a task that is to control use of a resource the metric may comprise any metric of usage of the resource.

[0182] In general observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and / or mechanical items of equipment. For example a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g. sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rateATTORNEY DOCKET NO.45288-0489WO1 or flow / movement path of one or more materials; physical or chemical conditions e.g. a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case of a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g. data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor. Sensors such as these may be part of or located separately from the agent in the environment.

[0183] In some implementations the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control e.g. cooling equipment, or air flow control or air conditioning equipment such as a heater, a cooler, a humidifier, or other hardware that modifies a property of air in the real-world environment. The task may comprise a task to control, e.g. minimize, use of a resource, such as a task to control electrical power consumption, or water consumption. The agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g. environmental, control equipment.

[0184] In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g. actions that result in changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment.

[0185] In general observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by anyATTORNEY DOCKET NO.45288-0489WO1 sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open.

[0186] The rewards or return may relate to a metric of performance of the task. For example in the case of a task to control, e.g. minimize, use of a resource, such as a task to control use of electrical power or water, the metric may comprise any metric of use of the resource.

[0187] In some implementations the environment is the real-world environment of a power generation facility e.g. a renewable power generation facility such as a solar farm or wind farm. The task may comprise a control task to control power generated by the facility, e.g. to control the delivery of electrical power to a power distribution grid, e.g. to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility. The agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid. The actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements e.g. to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power generation machine. Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g. an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output. Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated.

[0188] The rewards or return may relate to a metric of performance of the task. For example in the case of a task to control the delivery of electrical power to the power distribution grid the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility. In the case of a task to maximize the delivery of electrical power to the power distribution grid the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility.ATTORNEY DOCKET NO.45288-0489WO1

[0189] In general observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment. Such observations may thus include observations of wind levels or solar irradiance, or of local time, date, or season. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid e.g. from local or remote sensors. Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid.

[0190] As another example, the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals / intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that indirectly performs or controls the protein folding actions, or chemical synthesis steps, e.g. by controlling synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or indirect observations of a state of the protein or chemical / intermediates / precursors and / or may be derived from simulation. Thus the system may be used to automatically synthesize a protein with a particular function such as having a binding site shape, e.g. a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug. For example e.g. it may be an agonist or antagonist of a receptor or enzyme; or it may be an antibody configured to bind to an antibody target such as a virus coat protein, or a protein expressed on a cancer cell, e.g. to act as an agonist for a particular receptor or to prevent binding of another ligand and hence prevent activation of a relevant biological pathway.ATTORNEY DOCKET NO.45288-0489WO1

[0191] In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound, i.e. a drug, and the agent is a computer system for determining elements of the pharmaceutically active compound and / or a synthetic pathway for the pharmaceutically active compound. The drug / synthesis may be designed based on a reward derived from a target for the pharmaceutically active compound, for example in simulation. The agent may be, or may include, a mechanical agent that performs or controls synthesis of the pharmaceutically active compound; and hence a process as described herein may include making such a pharmaceutically active compound.

[0192] For example the environment may be an in silico drug design environment, e.g., a molecular docking environment, and the agent may be a computer system for determining elements or a chemical structure of the drug. The drug may be a small molecule or biologic drug. An observation may be an observation of a simulated combination of the drug and a target of the drug. An action may be an action to modify the relative position, pose or conformation of the drug and drug target (or this may be performed automatically) and / or an action to modify a chemical composition of the drug and / or to select a candidate drug from a library of candidates. One or more rewards may be defined based on one or more of: a measure of an interaction between the drug and the drug target, e.g., of a fit or binding between the drug and the drug target; an estimated potency of the drug; an estimated selectivity of the drug; an estimated toxicity of the drug; an estimated pharmacokinetic characteristic of the drug; an estimated bioavailability of the drug; an estimated ease of synthesis of the drug; and one or more fundamental chemical properties of the drug. A measure of interaction between the drug and drug target may depend on e.g. a protein-ligand bonding, van der Waal interactions, electrostatic interactions, and / or a contact surface region or energy; it may comprise, e.g., a docking score. Following identification of elements or a chemical structure of a drug in simulation, the method may further comprise making the drug. The drug may be made partly or completely by an automatic chemical synthesis system.

[0193] In some applications the agent may be a software agent i.e. a computer program, configured to perform a task. For example the environment may be a circuit or an integrated circuit design or routing environment and the agent may be configured to perform a design or routing task for routing interconnection lines of a circuit or of an integrated circuit e.g. an ASIC. The reward(s) may then be dependent on one or more routing metrics such as interconnect length, resistance, capacitance, impedance, loss, speed or propagation delay; and / or physical line parameters such as width, thickness or geometry, and design rules. The reward(s) may also or instead include one or more reward(s) relating to a global property of the routed circuitryATTORNEY DOCKET NO.45288-0489WO1 e.g. component density, operating speed, power consumption, material usage, a cooling requirement, level of electromagnetic emissions, and so forth. The observations may be e.g. observations of component positions and interconnections; the actions may comprise component placing actions e.g. to define a component position or orientation and / or interconnect routing actions e.g. interconnect selection and / or placement actions. The task may be, e.g., to optimize circuit operation to reduce electrical losses, local or external interference, or heat generation, or to increase operating speed, or to minimize or optimize usage of available circuit area. The method may include making the circuit or integrated circuit to the design, or with interconnection lines routed as determined by the method.

[0194] In some applications the agent is a software agent and the environment is a real-world computing environment. In one example the agent manages distribution of tasks across computing resources e.g. on a mobile device and / or in a data center. In these applications, the observations may include observations of computing resources such as compute and / or memory capacity, or Internet-accessible resources; and the actions may include assigning tasks to particular computing resources. The reward(s) may be configured to maximize or minimize one or more of: utilization of computing resources, electrical power, bandwidth, and computation speed.

[0195] In another example the software agent manages the processing, e.g. by one or more real-world servers, of a queue of continuously arriving jobs. The observations may comprise observations of the times of departures of successive jobs, or the time intervals between the departures of successive jobs, or the time a server takes to process each job, e.g. the start and end of a range of times, or the arrival times, or time intervals between the arrivals, of successive jobs, or data characterizing the type of job(s). The actions may comprise actions that allocate particular jobs to particular computing resources; the reward(s) may be configured to minimize an overall queueing or processing time or the queueing or processing time for one or more individual jobs, or in general to optimize any metric based on the observations.

[0196] As another example the environment may comprise a real-world computer system or network, the observations may comprise any observations characterizing operation of the computer system or network, the actions performed by the software agent may comprise actions to control the operation e.g. to limit or correct abnormal or undesired operation e.g. because of the presence of a virus or other security breach, and the reward(s) may comprise any metric(s) that characterizing desired operation of the computer system or network.

[0197] In some applications, the environment is a real-world computing environment and the software agent manages distribution of tasks / jobs across computing resources e.g. on a mobileATTORNEY DOCKET NO.45288-0489WO1 device and / or in a data center. In these implementations, the observations may comprise observations that relate to the operation of the computing resources in processing the tasks / jobs, the actions may include assigning tasks / jobs to particular computing resources, and the reward(s) may relate to one or more metrics of processing the tasks / jobs using the computing resources, e.g. metrics of usage of computational resources, bandwidth, or electrical power, or metrics of processing time, or numerical accuracy, or one or more metrics that relate to a desired load balancing between the computing resources.

[0198] In some applications the environment is a data packet communications network environment, and the agent is part of a router to route packets of data over the communications network. The actions may comprise data packet routing actions and the observations may comprise e.g. observations of a routing table which includes routing metrics such as a metric of routing path length, bandwidth, load, hop count, path cost, delay, maximum transmission unit (MTU), and reliability. The reward(s) may be defined in relation to one or more of the routing metrics i.e. configured to maximize one or more of the routing metrics.

[0199] In some other applications the environment is an Internet or mobile communications environment and the agent is a software agent which manages a personalized recommendation for a user. The observations may comprise previous actions taken by the user, e.g. features characterizing these; the actions may include actions recommending items such as content items to a user. The reward(s) may be configured to maximize one or more of: an estimated likelihood that the user will respond favorably to being recommended the (content) item, a suitability unsuitability of one or more recommended items, a cost of the recommended item(s), and a number of recommendations received by the user, optionally within a time span.

[0200] As a further example, the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users.

[0201] In some cases, the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location).

[0202] As another example the environment may be an electrical, mechanical or electro- mechanical design environment, e.g. an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. The observations may comprise observations that characterize theATTORNEY DOCKET NO.45288-0489WO1 entity, i.e. observations of a mechanical shape or of an electrical, mechanical, or electro- mechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may comprise actions that modify the entity e.g. that modify one or more of the observations. The rewards or return may comprise one or more metric of performance of the design of the entity. For example rewards or return may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed. The design process may include outputting the design for manufacture, e.g. in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus the design of an entity may be optimized, e.g. by reinforcement learning, and then the optimized design output for manufacturing the entity, e.g. as computer executable instructions; an entity with the optimized design may then be manufactured.

[0203] As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment.

[0204] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, may be deployed for controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real-world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment. For example the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment. Thus in such cases the observations of the simulated environmentATTORNEY DOCKET NO.45288-0489WO1 relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.

[0205] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0206] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0207] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computerATTORNEY DOCKET NO.45288-0489WO1 programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0208] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0209] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specificATTORNEY DOCKET NO.45288-0489WO1 design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0210] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0211] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The essential elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the AI model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.ATTORNEY DOCKET NO.45288-0489WO1

[0212] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0213] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0214] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0215] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN)ATTORNEY DOCKET NO.45288-0489WO1 including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0216] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0217] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0218] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described programATTORNEY DOCKET NO.45288-0489WO1 components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0219] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0220] What is claimed is:

Claims

ATTORNEY DOCKET NO.45288-0489WO1 CLAIMS 1. A computer implemented method of training a neural network system configured to process a policy input to generate a policy output according to a policy representing a probability distribution of the policy output given the policy input, the method comprising: obtaining a reference policy representing a reference probability distribution; obtaining one or more training data items, each training data item comprising an example policy input and a corresponding example policy output, and wherein at least one of the training data items is identified as a negative example; training the neural network system using the training data items to update the policy towards an improved policy that minimizes a likelihood of the training data items identified as negative examples, wherein the training includes penalizing a divergence from the reference policy.

2. The method of claim 1, wherein training the neural network system, including penalizing the divergence from the reference policy, comprises: updating parameters of the neural network system using an objective function that comprises: i) a first term that depends on a likelihood of the example policy output according to the policy given the example policy input, evaluated for each of the training data items identified as negative examples, and ii) a second term that measures a difference between the probability distribution and the reference probability distribution, wherein the first term and the second term have the same sign.

3. The method of claim 2, wherein the objective function comprises a loss function and the first term and the second term are both positive; or wherein the objective function defines an objective to be maximized and the first term and the second term are both negative.

4. The method of claim 2 or 3, further comprising, for each of the training data items identified as negative examples: processing the example policy input using the neural network system to determine the likelihood of the respective example policy output given the example policy input.ATTORNEY DOCKET NO.45288-0489WO1 5. The method of claim 4, wherein the training data items comprise a plurality of negative examples, the method comprising evaluating the first term for the plurality of negative examples by summing, for each of the negative examples, a log likelihood of the respective example policy output given the example policy input.

6. The method of any of claims 2-5, wherein the second term is evaluated using each of the training data items.

7. The method of any of claims 1-6, wherein: obtaining the reference policy comprises obtaining a reference neural network system configured to process a reference policy input to generate a reference policy output according to the reference policy; the method further comprising: processing the example policy input of each of the training data items using the neural network system to generate a respective example policy output; processing the example policy input of each of the training data items using the reference neural network system to generate a respective reference example policy output; and determining, for the second term of the objective function, a measure of a difference between the probability distribution obtained from using the neural network system and the reference probability distribution obtained from using the reference neural network system.

8. The method of claim 7, wherein the policy input, the policy output, the reference policy input, and the reference policy output each comprise a respective sequence of tokens, the method comprising, for each of the training data items: generating a reference policy sequence of tokens for the reference example policy output using the reference neural network system by iteratively, for each token of the reference policy sequence of tokens, processing i) an input sequence of tokens representing the example policy input and ii) previously generated tokens of the reference policy sequence of tokens, to generate the reference probability distribution for a current token of the reference policy sequence of tokens, and selecting the current token of the reference policy sequence of tokens using the reference probability distribution; for each token of the reference policy sequence of tokens, processing the input sequence of tokens and the previously generated tokens of the reference policy sequence ofATTORNEY DOCKET NO.45288-0489WO1 tokens using the neural network system to generate the probability distribution corresponding to current token of the reference policy sequence of tokens; and determining the measure of the difference between the probability distribution and the reference probability distribution for the second term by summing, for each token of the reference policy sequence of tokens, a measure of a difference between a) the reference probability distribution for the current token of the reference policy sequence of tokens and b) the probability distribution corresponding to current token of the reference policy sequence of tokens.

9. The method of any of claims 1-8, wherein at least one of the training data items has been identified as a positive example, the method further comprising: training the neural network system using the training data items to update the policy towards an improved policy that maximizes a likelihood of the training data items identified as positive examples.

10. The method of claim 9 when dependent on claim 2, wherein the objective function comprises a third term that depends on a likelihood of the example policy output according to the policy given the example policy input, evaluated for each of the training data items identified as positive examples, wherein the third term has an opposite sign to the sign of the first and second terms.

11. The method of any one of claims 1-10, further comprising setting a second relative weight, %, of the second term sufficiently high to omit just the negative examples from the probability distribution of the policy output.

12. The method of any one of claims 1-11, wherein the reference policy is the policy of the neural network system prior to the training.

13. The method of any one of claims 1-12, comprising iteratively training the neural network system by, at each of a succession of training iterations, using the policy of the trained neural network system as the reference policy for a subsequent training iteration.

14. The method of any one of claims 1-13, whereinATTORNEY DOCKET NO.45288-0489WO1 obtaining the reference policy comprises obtaining a reference neural network system configured to process a reference policy input to generate a reference policy output according to the reference policy; and wherein obtaining the training data items comprises, for each of the training data items: processing the example policy input using the reference neural network system to generate the corresponding example policy output.

15. The method of claim 14, wherein obtaining the training data items further comprises, for at least one of the training data items: identifying the training data item as a negative example by labelling the training data item dependent on the corresponding example policy output.

16. The method of claim 14 or 15 when dependent on claim 9, wherein obtaining the training data items comprises, for at least one of the training data items: processing the example policy input using the reference neural network system to generate first and second corresponding example policy outputs; pairing the example policy input with one of the first and second corresponding example policy outputs to obtain the training data item; determining a relative preference of the first and second corresponding example policy outputs; identifying the training data item as a negative example or as a positive example by labelling the training data item according to the determined relative preference.

17. The method of claim 16, wherein the method is implemented in a parallel processing computer system comprising a plurality of hardware devices configured to operate in parallel, the method further comprising: maintaining first and second respective instances of the reference neural network system on respective first and second respective hardware devices; and processing the example policy input using the first instance of the reference neural network system on the first hardware device to generate the first corresponding example policy output and, in parallel, processing the example policy input using the second instance of the reference neural network system on the second hardware device to generate the second corresponding example policy output.ATTORNEY DOCKET NO.45288-0489WO1 18. The method of claim 17, comprising obtaining two training data items, a negative example and a positive example, by labelling the training data item according to the determined relative preference of the first and second corresponding example policy outputs.

19. The method of claim 14, wherein obtaining the training data items further comprises, for one or more of the training data items: processing the example policy input and the corresponding example policy output using a scoring model to generate a score for the training data item; and labelling the training data item as a negative example dependent on the score.

20. The method of claim 19 when dependent on claim 9, wherein obtaining the training data items comprises, for one or more of the training data items: processing the example policy input using the reference neural network system to generate at least two corresponding example policy outputs, the example policy input and each respective corresponding example policy output defining a respective training data item; and processing the example policy input and the corresponding example policy output of each respective training data item using the scoring model to generate the score for the respective training data item; and labelling the respective training data item as either a negative example or a positive example dependent on the relative scores of the respective training data items.

21. The method of any one of claims 1-20, wherein the neural network system comprises a generative neural network system, wherein the policy comprises a data item generation policy, wherein the policy input comprises a conditioning input for the generative neural network system, and wherein the policy output defines a data item.

22. The method of claim 21, comprising: processing an input token sequence including the conditioning input, using the generative neural network system, to generate an output token sequence that defines the data item.ATTORNEY DOCKET NO.45288-0489WO1 23. The method of claim 21 or 22, wherein the generative neural network system comprises a language and / or image generation neural network system, wherein the conditioning input comprises a prompt for the generative neural network system, and wherein the data item comprises a language and / or image response to the prompt.

24. A computer-implemented method of generating a data item, comprising: obtaining a generative neural network system configured to process a conditioning input to generate a data item according to a data item generation policy, wherein the generative neural network system has been trained by the method of claim 21 or 22; obtaining a conditioning input for a data item to be generated; and processing the conditioning input for the data item to be generated using the trained generative neural network system to generate the data item based on the conditioning input.

25. A computer-implemented method of generating a data item, comprising: obtaining a generative neural network system, wherein the generative neural network system is configured to process a conditioning input to generate a data item according to a data item generation policy, wherein the generative neural network system has been trained to update the data item generation policy towards an improved policy that minimizes a likelihood of one or more training data items identified as negative examples whilst penalizing a divergence from a reference policy; obtaining the conditioning input for a data item to be generated; and processing the conditioning input for the data item to be generated using the trained generative neural network system to generate the data item based on the conditioning input.

26. The method of any one of claims 1-20, wherein the neural network system comprises an action selection policy neural network for selecting actions to be performed by an agent in an environment to perform a task, wherein the policy input comprises an observation characterizing a state of the environment, and wherein the policy output comprises an action selection output for selecting an action to control the agent to perform the task.

27. A system comprising: one or more computers; andATTORNEY DOCKET NO.45288-0489WO1 one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-16.

28. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-26.

Images