Training machine learning models using online data selection techniques

Abstract

A machine learning model is trained using a subset of training examples from a store of training data. Rather than randomly selecting the subset, the training examples in the subset are selected based on a score obtained using an online model. The online model is also trained using the subset of training examples, before performing another selection. As such, each successive selection of a subset of the training data, which are then provided to the machine learning model for further training, contains training examples which are better suited for efficiently training the machine learning model.

Description

TRAINING MACHINE LEARNING MODELS USING ONLINE DATA SELECTION TECHNIQUESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 600,568, filed on November 17, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to training machine learning models.

[0003] In general, a machine learning model can be trained to perform any sort of machine learning task or tasks. After the machine learning model has been trained it can be deployed for use in performing the machine learning task(s).

[0004] Training a model comprises backpropagating gradients of an objective function, e.g. a loss function, to update learnable parameters, e.g. weights, of the machine learning model. This may use any appropriate gradient descent optimization algorithm, e.g. Adam or Adam W, or another optimization algorithm. The model is trained using training data which includes a plurality of training data items referred to as training examples. The training examples may be of a data modality which relates to the task the machine learning model is being trained to perform.SUMMARY

[0005] This specification describes a system and a method, implemented as computer programs on one or more computers in one or more locations, for training a machine learning model to perform a task.

[0006] In the methods described herein, the machine learning model is trained using a subset of training examples from a store of training data. Rather than randomly selecting the subset, the training examples in the subset are selected based on a score obtained using an online model. The online model is also trained using the subset of training examples, before performing another selection. As such, each successiveselection of a subset of the training data, which are then provided to the machine learning model for further training, contains training examples which are better suited for efficiently training the machine learning model.

[0007] In a first example, a computer-implemented method of training a machine learning model to perform a task comprises obtaining training data comprising a plurality of training examples and selecting a first training subset comprising one or more training examples from the training data. The method includes determining a first score associated with one or more of the training examples of the first training subset by processing, using an online model, one or more training examples of the first training subset, comprises selecting a second training subset comprising one or more training examples from the training data, wherein each training example in the second training subset is selected based upon a prioritization score associated with the training example, the prioritization score based on a first score obtained using the online model. The method further comprises training, using the second training subset, the machine learning model and training, using the second training subset, the online model.

[0008] The method may further comprise, prior to selecting the second training subset, training, using the first training subset, the machine learning model and training, using the first training subset, the online model. Training the machine learning model may result in the generation of an updated machine learning model. That is, the parameters of the machine learning model may be updated. Training the online model may result in the generation of an updated online model. That is, the parameters of the online model may be updated. In such examples, each training example in the second subset may be selected based on a prioritization score associated with the training example, the prioritization score obtained based on a first score obtained using the updated online model. Training, using the second training subset, the machine learning model may comprise training, using the second training subset, the updated machine learning model. Training, using the second training subset, the online model may comprise training, using the second training subset, the updated online model.

[0009] Thus in another example a method of training a machine learning model to perform a task involves obtaining training data comprising a plurality of training examples, selecting a first training subset comprising one or more training examples from the training data, and training an online model using the first training subset to generate an updated online model. The machine learning model can be trained using the first training subset to generate an updated machine learning model. A secondtraining subset comprising one or more training examples can be selected from the training data. Each training example in the second training subset can be selected based upon a prioritization score associated with the training example. The prioritization score can be based on a first score obtained using the updated online model. The updated machine learning model and the updated online model can be trained using the second training subset. The first score can be indicative of a loss associated with processing one (or more) of the training examples using the updated online model.

[0010] The machine learning model may be referred to as a learner model. The training examples in the training data may be associated with the task which the machine learning model is being trained to perform. Associated with may include, for example, the training examples comprising one or more data modalities which are the same as one or more data modalities used in the particular task.[Oil] The machine learning model and / or online model may be trained using only the second training subset (i.e. only training examples in the second training subset and not other training examples from the training data which are not included in the second training subset). That is, the models may be trained using data consisting of the second training subset.

[0012] The method may be further referred to as a computer-implemented method of jointly training a machine learning model and an online model.

[0013] In another example, a computer-implemented method of performing a task comprises providing input data to a machine learning model, wherein the machine learning model has been trained to perform the task using the method of the above described example. The input data may be associated with (e.g. of the same modality as) training examples in the training data. The machine learning model may subsequently output data based on the input data according to the task it has been trained to perform.

[0014] In another example, a machine learning model is described, the machine learning model trained according to the training methods of the examples above. The machine learning model may have been trained to perform a particular task, for example using training data associated with the particular task.

[0015] In another example, a system comprises one or more computers and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of any of the example methods described above. The storage media can be implemented separately tothe one or more computers. That is, in another example, one or more non-transitory computer storage media store instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any of the example methods above.

[0016] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0017] Present-day machine learning models are capable of tackling complex tasks across various domains, driven by a large growth in, and utility of, in training data volume. Especially for vision models and language models, the error achieved by these models reduces as the volume of data used increases. As such, the feasibility of training such complex models is raising concerns. Some implementations of the described techniques can reduce by 25% the amount of computation required to train a large model, i.e. neural network, such as a large vision model, language model, or vision language model to a specified level of performance. Alternatively increased performance can be obtained for the same amount of computation.

[0018] Adaptively selecting training examples to provide to the machine learning model in training provides superior training compared to a training method in which training data is selected at a particular point in time. The training goals can be achieved more efficiently (e.g. with less compute) because the machine learning model can be provided with data that is most useful to it at its current training state, at each point during its training. Overall, adaptively selecting training examples can also lead to over computepositive processes compared to training methods which statically select training example subsets, for example prior to training or early on in training. Methods which statically select training example subsets typically require significant pre-training (for pre-training a data selection model) and selection inference (wherein the data selection model selects data). As such, even if the training process alone is computationally efficient, the overall pipeline of training (including pre-training and selection inference) can be computationally burdensome. On the other hand, adaptively selecting training examples to provide to the machine learning model in training can result in reduced overall computational burden, because the pre-training and inference processes can be minimized and / or avoided completely. In particular, for large machine learning models (e.g. with over 100 million parameters), the overall compute required is reduced compared to training methods which uniformly (e.g. IID) select training data. Reduced compute may be represented by, for example, a reduced FLOP count. Net savings incompute, e.g. FLOP count, can enable machine learning tasks to be realized using more readily available computer systems, given the compute limits of general purpose computer systems and processors.

[0019] In some implementations the compute required is further reduced by selecting training examples using a second score that can obtained from a reference model. For example the prioritization score used to select the training examples can be dependent upon a difference between, or some other function of, the first score and the second score, which can be referred to as a learnability score. It might be thought beneficial to train on difficult, e.g. high loss, examples since easy examples yield small gradients. On the other hand it might be thought beneficial to remove difficult, e.g. noisy examples from the training data. The learnability score selects training examples that are relatively easily solved by a trained model but are challenging to the model in its current state of training such that more computation dedicated to these examples can lower the loss. Both the online model and the reference model can also be significantly smaller than the machine learning model (learner model), e.g. each having at least 4x, 13x or 47x fewer trainable / trained parameters such as weights. This can result in substantially reduced, e.g. 25% less, total computation to reach the same performance.

[0020] Furthermore, methods which statically select training example subsets are intrinsically iterative (i.e. the pre-training must be performed before selection inference, and selection inference must be performed before machine learning model training). According to this method, however, data selection and machine learning model training can be performed in parallel, thereby enabling parallelization and faster training times. As such, training can be completed faster compared to sequential training processes. Training in parallel in this way can leverage the resources of a computer, computer system, or plurality of computers, in a more time-efficient manner. For example, training the machine learning model may be performed on a first computer core and training the online model may be performed on a second computer core. Parallelizing training in this way enables the resources of a general purpose computer (or a plurality of general purpose computers) to be better leveraged so as to perform training tasks faster. Faster training times may comprise faster ‘wall-clock’ training times, i.e. start to finish training times.

[0021] Up-to-date prioritization scores can be used to select training data for training the machine learning model. This provides the ability to adaptively select data for use in training, for example in a way that dynamically adjusts to the changing training state ofthe machine learning model over the course of training. Adaptively selecting data, by using evolving scoring methods, as described herein provides improved training methods compared to offline data selection methods. During the course of training, different training examples may become more or less informative (e.g. more or less difficult, more or less learnable). Updating the scoring for data selection during the training process enables these changes in usefulness of the training examples to be leveraged to improve training (e.g. reduce training time, reduce training compute), for example because a target performance can be reached sooner or with less compute if more beneficial training data is prioritized. The selection of appropriate training examples using a score from the online model is also facilitated, in implementations of the method, by training the online model as well as the machine learning model (learner model), so that these leam together.

[0022] Methods which statically select training example subsets typically require significant pre-training (for pre-training a data selection model) and selection inference (wherein the data selection model selects data). As such, even if the training process alone is computationally efficient, the overall pipeline of training (including pre-training and selection inference) can be computationally burdensome. Methods which statically select training example subsets are intrinsically iterative (i.e. the pre-training must be performed before selection inference, and selection inference must be performed before machine learning model training).

[0023] By decoupling the data scoring models (e.g. the online model and reference model 130) from the machine learning model, the data scoring models can be significantly downscaled (e.g. have a reduced number of trainable / learned parameters such as weights). Because the data scoring models are downscaled, this can lead to an overall reduction in compute for the training process compared to methods which randomly sample (e.g. IID) data. If training to a desired accuracy or performance, for example, training a machine learning model according to the methods described herein will achieve that desired accuracy or performance faster than a training method which does not adaptively sample data. That is, a training method using adaptive selection of training data can provide a machine learning model which performs as well as a machine learning model trained using a different method, in a shorter amount of time and with reduced compute.

[0024] In this training process, even though more inference steps are performed during the training process, (by scoring and selecting training examples), the inference stepsthemselves are less computationally burdensome than the pre-training inference sweep required by other data selection models. Furthermore, some aspects of the data scoring can be amortized, thereby reducing the overall compute required across many training runs. The resulting method can be ‘compute-positive’ compared to the combination of pre-training, inference for data selection, and finally training of prior models. The compute-positive benefits are particularly relevant at large scales.

[0025] These methods recognize the time-varying importance of training examples during the training process. The importance of the training examples can be quantified by considering their learnability (e.g. how difficult the model will find the training example in its current state and in a future, fully trained state). The learnability of different examples is leveraged to select data which is most suitable for training the machine learning model in its current state, at each point in the machine learning model’s training process.

[0026] The methods herein can be easily generalized across different datasets and models. For example, many different machine learning models may use aspects of the scoring models (e.g. the online model and / or reference model 130) and / or be trained on the same or overlapping training datasets. Redundancies and similarities between different machine learning models can be used to improve training performance by taking advantage of known properties.

[0027] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 shows an example of a system used for training a machine learning model to perform a task;

[0029] Figure 2 is a flow diagram of an example method for training a machine learning model;

[0030] Figures 3A and 3B show graphs illustrating the performance of the active learning training methods described herein;

[0031] Figure 4 illustrates speedup in training for some example implementations.

[0032] Figures 5A and 5B show some example results illustrating a reduction in training compute.

[0033] Figures 6A and 6B show further example results illustrating a reduction in training compute.

[0034] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0035] A proactive strategy of actively selecting and prioritizing data from a training dataset during training, as opposed to passively and randomly sampling from the training dataset, is described herein. Proactively and actively selecting and prioritizing data for training is also known as active learning. Using active learning, the most informative and pertinent training examples can be identified and used for training, thereby reducing the volume of training data needed while preserving, or even enhancing, the performance of the model being trained.

[0036] The systems and methods described herein are concerned with active learning. In order to train a main machine learning model, the systems and methods described herein also use an online model to aid in actively selecting data for providing to the machine learning model. Both the machine learning model and the online model are updated a plurality of times, and often in parallel, such that just as the machine learning model is trained, the online model is also trained. Therefore, the relevance of the training examples selected by the online model, with reference to how well they will train the machine learning model, generally improves with each training iteration.

[0037] Figure 1 shows an example of a system 100 used for training a machine learning model 110 to perform a task. The system 100 can be implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The system 100 comprises a machine learning model 110 and an online model 120.

[0038] The machine learning model 100 may be referred to as a learner model (e.g. because it is a model which is learning and / or being taught during training). The machine learning model 100 may be, for example, a neural network. The learner model 110 can be trained to generate any output appropriate for the task performed. For example, the learner model 110 output can be a classification output, a regression output, a sequence output (i.e., that includes a sequence of output elements), a segmentation output, or a combination thereof.

[0039] Training data 10 is provided for training the learner model 110. The training data 10 may be referred to as a dataset. The training data 10 contains multiple training examples. The training examples may be referred to as training data items. The training data 10 may be stored in and obtained from, for example, a memory associated with the one or more computers on which the system is implemented. The training data 10 may be any data appropriate to the task being performed. There are many publically available datasets, for example data sets appropriate for the tasks listed herein.

[0040] The online model 120 may be a machine learning model. The online model 120 may be trained to perform a task of generating a score associated with an input data item. For example, the online model 120 may be trained for the task of generating a score associated with a particular training example (e.g. from the training data 10 or a subset thereof), given that particular training example as an input. The online model 120 is provided to assist the training of the machine learning model. The online model 120 is generally associated with optimizing the selection of training examples from the training data 10 for providing to the learner model 110 for training the learner model 110.

[0041] The learner model 110 and online model 120 can both be trained, for example using the training data 10 or a subset of training examples therefrom. Training the learner model 110 and online model 120 may comprise updating learnable parameters 0i of the learner model 110 and learnable parameters 0Oof the online model 120, respectively, for example using backpropagation. The learnable parameters 0i of the learner model 110 and learnable parameters 0Oof the online model 120 may also be denoted 0learnerand 0onlme, respectively.

[0042] Training may be referred to as updating. Multiple training rounds can take place and, in particular, in the methods described herein the learner model 110 and online model 120 are trained using one or more of a first training subset 11 and a second training subset 12, each selected from the training data 10. Further training may be performed using further iterations of training with an (updated) second training subset 12, for example following re-selection of the second training subset 12. Selection of the first and second training subsets 11, 12 and the use of these subsets to train the learner model 110 and online model 120 are described below in more detail.

[0043] Optionally, the selection of the second training subset 12 may also utilize a reference model 130. The reference model 130 can be trained using a third training subset 13. The reference model 130 and its used in selecting the second training subset12 is described in more detail further below. The reference model 130 may be trained in multiple iterations, online, similar to the learner model 110 and online model 120.

[0044] Each of the learner model 110 and online model 120 may be implemented on different ones of multiple parallel processing hardware. As such, training the learner model 110 and the online model 120 may be performed in parallel. In general such parallel processing hardware comprises a plurality of coupled computing devices operating in parallel, typically each having associated (local) memory. Such a computing device may comprise, as some examples, a TPU (Tensor Processing Unit) or GPU (Graphics Processing Unit), or, general purpose computer system, or a computer system equipped with one or more GPUs or TPUs, or a combination of these. Some further examples of parallel processing hardware are described later.

[0045] Figure 2 is a flow diagram of an example method 200 for training a machine learning model, for example the learner model 110 of Figure 1. The method 200 will be described as being performed by a system implemented on one or more computers located in one or more locations, for example the system 100 of Figure 1. As part of the method 200 for training the machine learning model, an online model (e.g. the online model 120 of Figure 1) is also trained. As such, the method 200 may be referred to as a method 200 of jointly training two machine learning models (i.e. a machine learning model 110 and an online model 120).

[0046] The method 200 includes, obtaining 201 training data, for example the training data 10 of Figure 1. The training data 10 comprises a plurality of training examples. The training data 10 may be stored in and obtained from, for example, a memory associated with the one or more computers on which the method is performed. In general, the training data 10 will include training examples associated with the task which the learner model 110 is being trained to perform.

[0047] The method 200 further includes selecting 202 a first training subset 11 from the training data 10. The first training subset 11 comprises one or more training examples from the training data. As such, the first training subset 11 will typically have a number of training examples less than or equal to the number of training examples in the training data 10. Generally, the first training subset 11 will have a smaller number of training examples compared to the number of training examples in the training data 10. The step of selecting 202 may also be referred to as pruning (for example, data pruning) or sampling. The first training subset 11 may be selected uniformly, for example using independent and identically distributed (IID) data selected from the training data 10.The first training subset 11 may comprise less than or equal to half of the number of training examples in the training data 10.

[0048] Optionally, the method 200 includes training 204, using the first training subset 11, the learner model 110. By training the learner model 110, its parameters 0i are updated. Following training, the learner model 110 may be referred to as an updated learner model 110. It should be understood that the learner model 110 and updated learner model 110 comprise the same underlying model, but are in different states of training and may have different parameters.

[0049] Optionally, the method 200 includes training 203, using the first training subset 11, the online model 120. By training the online model 120, its parameters 0Oare updated. Following training, the online model 120 may be referred to as an updated online model 120. It should be understood that the online model 120 and updated online model 120 comprise the same underlying model, but are in different states of training and may have different parameters. As mentioned above, the online model 120 is trained to perform a task of generating a score associated with any training examples (e.g. from the training data 10 or first training subset 11) provided to the online model 120 as an input.

[0050] Training 203, 204 of the learner model 110 and online model 120 using the first training subset 11 can be performed in parallel, i.e. simultaneously, for example using parallel processing hardware.

[0051] The method 200 further includes selecting 205 a second training subset 12 from the training data 10. Each training example in the second training subset 12 is selected based upon a prioritization score. For example, the selection of a particular training example for the second training subset 11 may be weighted based upon its prioritization score.

[0052] The prioritization score is based on a first score obtained using the online model 120. If the online model 120 has been trained using the first training subset 11, the prioritization score may be said to be obtained using an updated online model. The first score may be generated by the online model 120 (e.g. as a direct output), or may be derived from an output of the online model 120, based on the online model 120 receiving the training example as an input. In examples where the first scores are obtained based on the updated online model 120 (i.e. following training of the online model 120 using the first training subset 11), they are obtained based upon an at least partially trained online model 120.

[0053] In other words, the method 200 may further comprise an additional step after selecting the first training subset and prior to selecting the second training, of determining a first score associated with one or more of the training examples of the first training subset by processing, using the online model, one or more training examples of the first training subset. The online model 120 may be used to generate first scores for some or all training examples of the training data 10. It is implicit that the online model 120 will have generated prioritization scores for at least the training examples in the second training subset 11.

[0054] Selecting 205 the second training subset 12 from the training data 10 may be referred to as sampling from the training data 10. The training data 10 for the second training subset 11 may be determined as x, ~ M where is representative of a training example i, M represents a plurality of scored training examples (i.e. training examples in the training data 10 which have an associated prioritization score), and TT is a selection policy. The policy TT may be determined as a softmax over prioritization scores sp, where each scored training example x, has an associated prioritization score sf, as Scoring is described in more detail further below.

[0055] The second training subset 12 comprises one or more training examples from the training data 10. The second training subset 12 may have a smaller number of training examples compared to the number of training examples in the training data 10. The second training subset 12 may have a smaller number of training examples compared to the number of training examples in the first training subset 11. The first and second training subsets 11, 12 may include some or all of the same training examples. The second training subset 12 may comprise less than or equal to half of the number of training examples in the training data 10. It has been found that training data can be pruned significantly using the methods described herein while still achieving significant increases in compute and a reduction in training time compared to other methods of pruning data.

[0056] The training examples of the second training subset 12 may include some ordering or weighting (e.g. based on the scores) which is different to any ordering or weighting (or lack thereof) associated with the training examples of the first training subset 11. That is, even if there is partial or complete overlap between the trainingexamples in the first and second training subsets 11, 12, the subsets will typically not be identical, for example due to the training examples having different prioritization scores associated with them. As will be described in more detail below, the method 200 may include multiple training rounds, with an updated second subset being used in each round, so as to successively improve the selection of training data used for training. Successive subsets used in training may become more similar as training continues. Successive subsets may become identical, or substantially identical, when the online model 120 has converged.

[0057] The method 200 further includes training 206, using the second training subset 12, the learner model 110. Training the learner model 110 may comprise further updating learnable parameters 0i of the machine learning model, for example using backpropagation. The second training subset 12 used for training the learner model 110 has been actively selected following a training step associated with the online model 110. As such, the second training subset 12 is actively prior to being provided to the learner model 110 for training. Given that the online model 120 is trained to select a subset of training examples which are well-suited for training the learner model 110, training the learner model on this selected second subset 12 can improve the training time and compute associated with training the learner model.

[0058] The method 200 further comprises training 207, using the second training subset 12, the online model 110. Training the online model 120 may comprise further updating learnable parameters 0Oof the online model 120, for example using backpropagation.

[0059] The online model 120 is actively updated each time the learner model 110 is updated during training such that the scoring associated with the online model is updated. The online model 120 is also trained using the same data that is used to train learner model 110, and so the online model 120 can be used to accurately score the relevance of training examples for the process of training the learner model 110 in its particular training state (e.g. a quantification of how well trained or converged the learner model 110 is). For example quantifying a position of the model 110 between being not trained (not converged) and well trained (converged). Such a position may be quantified, for example, temporally or pseudo-temporally relative to an iteration number and / or epoch. The training examples in the second training subset 12 may have been optimised for the current training state of the machine learning model 110, for example due to being selected using a model (the online model 120) which is actively updated with the same training data as the learning model 110 is updated with. By training theonline model 120 for scoring training examples, in tandem with training the learner model 110, the data provided to the learner model 110 can be adaptively selected to optimize training.

[0060] Training 206, 207 of the learner model 110 and online model 120 using the second training subset 12 can be performed in parallel, i.e. simultaneously, for example using parallel processing hardware.

[0061] Steps of the method 200 (and, in particular, the steps of selecting 205 an (updated) second training subset, and training 206, 207 the (updated) learner model 110 and (updated) online model 120) may be performed multiple times, for example in multiple iterations. With each iteration, the online model 120 is learning (i.e. converging upon) how to best select training data for training the learner model 110 for a particular task. As such, as the number of iterations increases, the learner model 110 will achieve a higher level of training (i.e. become more converged).

[0062] As described above, in some implementations, the online model 110 and the machine learning model 120 are trained in parallel. By parallelizing training, parallel processing hardware (e.g. multiple processing cores of a computer upon which this method is performed) can be leveraged, resulting in more time-efficient training. The online model 120 may be trained on a first core or processing device of a system comprising one or more computers and the learner model 110 may be trained on a second core processing device of the system. The training time (e.g. wall-clock training time) can therefore be significantly reduced compared to, for example, IID sampling of data prior to training or any training with an inference pass required for sampling data prior to training

[0063] Scoring methods for selecting training examples

[0064] As described above, the method 200 includes selecting 205 a second training subset 12 from the training data 10, based upon a prioritization score generated using the online model 120. The online model 120 may generate, as an output, a first score associated with each training example provided as an input to the online model 120. The first score may be equal to the prioritization score, or some conversion can be performed between the first scores and associated prioritization score for each training example (e.g. normalization).

[0065] In some implementations, the first score is indicative of, e.g. dependent upon, a loss associated with processing one or more of the training examples using the online model 120. This first score may be associated with how well the online model 120 isable to perform a task associated with the particular training example in its current training state. For example, if a task associated with the training example is relatively difficult for the online model 120 to perform (i.e. an output of the online model 120 is not a good prediction for the training example), a loss associated with the prediction is relatively large. On the other hand, if the online model 120 finds a task relatively easy to perform (i.e. an output of the online model 120 is a good prediction for the training example), a loss associated with the prediction is relatively small. The online model 120 may be relatively unable to perform a particular task in early training iterations and so may not be able to generate a good prediction associated with a particular training example, whereas later in the training process a task associated with the same training example may be more straightforward for the online model leading to a lower loss.

[0066] The first score may be referred to as a difficulty score or easiness score. The first score may be referred to as a current score, a current difficulty score, or a current easiness score, for example because it provides an indication of how difficult a current version of the machine learning model (e.g. the machine learning model in its current training state) would find the training example. The first score may also be referred to as an online score or online model score. The loss associated with processing one or more of the training examples with the online model may be referred to as an online model loss.

[0067] The online model lossmay be represented aswhere represents a training examplerefers to the online model parameters. The loss may be calculated as a cross-entropy loss between a model prediction and a label associated with the training example. The loss may be calculated as a similarity, for example a dot-product similarity. A similarity-based loss may be particularly used for multi-modal learning, where the similarity can be determined using a similarity between embeddings of different modalities of a training example. For example, the online model lossin the instance of multi-modal learning including images and text, may be calculatedare jmage and text embeddings, respectively andis defined analogouslyInother examples, a contrastive loss function may be used. As an example a contrastive loss function may be determined aswhere sirn(-,-) denotes a similarity measure such as dot productsimilarity.

[0068] In some implementations, the prioritization score is further based on a second score obtained using a reference model 130. The reference model 130 may comprise a trained model. The reference model 130 may be trained to perform the same task as the task which the learner model 110 has been trained to perform (or is being trained to perform). The reference model 130 can therefore provide (second) scores indicative of an ability of the learner model 110 to complete a task given a particular training example. However, the reference model 130 is configured to provide second scores indicative of an ability of the learner model 110 to complete a task given a particular training example once the learner model 110 has been trained (e.g. in this iteration of training).

[0069] The reference model 130 may provide an indication or prediction of the ability or likelihood that the learner model 110 will be able to solve a particular problem (e.g. generate useful output) based on a particular input (e.g. receiving a particular training example as input). The reference model 130 may be able to predict or approximate an output that the learner model 110 is likely to provide once the learner model 110 has been trained, or a metric associated with the predicted or approximated output, based on the particular example being provided as an input to the learner model 110 once it has been trained.

[0070] The reference model 130 may be a simplified version of the learner model 110. For example the reference model 130 can have, but need not have, the same architecture as the online model. Obtaining the second score based upon the reference model 130 is intended to mean processing, using the reference model 130, an input and generating the second score based on the output or another metric associated with the output. For example, the second score may be generated based on providing the training example as an input to the reference model 130. The second score may be a measure of how the reference model 130 has processed the training example.

[0071] The reference model 130 may be pre-trained, for example using a contrastive objective function. In general the contrastive objective function is dependent upon a positive example and one or more negative examples. Additionally or alternatively, the reference model 130 is trained online, for example in multiple iterations alongsidetraining the learner model 110 and online model 120. For example, the steps of training 203, 204 the learner model 110 and online model 120 using the first training subset 11 may be performed substantially concurrently with training the reference model 130, and / or the steps of training 206, 207 the learner model 110 and online model 120 using the second training subset 12 may be performed substantially concurrently with training the reference model 130. Therefore, in this implementation the method for training the learner model 110 can also be referred to as a method of jointly training the learner model 110, the online model 120 and the reference model 130.

[0072] In some implementations, for example where the reference model 130 is not initially a trained model, the method involves training a reference model 130 using a third training subset selected from the training data 10 and further basing the prioritization score on a second score obtained using the reference model 130. The reference model 130 in this implementation is generally implemented as described above, but is initially untrained. The reference model 130 becomes a trained model following training. The third training subset on which the reference model 130 is trained may comprise the first and / or second training subset 11, 12, or alternatively a different subset of training examples may be used. It can be advantageous if the reference model 130 is trained on a larger subset of training examples compared to the first and / or second training subset 11, 12.

[0073] The reference model 130 may be trained in a single iteration, e.g. in a one-pass or one-shot approach where the third training subset is only provided to the reference model 130 a single time. In other implementations, the reference model 130 is trained (e.g. either pre-trained or trained online) using a different training dataset compared to the training data used to train the learner model 110 and online model 120. It may be preferable for the different training dataset to have related properties to the training data. For example, they may comprise the same modality(ies) of data and / or be contextually similar. For example, if the training data comprises images of an object belonging a particular category (e.g. a real-world category such as a type of animal), the different training dataset may also comprise images of objects belonging to the same category.

[0074] In some implementations, training the reference model 130 is performed with a reference model 130 learning rate which is higher than a learner model learning rate used for training the learner model 110 and an online model learning rate used for training the online model 120. In this way, even an untrained / uninitialized referencemodel 130 will converge faster than the online model 120 and learner model 110, and so will rapidly take the form of a trained model.

[0075] The reference model 130 may be trained in parallel with the learner model 110 and the online model 120, e.g. using parallel processing hardware as previously described. The benefits of parallelized training are elsewhere described herein, for example with reference to the parallelized training of the online model 120 and the learner model 110.

[0076] It has been found that reference model 130s can be generalized across similar tasks and similar data. As such, a reference model 130 can be used to provide a foundation dataset which can be used for training multiple other machine learning models. A particular reference model 130 may be applied to a particular set of training data to generate a set of second scores. That set of second scores can be stored, for example in a memory of a computer system. The set of second scores can be obtained (e.g. retrieved from memory) for use in calculating the prioritization score. In this way, the overall training process can have reduced compute because a portion of the prioritization score (i.e. the second score) is already known to the system. Therefore, in some implementations, the second score is obtained from a pre-calculated set of second scores.

[0077] The data provided by a pre-calculated sets of second scores can be further leveraged. For example, while the second scores and first scores are distinct, there may be some loose dependence between them. Alternatively, some training examples may have such high or low second scores that they can be ranked (e.g. as important or trivial examples). Such information can be utilized to rank the training examples, for example so that first scores are not calculated for trivial examples.

[0078] In some implementations, the second score is indicative of, e.g. dependent upon, a loss associated with processing one or more of the training examples using the reference model 130. The loss may be a loss associated with predicting the training example from a plurality of model inputs. The loss associated with processing one or more of the training examples with the reference model 130 may be associated with how challenging the reference model 130 finds the particular training example. If the reference model 130 is trained on the same (or a similar) task to the machine learning model (for example using similar data, e.g. a similar data modality), it can provide an indication of how challenging a trained version of the machine learning model would find the particular training example.

[0079] The second score may be referred to as an estimated score or predicted score, for example because it may represent an estimated or predicted difficulty experienced by a trained model. The second score may be referred to as a difficulty score or easiness score, or for example an estimated difficulty or easiness score. It should be understood that a high difficulty generally corresponds to a low easiness and a high easiness generally corresponds to a low difficulty. The second score may be referred to as a reference score or a reference model 130 score. The loss associated with processing one or more of the training examples with the reference model 130 may be referred to as a reference model 130 loss.

[0080] The reference model 130 lossrgfmay be represented aswhere represents a training example i andrefers to the reference model 130 parameters.The loss may be calculated as a cross-entropy loss between model predictions and a label associated with the training example. The loss may be calculated as a similarity, for example a dot-product similarity. A similarity-based loss may be particularly used for multi-modal learning, where the similarity can be determined using a similarity between embeddings of different modalities of training example. For example, the reference model 130 loss £rsJf, in the instance of multi-modal learning including images and text, may be calculatedand z ^1—are image and text embeddings, respectively andis defined analogouslyIn other examples, a contrastive loss function may be used. As an example a contrastive loss function may be determined as where sim(y) denotes a similarity measure such as dot productsimilarity.

[0081] In some implementations, the prioritization score is based on a combination of the first and second scores. As such, training examples will be prioritized (e.g. have a higher score) in situations where the loss is large for the online model 120 and the loss is small for the reference model 130. This situation indicates that the machine learning model has the opportunity to learn a lot from the particular training example, but that the machine learning model also has the capability to eventually solve the problem easily once trained. The prioritization score may be referred to as a learnability score.By leveraging learnability in this way, the machine learning model can be trained very efficiently and in a data efficient way. The combination of the first and second scores may comprise a difference between, or a summation of, the first and second scores. The prioritization score may be determined as, for example,Alternatively, the first and second scores can be combined in any other way, for example a weighted combination or other function. In general such a prioritization score (learnability score) measures a difference in loss on one or more training examples between the current machine learning model (i.e. with current values of the trainable parameters) and a trained version of the machine learning model. This difference can be expressed by subtraction, a ratio, or some other mathematical function. The difference is small for (relatively) easy training examples with a low loss for both, and for very difficult (or mislabelled) training examples with high loss for both, i.e. the difference is (relatively) smaller for training examples with a relatively small loss discrepancy; but (relatively) larger for training examples where the difference is relatively larger.

[0082] As described above, the online model has online (trainable) model parameters 0Oand the learner model has (trainable) learner model parameters 0i. In some implementations, the number of online model parameters 0o is smaller than the number of machine learning model parameters 0i (i.e. 0O< 0i , or 0O« 0i). For example, the online model 120 may have more than or equal to (at least) 2x fewer, 3x fewer, 4x fewer, 13x fewer, 14x fewer, 18x fewer, 47x fewer, 50x fewer or lOOOx fewer parameters than the learner model 110. In this way, the online model 120 acts as an approximation of the learner model 110 and can provide information about the current and eventual losses output by the learner 110 model in a more compute efficient manner. The online model 120 may have, for example, approximately 6 x 106online model parameters, 22 x 106online model parameters or 86 x 106online model parameters. The online model 120 may have less than 3 x 106online model parameters, 10 x 106online model parameters or 100 x 106online model parameters. The learner model 110 may have, for example, over 100 x 106online model parameters or over 300 x 106machine learning model parameters

[0083] Similarly, the reference model 130 can be smaller than the learner model 110. For example, the reference model 130 can have (trainable) reference model 130 parameters and the number of reference model 130 parameters may be smaller than thenumber of learner model parameters. The number of reference model 130 parameters may be the same as the number of online model 120 parameters or may be different. For example, the reference model 130 may have more than or equal to (at least) 2x fewer, 3x fewer, 4x fewer, 13x fewer, 14x fewer, 18x fewer, 47x fewer, 50x fewer or lOOOx fewer parameters than the learner model 110. Having a relatively small number of reference model 130 parameters can also help reduce the overall compute required for training the learner model 110 compared to training a large reference model 130.

[0084] Merely as an illustrative example consider an implementation that includes the reference model 130 where it is desired to reduce computation rather than, e.g., to increase performance. This can be done by arranging for the online model and reference models to be smaller (have fewer parameters) than the learner model. More specifically this can be done by arranging for the online model and reference models to be small enough that when the computation associated with the data selection process is included there is still a saving when (more efficiently) training the learner model. As a particular example denote the cost of scoring a training example Fact (the cost of inference processing the training example); p as the number of training examples scored per training example used for training (e.g. p=5); fi as the efficiency saving from training using selected training examples (e.g. ?=50%=0.5); 3Fref as the cost of training the reference model (assuming as a heuristic that the training cost is 3x the inference cost); Fieam as the cost of training the learner model with uniform sampling; and Fact = Fref + Fonime = IFref, then a reduction in compute cost can be achieved if (3F / eara+ pF act P + 3Fref < 3Fle arn.

[0085] In other examples, the reference model 130 may have the same number of parameters as the learner model 110, or may have more reference model 130 parameters than the number of learner model parameters and / or the number of online model parameters. This implementation may be particularly beneficial when the reference model 130 is pre-trained, because for example it can provide a more accurate and / or precise representation of the predicted performance of the learner model 110 following training.

[0086] When the reference model 130 is trained online, the learner model 110, the online model 120 and the reference model 130 can be trained in parallel. The training time (e.g. wall-clock training time) can therefore be significantly reduced compared to, for example, HD sampling of data prior to training or any training with an inference pass required for sampling data prior to training, because in this implementation thesampling step does not need to be performed prior to beginning training of the learner model 110 and the online model 120. By parallelizing training, multiple processing cores of a computer upon which this method is performed can be leveraged, resulting in more time-efficient training.

[0087] Figures 3A and 3B show graphs illustrating the performance of the active learning training methods described herein.

[0088] Figure 3A depicts the accuracy during training of a machine learning model to perform a large scale classification task using active learning (i.e. using the methods herein to actively select data for use in training the machine learning model). A first curve 31 shows the accuracy of a machine learning model trained according to the methods described herein, between an accuracy of 75% and a target performance 30 of 100%, using approximately 500 million training examples. A second curve 32 shows the accuracy of a machine learning model trained using a statically IID selected training batch of training examples, referred to as an IID baseline, between the same accuracy of 75% and a target performance 30 of 100%, using 900 million training examples. It can be seen from the curves 31, 32 and their ranges that fewer training examples are required to train the machine learning model using the methods described herein, compared to training using IID selected training examples. Given that a finite time is required to process each training example due to the compute limitations of a computer system, the reduction in training examples corresponds to a 46% speedup for training using the methods described herein, compared to training using IID selected training data.

[0089] Figure 3B depicts the accuracy during training of a machine learning model to perform a multimodal learning task using active learning (i.e. using the methods herein to actively select data for use in training the machine learning model). A first curve 33 shows the accuracy of a machine learning model trained according to the methods described herein, between an accuracy of 75% and a target performance 30 of 100%, using approximately 380 million training examples. A second curve 34 shows the accuracy of a machine learning model trained using a statically IID selected training batch of training examples, referred to as an IID baseline, between the same accuracy of 75% and a target performance 30 of 100%, using approximately 700 million training examples. It can be seen from the curves 33, 34 and their ranges that fewer training examples are required to train the machine learning model using the methods described herein, compared to training using IID selected training examples. Given that a finitetime is required to process each training example due to the compute limitations of a computer system, the reduction in training examples corresponds to a 51% speedup for training using the methods described herein, compared to training using IID selected training data.

[0090] Figure 3 A and 3B show, in combination, that the same accuracy can be arrived at in a quicker (or more compute efficient) manner when using the methods described herein, compared to using statically-selected IID training data.

[0091] In both of Figures 3 A and 3B, the learner model (machine learning model) was randomly initialized with learner model parameters 0i , the online model was trained online to update its online model parameters 0O. The online model was smaller than the learner model. In both situations (the multimodal learning task and the large classification task), a small pre-trained reference model with reference model parameters 0rwas also used in calculating scores used for selecting the training data subsets. In training, the models use a loss fact for scoring data and another loss fleam for computing updates. The training data comprised a dataset of size D (i.e. D training examples), the first training subset had a size B (i.e. B training examples) and the second training subset had a size b (i.e. b training examples) where b < B.

[0092] In training, the following algorithm is used. First, a first training subset X (i.e. first training subset 11) is selected from the training data D (i.e. training data 10) according to X~D, where |X| = B. Secondly, scores S (i.e. first scores) are determined for the first training subset according to S = fact (X| 0O) - fact (X| 0r). Thirdly, a second training subset Y (i.e. the second training subset 12) is sampled from the training data by sampling indices I according to I ~ SoftMax(S), where |I| = b, and selecting the training examples for the second training subset Y according to Y = X[I], Finally, the parameters of the machine learning model and online model are updated according to 0i <- Adam [Voo fleam (Y| 0i)] and 0O<- Adam [Vo0fleam (Y| 0O)], respectively.

[0093] From the above, the optional steps of training the learner model and the online model using the first training subset 11 are not used. Rather, the first training subset 11 is used simply to generate scores, and then the second training subset 12 selected, based on the scores, and used to train the learning model 110 and online model 120.

[0094] Steps of this method, and in particular the steps of scoring training examples, selecting a second subset, and training the machine learning model and online model using the second subset can be performed repeatedly. When scoring training examples in subsequent iterations, the scoring may comprise scoring one or more trainingexamples of the second training subset of the previous iteration. In this way, the scores for the initial first training subset can be updated so that they converge to best prioritize data within the first training subset. Alternatively, in each iteration of scoring and training, the first batch may be re-initialised (e.g. randomly sampled a further time) and the training examples therein scored, such that more of the training examples in the training data are scored. This reduces the risk that important training examples that were omitted from the first training subset can be reintroduced into training in later iterations.

[0095] Figure 4 shows an example illustrating the compute required to train various models, comparing the known method of training a model using IID sampled data versus using the methods described herein. Figure 4 generally illustrates that implementations of the described techniques can achieve significant reductions in training compute for the same final performance.

[0096] In more detail the example of Figure 4 shows a first training process 410, second training process 420, third training process 430, fourth training process 440 and fifth training process 450 is shown. In each of the training processes 410, 420, 430, 440, 450, the learner model comprises a Vision Transformer ViT-L model, with 304.4 x 106parameters (approximately 300 x 106parameters). Each model is trained to the same accuracy, in particular so the model output corresponds to the output obtained when training conventionally using IID training examples.

[0097] In each of the training processes 410, 420, 430, 440, 450, the learner model is trained to perform the same task, using training examples selected from the same training data set. In each of the training processes 410, 420, 430, 440, 450, the learner model is trained to achieve the same accuracy (e.g. a target performance). Figure 4 is drawn to scale and the length of each bar indicates the number of floating-point operations per second (FLOPs) required to perform a particular portion of the training processes. As is described in more detail below, the first training process 410 comprises only a single bar 411 representing the training of the learner model, whereas the second, third, fourth and fifth training processes 420, 430, 440, 450 comprise a first bar 421, 431, 441, 451 representing the training of the learner model, and separate bars representing the sub-processes of generating first scores (using the online model), training the online model, generating reference scores (using a reference model) and training the reference model. In other words, the training in each of the second, third, fourth and fifth training processes 420, 430, 440, 450 (and the costs associated with each sub-process of the training process) is amortized. The total length of the bars forany particular training process 410, 420, 430, 440, 450 represents the total number of FLOPs required to train the learner model to a target performance using each training process 410, 420, 430, 440, 450.

[0098] In the first training process 410, training examples are selected from the training data using IID. No online scoring or online selection is performed. As such, the first training process 410 comprises only a single process, represented by a first process first bar 411, corresponding to training the learner model using HD-selected samples.

[0099] In the second, third, fourth and fifth training processes 420, 430, 440, 450, training examples are selected from the training data using active learning as described above. In particular, the training examples are selected using an online model (e.g. online model 120) and a reference model (e.g. reference model 130). In each of the second, third, fourth and fifth training processes 420, 430, 440, 450, the online model and reference model are smaller than (have fewer parameters compared to) the learner model.

[0100] In the second training process 420, the online model comprises a Vision Transformer ViT-B model, with 86.6 x 106parameters (approximately 86 x 106parameters). The reference model also comprises a Vision Transformer ViT-B model, with 86.6 x 106parameters (approximately 86 x 106parameters). As such, in the second training process 420, a smaller, less computationally intensive, model is used to score training examples and aid the selection of training examples for provision to the learner model in training. A second process first bar 421 represents the number of FLOPs required to train the learner model, and second, third, fourth and fifth bars 422, 423, 424, 425 represent the number of FLOPs required for each of generating first scores (using the online model), training the online model, generating reference scores (using a reference model) and training the reference model during the second training process 420. In the second training process 420, the brace and associated arrows indicate the different sub-processes following amortization of the second training process 420 compared the first training process 410.

[0101] While the number of FLOPs required to train the learner model using the second training process 420 is smaller than the number of FLOPs required to train the learner model using the first training process 400 (based on comparing the first process first bar 411 and the second process first bar 421), it can be seen, from the length of the first process first bar 411 compared to the total length of each of the second process first, second, third, fourth and fifth bars 421, 422, 423, 424, 425, that the overall number ofFLOPs required for training a ViT-L learner model using a ViT-B online model and reference model (using the methods described herein) is more computationally expensive compared to training the ViT-L learner using IID selected data alone. This applies if the reference model is trained in an online process. However, as described above, the reference model may be pre-trained. For example, the reference model may be a foundation model pre-trained on a variety of downstream tasks using a foundational dataset. As such, the second process third bar 424 and fifth bar 425 may be removed from the total computational cost of the second process 420. By using a pretrained reference model, the second training process 420 is more cost-neutral compared to the first training process 410.

[0102] When a pre-trained reference model is used, the size of the reference model (e.g. the number of parameters) may be altered without significant impact on the training process or computational cost. For example, a large off-the-shelf reference model with the same number as, or more, parameters than the learner model may be used without a significant computation cost, because the computation cost has already occurred in pretraining the reference model.

[0103] In the third training process 430, the online model comprises a Vision Transformer ViT-S model, with 22.1 x 106parameters (approximately 22 x 106parameters). The reference model also comprises a Vision Transformer ViT-S model, with 22.1 x 106parameters (approximately 22 x 106parameters). A third process first bar 431 represents the number of FLOPs required to train the learner model, and second, third, fourth and fifth bars 432, 433, 434, 435 represent the number of FLOPs required for each of generating first scores (using the online model), training the online model, generating reference scores (using the reference model) and training the reference model during the third training process 430. It can be seen, from the combined length of the third process first, second, third fourth and fifth bars 431, 432, 433, 434, 435 compared to the length of the first process first bar 411, that the third training process 430 is overall less computationally burdensome than the first training process 410.

[0104] In the fourth training process 440, the online model comprises a Vision Transformer ViT-Ti model, with 5.7 x 106parameters (approximately 6 x 106parameters). The reference model also comprises a Vision Transformer ViT-Ti model, with 5.7 x 106parameters (approximately 6 x 106parameters). A fourth process first bar 441 represents the number of FLOPs required to train the learner model, and second,third, fourth and fifth bars 442, 443, 444, 445 represent the number of FLOPs required for each of generating first scores (using the online model), training the online model, generating reference scores (using the reference model) and training the reference model during the fourth training process 440. It can be seen, from the combined length of the fourth process first, second, third fourth and fifth bars 441, 442, 443, 444, 445, that the fourth training process 440 is overall less computationally burdensome than the first training process 410 and second training process 420.

[0105] In the fifth training process 450, the online model comprises a Vision Transformer ViT-Ti model, with 5.7 x 106parameters (approximately 6 x 106parameters). The reference model also comprises a Vision Transformer ViT-Ti model, with 5.7 x 106parameters (approximately 6 x 106parameters). A fifth process first bar 451 represents the number of FLOPs required to train the learner model, and second, third, fourth and fifth bars 452, 453, 454, 455 represent the number of FLOPs required for each of generating first scores (using the online model), training the online model, generating reference scores (using the reference model) and training the reference model during the fifth training process 440. It can be seen, from the combined length of the fifth process first, second, third fourth and fifth bars 451, 452, 453, 454, 455, that the fifth training process 450 is overall less computationally burdensome than the first training process 410, second training process 420 and third training process 430.

[0106] Figure 5 illustrates speedups that can be provided by some example implementations of a system as described above. Figure 5A shows learner model training speedup, computed as a percentage of learner iterations saved in order to achieve the top performance of the baseline (uniform sampling) on the y-axis, against the additional computation in FLOPS required to score the training examples. The particular example of Figure 5 relates to Vision Transformers (as described above) performing classification on the JFT-300M dataset (Sun et al., “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”, ICCV 2017); and with two training examples scored per training example used (p =2). Curve 500 relates to use of a prioritization score (learnability score) that measures a difference in loss as described above, and curve 510 relates to an approach which selects training examples to remove “easy” (low loss) examples. This illustrates the robustness of the former approach to varying scoring (online and reference) model size. Figure 5B, in which the curves are correspondingly labelled, illustrates the total compute speedup, showing that (if desired)training computation can be saved by using a scoring (online and reference) model that is smaller than the learner model.

[0107] Figure 6A shows relates to the same systems as Figure 5, and shows a measure of classification accuracy on the y-axis and on the x-axis, learner compute in FLOPs (Figure 6A) and total compute in FLOPs (Figure 6B). Figure 6 illustrates that a 32% reduction in learner compute, and a 25% or greater total reduction in compute can be achieved for this particular example.

[0108] As described above, the learner model 110, also referred to more generally as the machine learning model 110, can be trained to perform a particular task. In some implementations, the task comprises one or more of classifying images, processing images, generating images, transcribing audio utterances into text, text translation, audio translation, document classification, object recognition, controlling a robot. The training examples in the training data 10 can correspond to the training task, for example the task that the trained machine learning model is used to perform may generally correspond to a type of the training data item required for the task, for example image, audio, text, sensor data etc.

[0109] In some implementations the training examples comprise images. In some implementations the training examples comprise text tokens. The training examples may comprise data representing any digital input, for example images, text, audio signal (e.g. audio utterances), text (e.g. sentences, documents, words). The training examples may be multi-modal, i.e. the training data may include more than one type of training data (e.g. both images and text). A training example may be referred to as a training data item.

[0110] Example tasks and associated training

[0111] After training, the machine learning model 110 can be used to perform an image processing task. Performing the image processing task can comprise providing a still or moving image to a machine learning model trained as described above. Pixels of the image are processed using the machine learning model 110 to perform the image processing task. The training data for training the machine learning model 110 can similarly comprise still or moving images. The training is performed by processing pixels of the image through the machine learning model 110.

[0112] In some implementations a still or moving (video) image processed by the machine learning model 110, either during or after training, or both, may be an image that has been captured by a camera, i.e. that has been captured from the real world.Elements of the image data may comprise monochrome or color pixels of the image or video. The image may be a 2D or 3D image. As defined herein an “image” includes a point cloud e.g. from a LIDAR system, and a “pixel” includes a point of the point cloud. Similarly references to a moving image or video include a time sequence of point clouds. Objects in the image or video may comprise objects, e.g. physical objects, represented by the image or video.

[0113] As one example, the trained machine learning model 110 can be used to perform a still or moving image classification task (zero-shot). For example to classify an image into one of a plurality of classes, e.g., as a pickup truck, car, or van the global text embedding can be determined for each of a set of words or sentences that describe the image as belonging to a different respective class, e.g. “this is a photograph of a pickup truck”, and so forth. The global image embedding can be determined for the image, and the class of the image can be determined from the word or sentence that has a global text embedding that is most similar to the global image embedding. A similar approach may be used to classify actions in moving images, e.g. gestures; and to perform a multilabel classification.

[0114] The task that the trained machine learning model 110 is used to perform may generally correspond to a type of the training data item. The type of data may be referred to as a data modality. For example where the training data item comprises an audio data item, an image data item, a multimodal data item, a text data item, or a graph data item, the trained machine learning model, or part thereof, may be used, correspondingly, to process input data comprising audio data, image data, multimodal data, text data, or graph data respectively to perform an audio signal processing task, an image processing task, a multimodal processing task, a text processing task, or a graph processing task

[0115] As one example the training data item, and input data, may comprise audio data representing values of a digitized audio waveform, e.g. a time sequence of waveformrepresenting elements. Such a representation may comprise, e.g., samples representing digitized amplitude values of the waveform or a time-frequency domain representation of the waveform such as a STFT (Short-Term Fourier Transform) or MFCC (Mel- Frequency Cepstral Coefficient) representation. The audio waveform may comprise e.g. a speech waveform or a waveform of a sound, e.g. a captured sound. As some examples of transformations that may be used, transformed views of the training data item may be obtained by transformations including: time or pitch warps; random cropsin the time or frequency domain, e.g. selections of portions of the audio data item with random start and end times or with randomly selected upper and lower frequencies; modifications to the amplitude of a data item e.g. by randomly increasing or diminishing the amplitude of the audio; or modifications to the frequency characteristics of the audio e.g. by randomly filtering the audio. Objects in the audio may comprise e.g. speech elements such as words, syllables, or phonemes; or events or other distinguishable audio objects in the sound.

[0116] The audio signal processing task may comprise, e.g.: processing audio data representing speech to provide output data that detects words or phonemes in the speech or categorizes words or phonemes in the speech into one or more of a plurality of categories; or processing audio data representing a sound to provide output data, e.g. likelihood data, that detects presence of a particular sound or audio object or event in the sound e.g. in a hotword detection or identification task; or processing audio data representing a sound to provide output data that categorizes a content of the sound into one or more of a plurality of categories (i.e. classifying a sound). In some further examples the audio signal processing task may comprise, e.g.: an identification or classification task such as a speech or sound recognition task, e.g. a hotword detection or identification task, a speaker or natural language classification task, or an audio tagging task, in which case the output data may comprise a category score or tag for the audio or for a segment of the audio; or a similarity determination task e.g. an audio copy detection or search task, in which case the output data may comprise a similarity score.

[0117] In some implementations the training data item, and input data, may comprise sensor data representing values of a digitized sensor waveform i.e. a sensor other than an audio sensor may be used to obtain the digitized waveform. The digitized sensor waveform may be treated similarly to a digitized audio waveform, and the transformed views may correspond with those described above. The sensor data may generated by sensors configured to monitor the real-world state, condition or environment of a physical system, e.g. of a mechanical or electronic physical system or machine, e.g. sensing force, pressure, movement, temperature, or vibration. The objects may comprise events or other distinguishable objects in the sensor data, or conditions of the physical system. The signal processing task may be to process the input data to provide output data that identifies the presence of one or more of the events, objects, conditions or environments.

[0118] As another example the training data item, and input data, may comprise image data representing a still or moving image, i.e. an image or video, e.g. an image or video that has been captured using a camera. Elements of the image data may comprise monochrome or color pixels of the image or video. As defined herein an “image” includes a point cloud e.g. from a LIDAR system, and a “pixel” includes a point of the point cloud. Similarly “video” includes a time sequence of point clouds. Objects in the image or video may comprise objects, e.g. physical objects, represented by the image or video.

[0119] The image processing task may comprise, e.g.: processing the image data to provide output data that identifies the location of one or more specified or unspecified objects in the image or video, e.g. output data that defines one or more object bounding shapes or boxes; or processing the image data to provide output data that segments pixels of the image or video into regions that represent one or more objects in the image or video signal; or processing the image data to provide output data that categorizes a content of the image or video into one or more of a plurality of categories; or processing the image data to provide output data that predicts depth values for pixels of the image or video. A task that segments the pixels, e.g. a semantic segmentation task, can associate each pixel with a category representing a class of objects, or an instance segmentation task that associates each pixel with a category representing an instance of an object, i.e. to distinguish between different instances of the same category of object. For example a medical image may be processed to label pixels of the medical image in accordance with which region of a human or animal body they show, or to identify pixels of the medical image in which a particular medical condition is present.

[0120] Where the image data comprises pixels of a video the image processing task may comprise, e.g.: processing the image data to provide output data that identifies the location of one or more actions represented in the video; or processing the image data to provide output data that categorizes one or more actions, e.g. gestures, represented in the video into one or more of a plurality of categories.

[0121] In general the image processing task may include any sort of image processing or vision task such as an image classification or scene recognition task, an image segmentation task e.g. a semantic or instance segmentation task, an object localization or detection task, or a depth estimation task. When performing such a task the input data may be derived from pixels of the image. For an image classification or scene recognition task the output may comprise a classification output providing a score foreach of a plurality of image or scene categories e.g. representing an estimated likelihood that the image data or an object represented in the image data, or that an action within image data representing a video, belongs to a category of a set of categories. For an image segmentation task the output may comprise, for each pixel, an assigned segmentation category or a probability that the pixel belongs to a segmentation category, e.g. to an object or action represented in the image or video. For an object localization or detection task the output may comprise data defining coordinates of a bounding box or region for one or more objects represented in the image. Such a bounding box or region may be defined in two, three or more dimensions (time counting as a dimension). For a depth estimation task the output may comprise, for each pixel, an estimated depth value. The output may define a continuous value or it may define a probability distribution over discrete depth value buckets, such that the output pixels define a (spatial 3D) depth map for the image. Such tasks may also contribute to higher level tasks, e.g. to object tracking across video frames; or to gesture recognition i.e. recognition of gestures that are performed by entities depicted in a video. As another example, the image processing task may include an image keypoint detection task in which the output comprises the coordinates of one or more image keypoints, such as landmarks of an object represented in the image, e.g. a human pose estimation task in which the keypoints may define the positions of body joints. A further example is an image similarity determination task, in which the output may comprise a value representing a similarity between two images, e.g. as part of an image search task.

[0122] As another example, in a pose estimation task, the pixel values may map the pixels to a 3D surface, e.g. of a human body or face. Or the pixel values may estimate a 6D pose representing translation and orientation components of an object in the image, e.g. in quaternion form. The set of pixel values for the image can estimate the pose of one or more objects in the image.

[0123] As another example, in a surface normal estimation task the pixel values for the image may comprise a vector in, e.g., three dimensions defining a surface normal. The set of pixel values for the image can provide a surface normal map for one or more objects in the image, e.g. for use in an augmented reality or other application.

[0124] As another example the input data, may comprise text data; elements of the text data may comprise e.g. sentences, words, or parts of words e.g. wordpieces. The text processing task may comprise, e.g.: a part-of-speech tagging task, in which case the output data may comprise e.g. a category score or tag for the text or for a segment of thetext; or a dependency parsing task, in which case the output data may comprise data representing a dependency parse of the text; or a text segmentation task, in which case the output data may comprise data that associates elements of the text with one or more of a plurality of categories for the text. Other example tasks include an identification or classification task, or a similarity determination task, e.g. to generate a category score, a similarity score, or a tag as described above; or a machine translation task.

[0125] As another example the training data item, and input data, may comprise multimodal data. In general such multimodal data is a combination of two or more different types of data, where the different types of data represent the same or overlapping objects using the different modalities (types). As one example the multimodal data may comprise audio-visual data, comprising a combination of pixels of an image or of video and audio data representing values of a digitized audio waveform. As another example the multimodal data may comprise a combination of i) text data representing text in a natural language and ii) pixels of an image or of video or audio data representing values of an audio waveform. Elements of the multimodal data may correspond to elements of the data types making up the combination. Optionally, but not necessarily, when processing multimodal data the data may be mapped into a common embedding space.

[0126] In general the multimodal processing task may correspond to any of the tasks previously described for any of the types of data making up the multimodal combination. For example, an accuracy of the previously described tasks may be increased when the task is applied to multimodal data combining the data for which the task has been previously described and another type of data. For example detection or classification of an object or event may be improved when data of multiple different types (modalities) is processed.

[0127] As one particular example, where the multimodal data comprises audio-visual data the multimodal processing task may comprise: processing the combination, i.e. the image / video and audio, to provide output data that detects presence of a particular multimodal object or event in the combination (e.g. to identify a phoneme or viseme when lip reading); or processing the combination to provide output data that categorizes the combination into one or more of a plurality of categories, e.g. by defining a score for each category of a plurality of possible categories for the combination. As another particular example, where the multimodal data comprises a combination of text data and image or video or audio data the multimodal processing task may comprise processingthe combination to provide output data that defines whether the image or video or audio waveform is described by the text, e.g. by a particular caption, e.g. by defining a score for the text or caption.

[0128] As another example the training data item, and input data, may comprise graph data; in such implementations the machine learning model 110 described herein may comprise graph neural networks. In general the graph data may define a graph structure having a set of nodes with associated node feature vectors connected by edges which may have associated edge feature vectors. A graph may, but need not be, defined by an adjacency matrix e.g. where N is the number of nodes, an A? x iV matrix defining which nodes are connected by edges. Elements of the graph data may comprise e.g. nodes or edges of a graph represented by the graph data.

[0129] A graph may represent a real-world physical system; merely as some examples, a mechanical structure in which bodies are connected by joints, or a structure of a molecule such as a drug molecule. The objects may comprise e.g. physical bodies or parts of a molecule e.g. chemical moieties. The graph processing task may comprise e.g.: characterizing a physical entity represented by the graph to provide output data that defines a predicted stability of the physical structure or molecule, or the binding affinity of a molecule represented by the graph with another molecule e.g. to identify a drug candidate (which may then be evaluated by synthesizing the molecule and e.g. testing the molecule in vitro or in vivo). The predicted stability of the physical structure may be used e.g. to design or evaluate a structure; the result may then be used to construct a structure to the design. As another example the graph may be a scene graph that represents a scene; the scene graph may have been generated from a captured real-world image. The graph processing task may then comprise generating output data that identifies or classifies the scene or one or more objects within the scene e.g. to facilitate object / scene editing or information extraction for scene interpretation.

[0130] In some applications the trained machine learning model 110 may comprise one or both of a trained image encoder neural network and a trained text encoder neural network, each configured to output a respective embedding. In general the image encoder neural network is configured to receive an input comprising pixels of a still or moving image, and to process the input, in accordance with the trained parameters of the neural network, to generate as an output a set embeddings for the image. Throughout this specification, an “embedding” of an entity (e.g., an observation of anenvironment) can refer to a representation of the entity as an ordered collection of numerical values, e.g., a vector or matrix of numerical values.

[0131] In general the text encoder neural network is configured to receive an input comprising text defining a sequence of text tokens from a vocabulary of tokens, and to process the input, in accordance with the trainable / trained parameters of the text encoder neural network, to generate as an output the sequence of token embeddings for the text.

[0132] As an example the visual encoder neural network and / or the text encoder neural network may comprise a Transformer neural network. For example the visual encoder neural network may have an architecture similar to a vision transformer neural network, and the text encoder neural network may comprise an encoder-only, encoder-decoder, or decoder-only Transformer neural network.

[0133] In general a multimodal machine learning model has a multimodal input configured to receive a first multimodal input and a second multimodal input. As used herein a “modality” refers to a type of data, and thus a multimodal machine learning model is one that can process multiple different types of data.

[0134] The first multimodal input may comprises a text input to receive a sequence of text. The second multimodal input may be configured to receive a different type of input data, e.g. it may comprises a visual input to receive an image or video.Alternatively the second multimodal input may be configured to receive, e.g., audio data representing values of an audio waveform, e.g. instantaneous amplitude data or time-frequency domain data; or data representing observations (not necessarily visual) of an environment with which an agent controlled by the multimodal machine learning model interacts. In some implementations there may be more than two different multimodal inputs, each configured to receive a different type of data.

[0135] The multimodal machine learning model may be configured to jointly process an encoded version of the text and an encoded version of the second multimodal input, e.g. of the image or video, to generate a model output that defines a result of a machine learning task. A few examples of machine learning tasks that can be performed by such a multimodal machine learning model are described later.

[0136] The text received may comprise text in one or more natural languages, or text in a computer language, or both. The computer language may be any formal language used to communicate with a computer, e.g. a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programminglanguage. The text may be received, e.g., as a series of encoded characters, e.g. UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like.

[0137] The multimodal machine learning model can include a text encoder 114 that processes the sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g. that each represent words, wordpieces or characters in a natural or computer language.

[0138] Where the second multimodal input comprises an image or video it may comprise image data defining color or intensity values for pixels of a still or moving image in one, two, or three dimensions. As used herein “image” includes a LIDAR point cloud, and the image data may also or instead define the locations of points of a still or moving point cloud. As another example, an image or video received by the second multimodal input may comprise a neural 3D representation, e.g. that represents a 3D scene as a set of latent feature vectors, e.g. a neural radiance field representation. The multimodal machine learning model can include second multimodal input encoder that processes the second multimodal input, e.g. using one or more convolutional, attention, fully connected, or recurrent layers, to generate the encoded version of the second multimodal input. In general such an encoder may implement any form of encoding appropriate for the type of data to be encoded. Merely as an example, where the second multimodal input comprises an image or video this may be encoded, e.g., as features for each of a set of patches that tile the image, or as a sequence of visual tokens selected from a vocabulary of visual tokens, or as a representation of distinct objects in the visual input. Such visual tokens may, but need not be, interleaved with text tokens processed by the model.

[0139] The model output may comprise any form of output appropriate to the machine learning task performed by the multimodal machine learning model. For example the model output may comprises text in a natural or computer language that defines a result of the task, e.g. for tasks such as image captioning, visual question answering, or object detection or instance segmentation. Also or instead the model output may comprise data defining an image, video or audio object, e.g. in a generative task; or the model output may comprise non-textual action selection data for selecting an action to be performed by an agent controlled by the model. As another example the model output may also or instead define an intermediate step to be performed during the task, e.g. a call to a software API for a software tool that is used when performing the task; themultimodal input may then receive an output from the software tool that is used to generate a final model output that performs the task. A few particular examples of model output are given later.

[0140] Such a multimodal model can be trained using very large (but possibly noisy) datasets in which text is paired with an image and / or with one or more other types of data, e.g. audio data, or data relating to the operation of an agent acting in an environment to perform a variety of tasks. Such a model is can be trained, e.g., using self-supervised learning. The pairing can often be imperfect, and the training dataset can, but may not, include any actual examples of a particular task to be performed, but nonetheless an ability to perform a particular task can emerge. There are many examples of suitable, publically available training datasets.

[0141] Some example multimodal machine learning models with which the techniques described herein may be used include: Flamingo (Alayrac et al. arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794); and PaLI- X (Chen et al. arXiv:2305.18565).

[0142] In some implementations, the second multimodal input can include an observation characterizing an environment of an agent performing a task, e.g. a mechanical agent or software agent. The observation may characterize the environment at a particular time step and the model output may define one or more actions to be performed by the agent at the time step. For example each action may be expressed as a sequence of text, e.g. as one or more characters such as letters and numbers, that represents the action, or as text that defines a low-level “skill” from a set of skills; or the model output may, e.g., define parameters of a probability distribution from which an action is selected. Optionally the text received by a text input may include text describing the task to be performed. Optionally the text input may include a description of one or more actions performed at a preceding time step. Where the agent is a software agent the model output may comprise a text output for calling a software API at a time step, and the model input at a subsequent time step, e.g. the text input, may comprise a response from the software agent, e.g. from the API.

[0143] Some examples of multimodal machine learning models controlling an agent, and with which the techniques described herein may be used, are described in: PaLM-E (Driess et al. arXiv:2303.03378); RT-1 (Brohan et al. arXiv:2212.06817); and RT-2 (Brohan et al. arXiv:2307.15818).

[0144] Such a multimodal machine learning model can have an audio input, or an agent action input to receive agent action data representing an action of an agent performing a task in an environment. Data received in this way may be jointly processed with data from a text input and from a second multimodal input to generate the model output.

[0145] Such a multimodal machine learning model has a multimodal input and can, in implementations, perform a range of different tasks. However in implementations not every task that the model performs requires a multimodal input, e.g. a task to generate an image from a text description of the image, or an image captioning task. In some implementations, after training, the text input can be used to specify a particular task that is to be performed by the multimodal machine learning model, e.g. by providing a “prompt” to the model describing the task to be performed or giving an example of the task as a prompt to the model. Such prompts may optionally be included in the training data.

[0146] The multimodal machine learning model is configured to process the multimodal input in accordance with the trainable parameters of the multimodal machine learning model, to generate a model output that defines a result of one or more machine learning tasks. A training system can include a training engine to train the multimodal machine learning model, i.e. to update values of the trainable parameters , to perform the machine learning task(s), using training data items stored in one or more training datasets.

[0147] In general each training data item comprises multimodal data for use in training the multimodal machine learning model, e.g. using a self-supervised training objective. There are many different types of self-supervised objective function that may be used. As one example the model may be trained using a softmax cross entropy loss, e.g. using language model style teacher forcing with a softmax cross entropy loss. As another example the model may be trained with an autoregressive negative log likelihood (NLL) loss, such as — SfL:1log p(yE| y<E,xsi) for a multimodal input comprising a sequence of text encoded as L tokens with the Ith text token yEconditioned on preceding second modality inputs x<E, such as one or more images or videos, and conditioned on preceding text tokens y<E. As another example the model may be trained with a masking loss, e.g. a loss that requires the model to predict masked-out data such as masked out text tokens. As another example the multimodal machine learning modelcan be trained using a self-supervised objective function that comprises a contrastive loss function (one that is dependent upon a positive example and one or more negative examples).

[0148] As another example such a multimodal machine learning model can also or instead be trained using a reinforcement learning objective, e.g. when the model is used to control an agent to perform a task. The methods of training described herein are particularly suited for reinforcement learning, where the process of exploration is decoupled from the process of learning. In reinforcement learning, prioritizing training examples from which the model will learn more can result in more efficient training.

[0149] Each training data item may comprise, e.g., an example sequence of text and an example of the second modality input, e.g. an example image or video; in general these are semantically related to one another (but not always, as the training dataset may be noisy). As an example, matched, text and image or video data and may be obtained from web pages, e.g. from images or videos and their corresponding alt-text (text from the HTML or XHTML alt attribute); or from web pages where images or video and text are interleaved with one another. One example of such a dataset is WebLI (Web Language Image, Chen et al. arXiv:2305.18565vl). Training datasets for other types of second modality input can similarly be obtained from web pages. Such training datasets can be large, e.g. > 107, 10sor 109items.

[0150] Also or instead, smaller but more specialized training datasets can be used, e.g. to fine tune a model for a particular task or tasks. A few examples for visual tasks are the Visual Genome dataset for Visual Question Answering (Krishna et al., arXiv: 1602.07332); Objects365 (Shao et al., “Objects365: A large-scale, high-quality dataset for object detection”, IEEE / CVF international conference on computer vision, pages 8430-8439); Open Images V4 (Kuznetsova et al., arXiv: 1811.00982); the SBU dataset (Ordonez et al. “Im2Text: Describing Images Using 1 Million Captioned Photographs”, NeurlPS 2011); the Conceptual Captions datasets, e.g. VI (2M images) or V2 (10M images) (Sharma et al., “Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning”, ACL 2018); and Kinetics for video (Kay et al., arXiv: 1705.06950). An example task-specific training dataset for audio data is AudioSet (Gemmeke et al., “Audio set: An ontology and human-labeled dataset for audio events,” ICASSP, IEEE, 2017, pp. 776-780). An example task-specific training dataset for agent (robot) control is described in Ebert et al., arXiv:2109.13396.

[0151] Example tasks for multimodal models such as VLMs

[0152] In general a multimodal machine learning model can be trained to perform any sort of machine learning task or tasks. After the multimodal machine learning model has been trained it can be deployed for use in performing the machine learning task(s). For instance, the machine learning model can be deployed in an environment that enables users to provide requests for the machine learning model to process specified multimodal inputs to generate corresponding model outputs. Users can provide the requests, e.g., by way of a user interface or through an application programming interface (API). The requests can be transmitted from a user device (e.g., over a data communication network, e.g., the internet) to one or more computers implementing the machine learning model, e.g., in a data center. The machine learning model can process multimodal inputs specified by user requests to generate corresponding model outputs, and then transmit the model outputs to user devices (e.g., over a data communication network).

[0153] In some implementations, after training, a particular task that is to be performed by the multimodal machine learning model can be described by part or all of the sequence of text in the multimodal input to the model. For example in a multimodal input that includes an image, video, or audio item such a prompt might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image, video, or audio item]”, or “Detect a person”. Where the model is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. Also or instead such a prompt may give one or more examples of a task to be performed. A multimodal machine learning model can be trained on multiple natural and / or computer languages and the prompt may then specify a language to use.

[0154] A few examples of some machine learning tasks that can be performed by a model trained as described herein follow.

[0155] For some tasks the second modality input represents an image or video as previously described, e.g. from a camera or other imaging device that captures the image or video from a real-world environment, and / or audio, e.g. audio data such as speech or other sounds captured from a real-world environment. In general the tasks described below may be tasks that require spatial awareness or other context from theimage, video, or audio item. For example, a prompt may ask “What is the object in the top left corner?”, or “What was the answer to the spoken question?”.

[0156] As one example the task may comprise an object or action detection task. A task-specific training data item may comprise an image, video, or audio item containing one or more objects or actions, and a sequence of text. The sequence of text may describe or otherwise label the object(s) or action(s) and (for an image or video) may include text giving bounding box coordinates for the object(s) or action(s). After training, when the model is used in inference, the model output 122 may comprise or represent text that describes or otherwise labels detected object(s) or action(s) in the second modality input, and may (for an image or video) include bounding-box coordinates for the detected object(s) or action(s), e.g. " 10 20 90 100 cat 20 30 100 100 dog”.

[0157] As another example the task may comprise a classification task, e.g. an object or action classification task. A task-specific training data item may comprise an image, video, or audio item containing one or more objects or actions and a sequence of text. The sequence of text may describe or otherwise classify the object(s) or action(s). After training, when the model is used in inference, the model output may comprise data, e.g. text, that classifies the object(s) or action(s) in the second modality input into one of a plurality of classes.

[0158] As another example the task may comprise an image, video, or audio item describing task, e.g. a captioning task (which, as used here, includes an audio description task to explain what is happening in a video). A task-specific training data item may comprise an image, video, or audio item and a sequence of text describing the image, video, or audio item. After training, when the model is used in inference, the model output may comprise data, e.g. text, describing an image, video, or audio item in the second modality input. For example the model output may provide a caption or description for a second modality input item, or it may count objects in the second modality input item, or it may provide some other form of description of the second modality input item.

[0159] As another example the task may comprise an image, video, or audio questionanswering task. A task-specific training data item may comprise an image, video, or audio item and a sequence of text that describes the image, video, or audio item. After training, when the model is used in inference, the model output may comprise data, e.g. text, that answers a question about the second modality input specified in a promptsequence of text, e.g. as described above. This may be used, e.g., to answer questions about visual plots and charts or about sounds.

[0160] As another example the task may comprise a character or word recognition task, e.g. an OCR (optical character recognition) task. A task-specific training data item may comprise an image, video, or audio item and a sequence of text that includes text that is depicted in the image or video, or that is represented as speech in the audio item. After training, when the model is used in inference, the model output may comprise text that represents characters or words in the second modality input, e.g. in a natural language.

[0161] As another example the task may comprise a still or moving image or audio generation task. A task-specific training data item may comprise an image, video, or audio item and a sequence of text that describes the image, video, or audio item. After training, when the model is used in inference, the model output may comprise data for an image, video, or audio item, e.g. image data defining values for pixels of a still or moving image or audio data representing values of an audio waveform, and the sequence of text in the multimodal input to the model may describe or characterize the image, video, or audio item to be generated.

[0162] As another example the task may comprise a computer language text generation task. A task-specific training data item may comprise an image, video, or audio item and a sequence of text in a computer language for generating the image, video, or audio item. After training, when the model is used in inference, the model output may comprise text in the or another computer language for generating or rendering an image, video, or audio item in the second modality input, e.g. a web page, plot, or chart.

[0163] In another example of a computer language text generation task a task-specific training data item may comprise an image, video, or audio item and a sequence of text in a computer language for performing a task in relation to the image, video, or audio item, e.g. a data processing task that involves analyzing the content of the image, video, or audio item to provide a result of the analysis or, e.g., a search to search for information relating to the content of the image, video, or audio item. The computer language in the model output may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output may be formatted as a JSON object. As previously, the sequence of text in the multimodal input may define the task to be performed and the second modality input may comprise, e.g. an image, video, or audio item in relation to which the task is to be performed, e.g. a task that involves manipulation of particular types of data that may benefit fromaccess to an API such as mathematical data, date / time related data, scientific data, recent data that may post-date training of the model (that may be accessed by a search function or API), and so forth. After training, when the model is used in inference, the model output may comprise text in the or another computer language for performing a task, e.g. as described above, in relation to an image, video, or audio item in the second modality input. The method may then include using the text in the computer language to perform the task.

[0164] In general where the model output comprises text this may be provided as speech representing the text.

[0165] In some implementations the machine learning task comprises an agent control task in which the agent interacts with an environment to perform the agent control task. In these implementations the multimodal input includes an observation characterizing the environment. For example the multimodal input can include a sequence of text that defines the task to be performed by the agent and the second modality input can represents an image, video, audio, or other observation of the environment, e.g. captured by a camera or other imaging device, or by a microphone, from a real-world environment. A task-specific training data item may comprise a sequence of text representing one or more actions of the agent, and a second modality input representing an observation of the environment. After training, when the model is used in inference, the model output comprises an action selection output, e.g. including text, that is used to select one or more actions to be performed by the agent in the environment in response to the observation. As an illustration the model output 122 may define an action as text such as “A: 132 114 128 5 25 156”, that can be converted into a control signal for a mechanical agent, such as a robot, e.g. “AT = [0.1, -0.2,0] AJ? = |[10e,25e,— 7°]”. As another example the action selection output may also or instead define one or more low- level skills, e.g. from a vocabulary of previously leamt skills. As before, the sequence of text in the multimodal input to the model may describe the task to be performed, e.g. “What action should the robot take to [perform task]”.

[0166] In some agent control implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real -world environment to perform the task. Forexample, the agent may be a robot or other mechanical agent interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. In these implementations, the observations may include, e.g., one or more of images, object position data, and sensor data to capture observations as the agent interacts with the environment. The actions may define control signals to control the robot or other mechanical agent, e.g., positions, torques, or other control signals for the parts of the mechanical agent, or higher-level control commands.

[0167] In some agent control implementations the agent can be a software agent, i.e. a computer program, configured to perform a task. Some examples where the agent is a software agent now follow.

[0168] As one example the environment may be an integrated circuit design and the task may be a routing task for routing interconnection lines of the integrated circuit. The observations may be of component positions and / or interconnections, and the actions may comprise component placing or interconnect routing actions. An integrated circuit with interconnection lines routed as determined may then be fabricated.

[0169] As another example the environment may be a real-world computing environment and the task may be to manage the distribution of jobs or tasks across computing resources e.g. on a mobile device and / or in a data center. The observations may include observations of computing resources such as compute or memory capacity, or Internet-accessible resources, or that relate to the operation of the computing resources in processing the jobs or tasks; and the actions may include assigning jobs or tasks to particular computing resources.

[0170] As another example the environment may be a real-world computing environment and the task is to manage the processing, e.g. by one or more real-world servers, of a queue of continuously arriving jobs. The observations may comprise observations of the times of departures of successive jobs, or the time intervals between the departures of successive jobs, or the time a server takes to process each job, or the arrival times, or time intervals between the arrivals, of successive jobs, or data characterizing the type of job(s). The actions may comprise actions that allocate particular jobs to particular computing resources.

[0171] As another example the environment may comprise a real-world computer system or network and the task may be to maintain security of the computer system ornetwork. The observations may comprise any observations characterizing operation of the computer system or network, and the actions may comprise actions to control the operation e.g. to limit or correct abnormal or undesired operation e.g. because of the presence of a virus or other security breach.

[0172] As another example the environment may comprise a data packet communications network environment, and the task may be to route packets of data over the communications network. The actions may comprise data packet routing actions and the observations may comprise, e.g., observations of a routing table which includes routing metrics such as a metric of routing path length, bandwidth, load, hop count, path cost, delay, maximum transmission unit (MTU), and reliability.

[0173] In some agent control implementations the agent may be a human agent and the environment may be a real -world environment. For example the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real-world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g. a monitoring system such as a video camera or sound capture system, to capture visual and / or audio observations of the user performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task.

[0174] Certain novel aspects of the subject matter of this specification are set forth in the claims below.

[0175] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0176] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0177] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field- programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0178] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0179] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0180] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantlyaccelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0181] Computers capable of executing a computer program can be based on general- purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The essential elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0182] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, accessspeed requirements, cost considerations, and the desired level of portability or permanence.

[0183] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0184] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0185] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on- device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice ofcomponents will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0186] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0187] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0188] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above shouldnot be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0189] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A computer-implemented method of training a machine learning model to perform a task, comprising: obtaining training data comprising a plurality of training examples; selecting a first training subset comprising one or more training examples from the training data; determining a first score associated with one or more of the training examples of the first training subset by processing, using an online model, one or more training examples of the first training subset; selecting a second training subset comprising one or more training examples from the training data, wherein each training example in the second training subset is selected based upon a prioritization score associated with the training example, the prioritization score based on the first score obtained using the online model; training, using the second training subset, the machine learning model; and training, using the second training subset, the online model.

2. The method of claim 1, further comprising, prior to selecting the second training subset: training, using the first training subset, the machine learning model; training, using the first training subset, the online model.

3. The method of claim 1 or 2, wherein the first score is indicative of a loss associated with processing one or more of the training examples using the online model.

4. The method of any of claims 1 to 3, wherein the online model and machine learning model are trained in parallel.

5. The method of any of claims 1 to 4, wherein the prioritization score is further based on a second score obtained using a reference model, wherein the reference model comprises a trained model.

6. The method of any of claims 1 to 4, wherein the prioritization score is further based on a second score obtained using a reference model, the method further comprising training the reference model using a third training subset selected from the training data.

7. The method of claim 6, wherein training the reference model is performed with a reference model 130 learning rate which is higher than a machine learning model learning rate used for training the machine learning model and an online model learning rate used for training the online model.

8. The method of claim 6 or 7, wherein the reference model 130 is trained in parallel with the machine learning model and online model.

9. The method of claim 5, wherein the second score is obtained from a precalculated set of second scores.

10. The method of any of claims 5 to 9, wherein the second score is indicative of a loss associated with processing one or more of the training examples using the reference model.

11. The method of any of claims 5 to 9, wherein the prioritization score is based on a combination of the first and second scores.

12. The method of any of claims 5 to 10, wherein the reference model has reference model parameters and wherein the number of reference model parameters is smaller than the number of machine learning model parameters.

13. The method of any preceding claim, wherein the online model has online model parameters and the machine learning model has machine learning model parameters; wherein the number of online model parameters is smaller than the number of machine learning model parameters.

14. The method of any preceding claim, wherein the first and / or second training subset comprise less than or equal to half of the plurality of training examples in the training data.

15. The method of any preceding claim, wherein the online model, and when dependent on claim 5 the reference model, have at least 4x fewer, 13x fewer, or 37x fewer parameters than the machine learning model.

16. The method of any preceding claim, wherein the training examples comprise images.

17. The method of any preceding claim, wherein the training examples comprise text tokens.

18. The method of any preceding claim, wherein the task comprises one or more of classifying images, processing images, generating images, transcribing audio utterances into text, text translation, text generation audio translation, document classification, object recognition, controlling a robot.

19. A computer-implemented method of performing a task, the method comprising: providing input data to a machine learning model, wherein the machine learning model has been trained to perform the task, the training comprising: obtaining training data comprising a plurality of training examples; selecting a first training subset comprising one or more training examples from the training data; determining a first score associated with one or more of the training examples of the first training subset by processing, using an online model, one or more training examples of the first training subset; selecting a second training subset comprising one or more training examples from the training data, wherein each training example in the second training subset is selected based on a prioritization score associated with the training example, the prioritization score based on the first score obtained using the online model; training, using the second training subset, the machine learning model; and training, using the second training subset, the online model.

20. The method of claim 19, further comprising, prior to selecting the second training subset: training, using the first training subset, the machine learning model; training, using the first training subset, the online model.

21. A machine learning model trained according to any of claims 1 to 18.

22. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-20.

23. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the method of any one of claims 1-20.

Images