Allocating computing resources between model size and training data during machine learning model training
Through allocation mapping technology, the size and training data volume of machine learning models are optimized, and the resource waste and performance problems of training large models under limited computing resources are solved, achieving efficient use and performance improvement of computing resources.
Patent Information
- Application Number
- CN202380037218.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-29
- Filing Date
- 2023-03-29
- Publication Date
- 2024-11-29
AI Technical Summary
In the case of constrained computing resources, it is difficult to optimize the balance between model size and training data volume when training large machine learning models, resulting in waste of resources and poor performance.
Through allocation mapping technology, the target model size and target training data volume are determined, and the performance of the machine learning model is optimized, so that it can achieve efficient use of computing resources under a given computing budget.
It significantly improves the efficiency of computing resource usage during training, reduces overtraining and resource waste, and improves the performance of the model on machine learning tasks.
Smart Images

Figure CN119053950A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to processing data using machine learning models. Background Art
[0002] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values of the model parameters.
[0003] Some machine learning models are deep models that use multiple layers of models to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the invention
[0004] This specification generally describes a training system, which is implemented as a computer program on one or more computers located in one or more locations, which can train a machine learning model to perform a machine learning task. As described below, the training system can perform various methods and can be implemented in various systems to train a machine learning model.
[0005] In one aspect, a method for training a machine learning model performed by one or more computers is described. The method includes: obtaining data defining a computational budget, the computational budget characterizing an amount of computational resources allocated for training a machine learning model to perform a machine learning task; and processing the data defining the computational budget using an allocation map according to a set of allocation map parameters to generate an allocation tuple, the allocation tuple defining: (i) a target model size for the machine learning model, and (ii) a target amount of training data for training the machine learning model. According to the method, selecting the model size of the machine learning model as the target model size and training the machine learning model on the target amount of training data is predicted to optimize the performance of the machine learning model on the machine learning task subject to the constraint that the amount of computational resources used to train the machine learning model meets a threshold defined by the computational budget. The method further includes: instantiating the machine learning model, wherein the machine learning model has a target model size; obtaining the target amount of training data for training the machine learning model; and training the machine learning model having the target model size on the target amount of training data.
[0006] In some embodiments, the value of the allocation mapping parameter set is determined by operations including the following: identifying multiple trial allocation tuples, each trial allocation tuple defining: (i) a trial model size of the machine learning model, and (ii) a trial amount of training data for training the machine learning model; for each of the multiple trial allocation tuples, determining a performance metric characterizing the performance of the trial machine learning model on the machine learning task, wherein the performance metric is a result of selecting the model size of the trial machine learning model as the trial model size and training the trial machine learning model on the trial amount of training data; and determining the value of the allocation mapping parameter set based on the performance metric corresponding to the multiple trial allocation tuples.
[0007] In some embodiments, determining the value of the allocation mapping parameter set based on performance metrics corresponding to multiple trial allocation tuples includes: for each of multiple computational budgets, determining an optimal model size and an optimal amount of training data corresponding to the computational budget based on the performance metrics corresponding to the multiple trial allocation tuples; and determining the value of the allocation mapping parameter set based on the optimal model size and the optimal amount of training data corresponding to each of the multiple computational budgets.
[0008] In some embodiments, determining the values of the allocation mapping parameter set based on the optimal model size corresponding to each of the multiple computational budgets and the optimal amount of training data includes: fitting the values of the allocation mapping parameter set based on the optimal model size corresponding to each of the multiple computational budgets and the optimal amount of training data.
[0009] In some embodiments, determining an optimal model size and an optimal amount of training data corresponding to a computational budget for each of a plurality of computational budgets includes: determining a corresponding performance curve for each of a plurality of experimental model sizes based on performance metrics corresponding to a plurality of experimental allocation tuples, wherein the performance curve for the experimental model size defines a continuous mapping from possible computational budgets to predictive performance metrics, and wherein the predictive performance metrics corresponding to the possible computational budgets define predictive performance of an experimental machine learning model with the experimental model size trained using an amount of computational resources that satisfies a threshold defined by the possible computational budgets; and determining an optimal model size and an optimal amount of training data corresponding to each computational budget using the performance curve.
[0010] In some implementations, determining the performance curve for the trial model sizes includes determining the performance curve for the trial model sizes by interpolating performance metrics for trial allocation tuples corresponding to the trial model sizes.
[0011] In some embodiments, using performance curves to determine the optimal model size and the optimal amount of training data corresponding to each computational budget includes: for each computational budget among multiple computational budgets: determining an optimal performance curve that achieves the optimal performance metric for the computational budget from the performance curves; determining the optimal model size as the experimental model size corresponding to the optimal performance curve; and determining the optimal amount of training data based on the computational budget and the optimal model size.
[0012] In some embodiments, determining, for each of a plurality of computational budgets, an optimal model size and an optimal amount of training data corresponding to the computational budget includes: determining, for each of the plurality of computational budgets, a corresponding performance curve based on performance metrics corresponding to a plurality of trial allocation tuples, wherein the performance curve for the computational budget defines a continuous mapping from possible model sizes to predictive performance metrics, and wherein the predictive performance metrics corresponding to the possible model sizes define predictive performance of experimental machine learning models with possible model sizes trained using an amount of computational resources that satisfies a threshold defined by the computational budget; and determining, using the performance curve, an optimal model size and an optimal amount of training data corresponding to each computational budget.
[0013] In some embodiments, determining a performance curve for a computational budget includes determining the performance curve for the computational budget by interpolating performance metrics of trial allocation tuples corresponding to the computational budget, wherein the trial allocation tuple corresponds to the computational budget if training an experimental machine learning model having an experimental model size defined by the trial allocation tuple on the experimental amount of training data defined by the trial allocation tuple will use an amount of computing resources that meets a threshold defined by the computational budget.
[0014] In some embodiments, using a performance curve to determine an optimal model size and an optimal amount of training data corresponding to each computational budget includes, for each computational budget in a plurality of computational budgets: determining an optimal model size as a model size that optimizes a performance curve corresponding to the computational budget; and determining an optimal amount of training data based on the computational budget and the optimal model size.
[0015] In some embodiments, determining values of a set of allocation mapping parameters based on performance metrics corresponding to multiple trial allocation tuples includes: determining a set of parameters of a performance estimation function, the performance estimation function being configured to process data defining: (i) an input model size, and (ii) an input amount of training data to generate a predictive performance metric characterizing the predictive performance of a machine learning model having the input model size and trained on the input amount of training data on a machine learning task, including: fitting values of the set of parameters of the performance estimation function based on the performance metrics corresponding to multiple trial allocation tuples; and determining values of the set of allocation mapping parameters using the performance estimation function.
[0016] In some embodiments, determining the values of an allocation mapping parameter set using a performance estimation function includes determining the values of the allocation mapping parameter set so that each input computational budget is mapped to a target model size and a target amount of training data that optimizes the performance estimation function subject to the constraint of training a machine learning model with a target model size on the target amount of training data using an amount of computational resources given by the input computational budget.
[0017] In some embodiments, fitting values of a set of parameters of a performance estimation function based on performance metrics corresponding to multiple trial allocation tuples includes: fitting values of the set of parameters of the performance estimation function to minimize, for each trial allocation tuple, an error metric between: (i) the performance metric corresponding to the trial allocation tuple, and (ii) a predicted performance metric generated by using the performance estimation function to process a trial model size defined by the trial allocation tuple and a trial amount of training data.
[0018] In some implementations, the error metric includes Huber loss.
[0019] In some embodiments, for each of a plurality of trial allocation tuples, determining a performance metric corresponding to the trial allocation tuple includes training a trial machine learning model having a trial model size on the trial amount of training data using a learning rate schedule selected based on the trial amount of training data.
[0020] In some implementations, the allocation mapping causes the target model size and the target amount of training data to increase at substantially the same rate in response to an increase in the computational budget.
[0021] In some implementations, the machine learning task includes a language modeling task.
[0022] In some embodiments, the machine learning model comprises a neural network model.
[0023] In some embodiments, the method further includes: receiving a model input to the machine learning model; and processing the model input using the machine learning model to generate a model output based on training values of a set of model parameters of the machine learning model.
[0024] In some embodiments, the machine learning model includes a multimodal model wherein one or both of the model input and the model output include images or audio, and the multimodal model is configured to process the model input comprising at least one of visual tokens representing pixels of still or moving images, data representing audio waveforms, and text tokens representing text sequences to generate a model output comprising text tokens, images, or audio representing the model input.
[0025] In some embodiments, the method is used to adapt a machine learning model to specific computing hardware, wherein the machine learning model includes a neural network model, and the specific computing hardware includes a plurality of neural network accelerators. In this case, the method further includes: determining an energy budget for training the machine learning model, wherein the energy budget defines the total number of floating point operations used to train the machine learning model; determining a computing budget based on the energy budget; determining the hardware specifications of the specific computing hardware on which the machine learning model is to be trained, wherein the hardware specifications define the number of neural network accelerators in the specific computing hardware; determining a target model size for the machine learning model using any of the methods mentioned above, wherein the target model size defines the number of trainable parameters of the machine learning model; determining a target amount of training data for training the machine learning model using any of the methods mentioned above, wherein the target amount of training data defines the number of training tokens used to train the model; and training a machine learning model with a defined number of trainable parameters on the specific computing hardware using the defined number of training tokens.
[0026] In a second aspect, a method performed by one or more computers is described. The method includes: receiving a model input to a machine learning model; and processing the model input using the machine learning model to generate a model output according to training values of a set of model parameters of the machine learning model. In this case, the machine learning model has been generated by operations including: obtaining data defining a computational budget, the computational budget characterizing an amount of computational resources allocated for training the machine learning model to perform a machine learning task; and processing the data defining the computational budget using an allocation mapping according to a set of allocation mapping parameters to generate an allocation tuple, the allocation tuple defining: (i) a target model size for the machine learning model, and (ii) a target amount of training data for training the machine learning model. According to the method, selecting the model size of the machine learning model as the target model size and training the machine learning model on the target amount of training data is predicted to optimize the performance of the machine learning model on the machine learning task subject to the constraint that the amount of computational resources used to train the machine learning model meets a threshold defined by the computational budget. The method further includes: instantiating the machine learning model, wherein the machine learning model has a target model size; obtaining the target amount of training data for training the machine learning model; and training the machine learning model having the target model size on the target amount of training data.
[0027] In a third aspect, a system is described. The system includes one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any of the methods mentioned above.
[0028] In a fourth aspect, a system is described. The system includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of any of the methods mentioned above.
[0029] As mentioned above, the training system can train the machine learning model to perform the machine learning task, such as by implementing any of the methods mentioned above. The machine learning model can be any suitable type of machine learning model. For example, the machine learning model can be a neural network model, a random forest model, a support vector machine model, or any combination thereof, or any other type of machine learning model.
[0030] The machine learning model may have any suitable machine learning model architecture. For example, if the machine learning model is a neural network model, the neural network model may have an attention-based neural network architecture (e.g., a transformer architecture), a convolutional architecture, a fully connected architecture, or any other suitable neural network architecture. Specifically, the neural network model may include any suitable type of neural network layers (e.g., convolutional layers, attention layers, fully connected layers, recurrent layers, etc.) connected in any suitable number (e.g., 10 layers, 100 layers, or 1000 layers) and in any suitable configuration (e.g., as a linear sequence of layers or a directed graph of layers).
[0031] Throughout this specification, the “model size” of a machine learning model may refer to the number of (trainable) parameters, such as weights, biases, matrix elements, etc., required to implement the machine learning model.
[0032] Throughout this specification, a "computational budget" characterizes the amount of computing resources allocated to train a machine learning model to perform a machine learning task. The computational budget can be measured in floating point operations (FLOPs), which is the total number of operations available for training the machine learning model. Note that FLOPs, represented by the lowercase letter "s", should not be confused with the number of floating point operations per second (FLOPS), which is represented by the uppercase letter "S" and can correspond to the rate of operations performed by specific computing hardware. In some cases, FLOPs can be determined by multiplying FLOPS by the total computation time as FLOPs = FLOPS × time.
[0033] In some embodiments, the method is used to adapt a machine learning model to specific computing hardware. In such a case, the machine learning model includes a neural network model, and the specific computing hardware includes a plurality of neural network accelerators. A neural network accelerator is a specialized hardware for accelerating neural network computations, such as a GPU (graphics processing unit) or a TPU (tensor processing unit). Typically, a neural network accelerator is configured to perform hardware matrix multiplication, for example using parallel computing (e.g., it may include a set of one or more multiply-accumulate units (MACs)).
[0034] The method may include determining an energy budget for training a machine learning model, wherein the energy budget defines a total available number of floating point operations for training the machine learning model. The energy budget may be determined based on a target carbon footprint for the training. The computational budget may be determined based on the energy budget. Depending on how the energy budget is expressed, for example, depending on the units in which it is expressed, both the energy budget and the computational budget may be defined as a total available number of floating point operations (FLOPs). Alternatively, the computational budget may be determined from an energy budget expressed in electrical energy based on known (e.g., average) energy usage of the computing hardware. In some embodiments, a majority (e.g., nearly all) of the floating point operations (FLOPs) performed during training are performed by a neural network accelerator, and thus the energy budget may be approximated based on this assumption, for example, by determining the computational budget using the energy consumption of floating point operations on one of the neural network accelerators.
[0035] The method can be used to determine the hardware specifications of a specific computing hardware on which a machine learning model is to be trained - the hardware specifications define the number of neural network accelerators included in the specific computing hardware. The method is then used to determine a target model size for the machine learning model, which target model size defines the number of trainable parameters of the machine learning model. The method is also used to determine a target amount of training data for training the machine learning model, which target amount of training data defines the number of training data items, in particular the number of training tokens used to train the model. As described later, such training tokens represent training data items, such as text tokens representing words or word fragments, or visual tokens representing pixel intensity values of still or moving images, for example, for regions of images. The method trains a machine learning model, i.e., a neural network with a defined number of trainable parameters, on specific computing hardware using a defined number of training tokens. It has been found that many neural network models are too large for a given energy and computational budget and are trained using too few tokens, indicating that some previously held assumptions about machine learning models are incorrect. That is, it has been found that the general trend of increasing model size produces models that actually perform poorly, and surprisingly, it has been found that constraining the model to fit within specific hardware constraints, and applying the techniques described in this paper, can produce better performance than hitherto expected. Some results supporting the described techniques are presented later.
[0036] The machine learning models used for the techniques described herein can be configured to perform any suitable machine learning task.
[0037] In particular, the machine learning model can be configured to process any suitable model input, including, for example, one or more of: images, audio waveforms, point clouds (e.g., generated by a lidar or radar sensor), representations of proteins, representations of molecules, word sequences (e.g., forming one or more sentences or paragraphs), videos (e.g., representing a sequence of video frames), or combinations thereof.
[0038] A machine learning model can be configured to generate any model output that represents a model input. For example, a model output can be a classification output, a regression output, a sequence output (i.e., it includes a sequence of output elements), a segmentation output, or a combination thereof.
[0039] Next, we describe in more detail some examples of machine learning tasks that can be performed by a machine learning model.
[0040] In some implementations, the machine learning model is configured to process a model input representing image pixels to generate a classification output that includes a corresponding score for each object class in a set of possible object classes (e.g., vehicle, pedestrian, bicycle, etc.). The score for an object class can define the likelihood that the image depicts an object belonging to that object class.
[0041] In some embodiments, the machine learning model is configured to process a model input representing audio samples in an audio waveform to perform speech recognition, i.e., generate an output defining a sequence of phonemes, graphemes, characters, or words corresponding to the audio waveform.
[0042] In some embodiments, the machine learning model is configured to process a model input representing words in a word sequence to perform a natural language processing task, such as topic classification or summarization. To perform topic classification, the machine learning model generates an output that includes a corresponding score for each topic category in a set of possible category categories (e.g., sports, business, science, etc.). The score for a topic category can define the likelihood that a word sequence belongs to that topic category. To perform summarization, the machine learning model generates an output that includes an output word sequence that has a shorter length than the input word sequence and that captures important or relevant information from the input word sequence.
[0043] In some embodiments, the machine learning model performs a machine translation task, for example, by processing a model input representing a sequence of text, such as a sequence of words, phrases, characters, or word fragments in one language, to generate an output, which can be a translation of the text sequence into another language, that is, a text sequence in another language that is a translation of the input text sequence. As a specific example, the task can be a multilingual machine translation task, in which the machine learning model is configured to translate between multiple different source language-target language pairs. In this example, the source language text can be enhanced with an identifier that indicates the target language into which the machine learning model should translate the source language text.
[0044] In some embodiments, the machine learning model is configured to perform audio processing tasks. For example, if the model input represents a spoken utterance, the output generated by the machine learning model can be a score for each of a set of text pieces, each score representing an estimated likelihood that the text piece is a correct transcription of the utterance. As another example, if the model input represents a spoken utterance, the output generated by the machine learning model can indicate whether a particular word or phrase ("hot word") is spoken in the utterance. As another example, if the model input represents a spoken utterance, the output generated by the machine learning model can identify the natural language in which the utterance is spoken.
[0045] In some embodiments, the machine learning model is configured to perform natural language processing or understanding tasks, such as entailment tasks, paraphrase tasks, text similarity tasks, sentiment tasks, sentence completion tasks, grammar tasks, etc., which operates on a set of model inputs representing text in some natural language.
[0046] In some embodiments, the machine learning model is configured to perform a text-to-speech task where the model input represents text in a natural language or features of text in a natural language, and the model output is a spectrogram, waveform, or other data defining audio of the text spoken in the natural language.
[0047] In some embodiments, the machine learning model is configured to perform a text generation task where the model input represents a text sequence and the output is another text sequence, e.g., a completion of the input text sequence, a response to a question posed in the input sequence, or a text sequence about a topic specified by the first text sequence. As another example, the model input may represent data other than text, such as an image, and the output sequence may be text describing the data represented by the model input.
[0048] In some implementations, the machine learning model is configured to perform an image generation task where the model input represents a conditional input and the output is an input sequence of intensity values for image pixels.
[0049] In some embodiments, a machine learning model is configured to perform an agent control task, where the model input represents a sequence of one or more observations or other data characterizing an environment state, and the output defines an action to be performed by the agent in response to the latest data in the sequence. The agent can be a mechanical agent that acts in a real environment to perform a task; the observations can include any type of observation, such as image observations; the model output can include control signals to control the agent to perform the task. Optionally, the model input can include other information, such as text tokens that define the task to be performed. The agent can be a real-world or simulated robot, a control system for an industrial facility, or a control system that controls different types of agents.
[0050] In some embodiments, a machine learning model is configured to perform genomics tasks, where the model input represents a fragment of a DNA sequence or other molecular sequence, and the output includes, for example, promoter site prediction, methylation analysis, prediction of functional effects of non-coding variants, etc.
[0051] In some embodiments, a machine learning model is configured to perform a protein modeling task, e.g., where the model input represents a protein and the model output characterizes the protein. For example, the model output can characterize the predicted stability of the protein or the predicted structure of the protein.
[0052] In some embodiments, a machine learning model is configured to perform a point cloud processing task, e.g., where the model input represents a point cloud (e.g., generated by a lidar or radar sensor) and the model output represents, for example, the type of object represented by the point cloud.
[0053] In some embodiments, the machine learning model is configured to perform a language modeling task, for example, by autoregressively generating an output sequence of text data. More specifically, the machine learning model can be configured to generate a sequence of output text tokens (wherein the text tokens can include, for example, characters, word fragments, words, n-grams, etc.). The machine learning model can generate an output sequence of text tokens over a sequence of time steps. At each time step, the machine learning model can generate an output token at a corresponding position in the sequence of output text tokens. The machine learning model can condition the generation of a text token at a certain position in the output sequence based on the text token generated at each of one or more previous positions in the output sequence. For example, in order to generate a text token at a certain position in the output sequence of text tokens, the machine learning model can process data including text tokens generated at one or more previous positions in the output sequence of text tokens to generate a score distribution for a set of possible text tokens. The machine learning model can then select the token at the position using the score distribution, for example, by selecting the token with the highest score under the score distribution.
[0054] Optionally, a machine learning model configured to perform a language modeling task can be conditioned using one or more conditional inputs. For example, a machine learning model can be conditioned on a question, and the machine learning model can autoregressively generate an output sequence of text data that provides an answer to the question. As another example, a machine learning model can be conditioned on a task and a programming language, and the machine learning model can autoregressively generate an output sequence of text data that defines instructions in a programming language to complete the task. As another example, a machine learning model can be conditioned on a set of input instructions, such as text instructions, and the machine learning model can autoregressively generate an output sequence of text data in response to the set of input instructions.
[0055] In some cases, a machine learning model configured to perform a language modeling task can be implemented as a neural network model. The neural network model can include an attention neural network layer, such as a self-attention neural network layer, a cross-attention neural network layer, or both.
[0056] In some embodiments, the machine learning model is configured to perform a combination of multiple separate machine learning tasks, such as two or more of the machine learning tasks mentioned above. For example, the machine learning model can be configured to perform multiple separate natural language understanding tasks, where the model input processed by the machine learning model includes an identifier of the separate natural language understanding task to be performed on the model input.
[0057] As a specific example, a machine learning model may include a multimodal model in which one or both of the model input and the model output include an image or audio. For example, a multimodal machine learning model may be configured to process a model input that includes visual words representing pixels of a still or moving image (e.g., a point cloud image) and / or data representing an audio waveform (e.g., values or features of an audio waveform, such as audio words and / or text words representing a sequence of text) to generate a model output (e.g., a text word representing a still or moving image or audio waveform and / or a sequence of intensity value inputs of image pixels or a sequence of values defining an audio waveform). A visual word may represent multiple pixels in an image region, for example as a feature of the region. Such a multimodal model may perform any of the previously described tasks using multimodal inputs, or by providing multimodal outputs, or by converting between different input and output modes (e.g., text / image / audio). For example, a multimodal model may generate text that represents, describes (e.g., captions), or otherwise characterizes an image or audio input, for example, by answering questions related to the image or audio input, such as physical predictions of the state of an object represented by the image or audio. As another example, a multimodal model may generate an image or audio represented, described, or otherwise characterized by a text input, or otherwise respond to a text input, for example, representing an image or audio answer to a text question.
[0058] Throughout this specification, the term "optimization" may refer to predictive optimization or approximate optimization, ie, rather than exact optimization.
[0059] Throughout this specification, a "performance metric" of a machine learning model on a machine learning task may refer to a measure of how effectively the machine learning model performs the machine learning task. For example, the system described in this specification may measure the performance of a machine learning model using a loss or objective function, for example, which characterizes the predictive accuracy of the machine learning model. That is, the performance metric may be expressed as the value of the loss or objective function used to train the machine learning model. The system may measure the performance of the machine learning model, for example, on a training data set used to train the machine learning model, or on a validation data set retained from the training of the machine learning model, i.e., without the machine learning model being trained on the validation data set. Examples of loss / objective functions may include, for example, a cross entropy objective function, a squared error objective function, and the like.
[0060] For example, determining the optimal model size and the optimal amount of training data for a given computational budget can involve determining an optimization for each of the optimized performance metrics, e.g., optimizing the value of an objective function for training a machine learning model or minimizing the value of a loss function for training a machine learning model.
[0061] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.
[0062] The training system described in this specification trains a machine learning model to perform a machine learning task, subject to a computational budget that allocates a limited amount of computing resources (e.g., FLOPs) for training the machine learning model. Training a machine learning model subject to a computational budget involves a trade-off between: (i) the model size of the machine learning model, and (ii) the amount of training data used to train the machine learning model. For example, increasing the model size of the machine learning model may require reducing the amount of training data available for training the machine learning model, i.e., ensuring that the computing resources consumed during training do not exceed the computational budget. Specifically, the amount of training data used to train a machine learning model generally refers to the amount of training data seen by the machine learning model during training, and not necessarily the total amount of training data available to the training system. For example, if the amount of training data available for a particular machine learning model and / or a particular machine learning task is limited, the training system may sample multiple times from the available training data (if necessary). In this case, the amount of training data used to train the machine learning model may include multiple instances of the same data (e.g., word units).
[0063] The training system uses an allocation map to determine a tradeoff between a model size of a machine learning model and an amount of training data used to train the machine learning model. Specifically, the allocation map processes data defining a computational budget to generate data defining a target model size and a target amount of training data that are predicted to optimize the performance of the machine learning model subject to constraints defined by the computational budget. That is, the training system uses an allocation map to determine an allocation of computational resources between model size and training data in order to optimize the performance (e.g., prediction accuracy) of the machine learning model.
[0064] The experimental system can determine mapping parameters for the allocation mapping by training a series of experimental machine learning models, varying the model size, the amount of training data used for training, and the available computing resources specified by the computing budget to determine empirical data for various experimental machine learning models. The optimization system can then fit the parameters of the allocation mapping using the resulting data that characterizes the performance of the machine learning model across model size, amount of training data, and computing budget. For example, the optimization system can interpolate (and extrapolate) data to different computing budgets while determining the optimal model size and the optimal amount of training data for the computing budget. After determining the parameters for the allocation mapping, the training system can then use the allocation mapping to determine the optimal tradeoff between model size and amount of training data each time the training system trains the machine learning model to perform a machine learning task.
[0065] By optimally allocating computing resources between model size and amount of training data, the training system can significantly improve the efficiency of the use of computing resources during training. For example, the training system can reduce the likelihood of wasting computing resources by overtraining a machine learning model, i.e., by training the machine learning model on a training data set beyond the point where no further gains in machine learning model performance can be achieved. As another example, the training system can select a significantly smaller model size for the machine learning model than would otherwise be selected (for the same training data set), thereby reducing the use of computing resources during fine-tuning and downstream use of the trained machine learning model.
[0066] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 is a block diagram of an example training system that can train a machine learning model with a target model size on a target amount of training data to perform a machine learning task.
[0068] Figure 2 is a flowchart of an example process for training a machine learning model with a target model size on a target amount of training data to perform a machine learning task.
[0069] Figure 3 is a block diagram of an example experimental system that can determine values for a set of allocation mapping parameters based on a performance metric of an experimental machine learning model.
[0070] Figure 4 is a flow chart of an example process for determining values for a set of allocation mapping parameters based on a performance metric of a trial machine learning model.
[0071] Figure 5 are block diagrams of two example optimization systems that may determine values for a set of allocation mapping parameters based on a performance curve.
[0072] Figure 6 is a flow chart of an example process for determining values for a set of allocation mapping parameters based on an optimal model size and an optimal amount of training data for a given computational budget.
[0073] Fig. 7A is a flow chart of an example process for determining an optimal model size and an optimal amount of training data for a given computational budget based on performance curves.
[0074] Figure 7BAn example of generating a set of allocation mapping parameters using a performance curve that defines a continuous mapping from possible computational budgets to predicted performance metrics is shown.
[0075] Fig. 8A is a flow chart of another example process for determining an optimal model size and an optimal amount of training data for a given computational budget based on performance curves.
[0076] Figure 8B An example of generating a set of allocation mapping parameters using a corresponding performance curve for each of a plurality of possible computational budgets is shown.
[0077] Fig. 9A and 9B is a block diagram of another example optimization system that can use a performance estimation function to determine values for a set of allocation mapping parameters.
[0078] Fig.10 is a flow chart of an example process for determining values for a set of allocation mapping parameters using a performance estimation function.
[0079] Fig.11A and 11B Examples of experimental results comparing the performance of: (i) a “computationally optimal” machine learning model generated by the training system described in this specification, and (ii) an alternative machine learning model are shown.
[0080] Like reference numbers and names in the various drawings indicate like elements. DETAILED DESCRIPTION
[0081] Large machine learning models, such as large language models (e.g., machine learning models including neural networks that perform the above-mentioned language modeling tasks), deep learning models, generative models, discriminative and classification models, regression models, etc., have been implemented with a large number of parameters, for example, more than 10 billion parameters, or more than 50 billion parameters, or more than 100 billion parameters, or more than 250 billion parameters, or more than 500 billion parameters. Large language models (LLMs) have shown outstanding performance on many machine learning tasks (e.g., language modeling tasks) using various training and evaluation protocols including zero-trial, few-trial, and fine-tuning.
[0082] However, the computational and energy costs of training large machine learning models (e.g., LLMs) are significant and increase with model size. In practice, the allocated training compute (i.e., the compute budget) may be known in advance, for example, how many accelerators (e.g., high-performance computing units) are available and for how long the accelerators are available. In some cases, a machine learning model may only be trained once (or a few times), so accurately estimating the optimal model hyperparameters for a given compute budget may be very valuable. For example, reducing the model size of a machine learning model can greatly reduce the cost of inference and facilitate downstream implementation in resource-constrained environments. The energy cost of a large machine learning model is amortized through its use for inference and fine-tuning. Therefore, the benefits of a smaller model that is more optimally trained go beyond the immediate benefit of its improved performance.
[0083] In this regard, the training system described herein can predict a target model size and a target amount of training data in a manner that is predicted to (approximately) optimize the performance of the machine learning model for a given computational budget - i.e., such that training the machine learning model is computationally optimal. In some cases, the training system can determine that training a computationally optimal machine learning model on a given computational budget may require a substantial increase in the amount of training data, e.g., rather than an increase in model size. For example, for some computationally optimal machine learning models (e.g., LLM), the training system can determine that the model size and the training data size scale in (approximately) equal proportions to the computational budget.
[0084] In addition, as described herein, a large machine learning model may not need to be trained to its lowest possible loss to achieve computational optimality. That is, the described techniques describe how to optimize the loss for a given computational budget while taking into account that the machine learning model may not have been trained to convergence. For example, as described later, some embodiments of the system use a performance estimation function that takes this into account, e.g., a function that includes a term representing the residual portion of the loss due to the machine learning model not being trained to convergence.
[0085] For reference, some LLMs include transformer neural networks, i.e., neural network models with transformer architecture. In general, a transformer neural network can be a neural network model characterized by having a series of self-attention neural network layers. The self-attention neural network layer has an attention layer input for each element of the input, and is configured to apply an attention mechanism on the attention layer input to generate an attention layer output for each element of the input. Many different attention mechanisms can be used. Some of these LLMs may use a transformer neural network as an encoder, some may use a transformer neural network as a decoder, and some may use one transformer neural network as an encoder and another transformer neural network as a decoder coupled to the encoder. Just as an example, some LLMs are decoder-only models.
[0086] These and other features are described in greater detail below.
[0087] Figure 1 An example training system 100 is shown that can train a machine learning model 102 having a target model size 132 on a target amount of training data 134 to perform a machine learning task 104. Training system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.
[0088] In general, for any particular machine learning model 102 configured to perform any particular machine learning task 104, the training system 100 is able to select a target model size that is predicted to be computationally optimal. 132 and the target amount of training data 134. In other words, target sizes 132 and 134 are predicted to optimize the (prediction) performance of model 102 on task 104 subject to the following constraints: the amount of computational resources used for training ( ) meets the calculation budget 112 defines a threshold value, for example, such that = , or make ≤ . The computational budget 112 defines the amount of computational resources allocated to training. For example, the allocated computational resources may be fixed due to the available computing architecture (e.g., the number of accelerators, servers, GPU clusters, supercomputers, combinations thereof, etc.) and may not (or should not) be exceeded. Alternatively or additionally, the amount of allocated resources can be fixed to limit the energy consumption associated with training the machine learning model 102, for example, to reduce environmental impact, to allow multiple machine learning models to be trained in parallel, etc. In any case, the training system 100 can reduce the amount of computational and energy resources consumed by training the machine learning model 102, while enabling the machine learning model to achieve acceptable performance on the machine learning task 104.
[0089] For reference, the model size The amount of training data may refer to the number of parameters that may be employed by the machine learning model 102, for example, the number of parameters required to implement the machine learning model 102. Or the training data size can refer to the specific size of the specific training data set 144 that can be used to train the machine learning model 102. For example, the training data size can refer to the number of tokens included in the training data set 144. More specifically, the amount of training data used to train the machine learning model 102 Refers to the amount of training data seen by the machine learning model 102 during training. Therefore, if the total training data available to the training system 100 is limited, the training data set 144 may include multiple instances of the same word. As mentioned above, the computational budget 112 can refer to the amount of computational resources allocated to training the machine learning model 102, and can be measured in terms of the total number of floating point operations (FLOPs). In some cases, the computational budget 112 can also be measured in terms of the total number of instructions, total computation time, memory space, or a combination thereof (e.g., as a weighted sum). The amount of computational resources used during training F (also referred to as total computation) can be measured in the same units as the computational budget 112.
[0090] To determine the target sizes 132 and 134 for the machine learning model 102, the training system 100 first obtains (e.g., receives) data defining the computational budget 112. For example, the data may be provided to the training system 100 by a user or an automated process in an effort to seek to perform a computationally optimal training scheme for the machine learning model 102 under the computational budget 112. For ease of description, the data defining the computational budget 102 may be described as being provided by a server 110, such as a cloud server, a local server, or a remote server.
[0091] Training system 110 uses allocation mapping 120 Processing of data defining the computational budget 112 to generate allocation tuples 130. Allocation tuple 130 is used to define the target model size 132 and target data size 134. Typically, the allocation mapping 120 is a set of allocation mapping parameters 126 parameterized function. Mapping parameters 126 specify how allocation mapping 120 determines the computational budget In possible model size and the possible data size D. As mentioned above, computing the optimal allocation corresponds to choosing a target size and As model and data size:
[0092] For the sake of clarity, and is a subset of mapping parameter set 126, which specifies the allocation mapping 120 How to calculate the budget Continuous mapping to target model size and target data size . Subset and Common parameters may be shared, and not necessarily have the same number of parameters. In general, the allocation map 120 may take any functional form based on a particular set of mapping parameters 126. Some examples are described below.
[0093] In some implementations, the allocation map 120 may be represented as a linear function such that the mapping parameters 126 are the slope and intercept, for example:
[0094] In some implementations, the allocation map 120 may be represented as a power law, such that the mapping parameters 126 are coefficients and exponents, such as:
[0095] In this case, when the machine learning system 102 is an LLM, the training system 100 may determine, in some cases, Characterizes computationally optimal scaling of model size and data size with respect to computational budget. That is, in these cases, target model size 132 and target data size 134 should scale with computational budget 112 in substantially equal proportions.
[0096] In some implementations, the allocation map 120 may be represented as a certain order A polynomial or Taylor series such that the mapping parameter 126 is the coefficient of the polynomial, for example:
[0097] More generally, in some embodiments, the allocation map 120 can be represented as a function basis (e.g., with an order of ) such that the mapping parameters 126 are coefficients of the basis functions, for example:
[0098] Basis Functions It may be a polynomial basis function, a Lagrangian basis function, a B-spline basis function, a Fourier basis function, an exponential basis function, or any suitable set of basis functions of a desired order. In some cases, the basis function itself may also depend on the allocation mapping parameters 126 .
[0099] The values of mapping parameters 126 determine the precise functional dependency of allocation mapping 120 on computational budget 112. Specifically, training system 100 uses values such that Equal to the calculation budget Under the constraint of and Optimizing the performance of the machine learning model 102 on the machine learning task 104 .in other words:
[0100] The above equation shows that, with the allocation tuple 130 Associated machine learning models 102 in training During this period, all computational budget 112 is consumed, while optimizing its post-training machine learning task 104. For reference, the computation function Indicates a specific amount training data with a specific model size The total computation used by the machine learning model 102. Performance function Indicates the specific size of a given model 102 and In the case of , the performance measure (e.g., pre-training loss) of the machine learning model 102 on the machine learning task 104. Note that the calculation function and performance functions The exact functional dependencies of are often unknown a priori, as they depend on the size of the specific machine learning model 102 and , which characterizes its overall architecture (e.g., “global” properties). Therefore, determining an appropriate allocation mapping 120 that satisfies the above constraints is a challenging problem. Figure 3-10Various systems and methods for determining (eg, empirically estimating) the allocation map 120 are described in detail.
[0101] After generating the allocation tuple 130, the training system 100 instantiates 142 the machine learning model 102 using the target model size 132. The training system 100 then trains the machine learning model 102 on a training data set 144 having training data of the target amount 134. For example, the training system 100 may obtain the training data set 144 from the server 110 or other device. As mentioned above, given the allocation tuple, the training system 100 may instantiate 142 the machine learning model 102 using the target model size 132. The training may be computationally optimal given the target model 132 and target data 134 size defined by the computational budget 112. In other words, the training consumes the allocated computational resources defined by the computational budget 112, and the performance of the machine learning model 102 may be optimized for the machine learning task 104 given the computational budget 112.
[0102] After training, the machine learning model 102 can be deployed to perform machine learning tasks 104. For example, the machine learning model 102 can be deployed in an environment that enables a user to provide a request for the machine learning model 102 to process a specified model input to generate a corresponding model output. The user can provide the request, for example, with the help of a user interface or through an application programming interface (API). The request can be transmitted from the user device (e.g., through a data communication network, such as the Internet) to one or more computers that implement the machine learning model 102, for example in a data center. The machine learning model 102 can process the model input specified by the user request to generate a corresponding model output, and then transmit the model output to the user device (e.g., through a data communication network).
[0103] Figure 2 is a flow chart of an example process for training a machine learning model with a target model size on a target amount of training data to perform a machine learning task. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to the present specification, such as Figure 1 The training system 100 can perform process 200.
[0104] The training system obtains data defining a computing budget, the computing budget characterizing an amount of computing resources allocated for training the machine learning model to perform the machine learning task (210). The training system can obtain the data defining the computing budget, such as from a user through a user interface or an application programming interface (API), or from an external resource management system, such as a system that manages computing resources in one or more data centers.
[0105] The training system processes the data defining the computational budget using an allocation map according to the set of allocation map parameters to generate an allocation tuple that defines: (i) a target model size for the machine learning model, and (ii) a target amount of training data for training the machine learning model (220). The training system generates the allocation tuple such that selecting the model size of the machine learning model as the target model size and training the machine learning model on the target amount of training data is predicted to optimize the performance of the machine learning model on the machine learning task subject to the constraint that the amount of computational resources used to train the machine learning model meets a threshold defined by the computational budget.
[0106] The training system instantiates a machine learning model, wherein the machine learning model has a target model size (230). For example, the training system can generate an instance of the machine learning model, including determining an architecture of the machine learning model and initializing values of a set of model parameters of the machine learning model. The training system can determine the architecture of the machine learning model, for example, by mapping a target model size of the machine learning model to a corresponding machine learning model architecture (e.g., according to a predefined architecture mapping). The architecture of the machine learning model can be defined, for example, by a set of architecture hyperparameters, and the system can generate a value for each architecture hyperparameter based on the target model size. For example, in an embodiment where the machine learning model is implemented as a neural network, the set of architecture hyperparameters can include hyperparameters that specify the number of layers in the neural network, the configuration of each layer in the neural network, and a directed graph that defines connectivity between neural network layers. The training system can initialize the values of the set of model parameters of the machine learning model using any appropriate initialization technique, such as random initialization or Glorot initialization.
[0107] The training system obtains a target amount of training data for training the machine learning model (240). For example, to obtain the target amount of training data, the training system can access one or more data storage devices storing a corpus of training data. The system can identify a subset of the corpus of training data that includes the target amount of training data, for example, by randomly sampling training data from the corpus of training data, and then retrieve the selected training data for training the machine learning model.
[0108] The training data for training the machine learning model can be generated in any of a variety of possible ways. For example, the training data can include text sequences, such as text sequences captured (e.g., extracted using systems and automated techniques) from one or more data sources such as one or more databases or the Internet. The training system can use the text sequences to train the machine learning model to perform a language modeling task, as will be described in more detail below. As another example, the training data can include a set of training examples, each of which includes: (i) a model input to the machine learning model (e.g., an image), and (ii) a target output (e.g., an image label), that is, what should be generated by the machine learning model by processing the model input. The target output can be generated, for example, by manual annotation or in any other appropriate manner.
[0109] The training system trains a machine learning model with a target model size on a target amount of training data (250). The training system can use any suitable machine learning training technique to train the machine learning model on the training data. Some example techniques for training a machine learning model on a set of training data are described below.
[0110] In some embodiments, the machine learning model is a neural network model, the training data set includes a text sequence set, and the training system trains the neural network to perform a language modeling task. In these embodiments, for each text sequence, the training system can use a neural network to process (at least a portion of) the text sequence to generate a score distribution over a set of possible word units (e.g., text word units including characters, word fragments, words, n-grams, etc.) for each of one or more positions in the text sequence. The neural network can be configured to generate a score distribution for a position in the text sequence by processing word units from previous positions in the text sequence, but not based on word units at the position or word units at subsequent positions in the text sequence. The training system can train the neural network based on an objective function that measures the error (e.g., cross entropy error) between the following two for each of the one or more positions in the text sequence: (i) the word unit at the position in the text sequence, and (ii) the score distribution over the set of possible word units generated by the neural network for the position. Training a neural network based on an objective function may include, for example, determining the gradient of the objective function with respect to neural network parameters (e.g., using backpropagation), and adjusting values of the neural network parameters using the gradients (e.g., using an update rule such as a suitable gradient descent optimization technique such as RMSprop or Adam).
[0111] In some embodiments, the training system trains the machine learning model to perform a supervised machine learning task. For example, the training system may train the machine learning model on a set of training examples, each training example comprising: (i) a model input, and (ii) a target output. Training the machine learning model on the training examples may include training the machine learning model to process the model input of the training examples to generate a predicted output that matches the target output of the training examples. Specifically, the training system may train the machine learning model to optimize an objective function that measures, for each training example, the error (e.g., a cross entropy error or a squared error) between: (i) the target output of the training example, and (ii) the predicted output generated by the machine learning model for the training example.
[0112] Figure 3 An example testing system 300 is shown that can determine values for the allocation mapping parameter set 126 based on a performance metric 350 of a testing machine learning model 302. The testing system 300 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.
[0113] The experimental system 300, in combination with the optimization system 500, can determine the values of the allocation mapping 120 and its mapping parameters 126. That is, given a particular machine learning model 102 and a particular machine learning task 104, the experimental system 300 can determine a corresponding allocation mapping 120 that provides computationally optimal training of the model 102 for the task 104. The experimental system 300 can do this by empirically evaluating the performance of multiple experimental machine learning models 302 with different experimental model 332 and experimental data 334 sizes. The optimization system 500 can then interpolate (and / or extrapolate) the performance of the experimental sizes 332 / 334 to different possible sizes to determine the optimal size. Based on these results, the optimization system 500 can determine the values of the mapping parameters 126. About Figure 5-10 Three variations of the optimization system 500 are described that utilize new methods of specifying values for the mapping parameters 126 .
[0114] The testing system 300 may begin by identifying a plurality of testing allocation tuples 330. Each testing allocation tuple 330.ij are all 2-tuples, which define the experimental model size of the machine learning model 102 332.i and the experimental amount of training data used to train the machine learning model 102 334.j. The test system 300 can obtain the test allocation tuple 330 in various ways. For example, the test system 300 can obtain the test allocation tuple 330 from the joint probability distribution Medium test model size and the test data size Random sampling is performed, or they are sampled separately and trial allocation tuples 330.ij are generated from various trial size pairs 332.i / 334.j. In other cases, the trial allocation tuples 330 can be specified by the user. In addition, the trial system 300 can select the range and granularity of the trial size based on the desired accuracy level of the resulting mapping parameters 126. A larger range and more granularity can provide higher accuracy. For example, the trial system 300 can use more than four hundred trial allocation tuples 330, where the trial model size 332 ranges from 70M to 16B parameters, and the trial data size 334 ranges from 5B to more than 400B words. Please note that a single trial model size 332.i can be associated with multiple different trial data sizes 334.j (and vice versa). This allows the trial system 300 to determine the performance of a trial machine learning model 302.ij with a specific trial model size 332.i on multiple training sets 344.j of different sizes. Similarly, a single test model size 332.i is not necessarily associated with each test data size 334.j (and vice versa). Thus, depending on the implementation, the test system 300 may or may not use and every combination of .
[0115] For each trial allocation tuple 330.ij, the trial system 300 instantiates 142 a trial machine learning model 302.ij with a corresponding trial model size 332.i. The trial system 300 then trains the trial machine learning model 302.ij on a training data set 344.j with a trial amount 334.j of corresponding training data. As mentioned above, the trial system 300 may obtain the training data 344.j from the server 110 or other device. The trial system 300 may also determine the total computational overhead for each trial machine learning model 302.ij. , which characterizes the amount of computational resources used during the training of the experimental machine learning model 302.ij. Therefore, each experiment is assigned a tuple 330.ij provides calculation functions In some embodiments, the experimental system 300 trains the experimental machine learning model 302.ij using a learning rate corresponding to its experimental data size 334.j. For example, the experimental system 300 can decay (reduce) the learning rate for larger experimental data sizes 334.j.
[0116] The test system 300 determines the corresponding performance metric = 350.ij is used to determine the performance of each experimental machine learning model 302.ij on the machine learning task 104. Therefore, each experimental allocation tuple 330.ij also provides performance functions The test system 300 then processes the performance metrics using the optimization system 500 To determine the value of the allocation mapping parameter 126. As mentioned above, for Figure 5 to 1 1 depicts three variations of the optimization system 500, each of which may process the performance metric 350 in a different manner to determine the value of the mapping parameter 126.
[0117] Figure 4 is a flow chart of an example process 400 for determining values for a set of allocation mapping parameters based on a performance metric of a test machine learning model. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a test system appropriately programmed according to the present specification—e.g., Figure 3 The test system 300 in the embodiment of the present invention can perform the process 400.
[0118] The experiment system identifies a plurality of experiment allocation tuples, wherein each experiment allocation tuple defines: (i) an experiment model size for a machine learning model, and (ii) an experiment amount of training data for training the machine learning model (410).
[0119] The experimental system determines a performance metric for each of the plurality of experimental allocation tuples, the performance metric characterizing the performance of the experimental machine learning model on the machine learning task, the performance metric being a result of selecting a model size of the experimental machine learning model as the experimental model size and training the experimental machine learning model on the experimental amount of training data (420).
[0120] The testing system determines values for a set of allocation mapping parameters based on performance metrics corresponding to the plurality of trial allocation tuples (430).
[0121] Figure 5 Two example optimization systems 500-1 / 500-2 are shown that can determine values for allocation mapping parameter sets 126 based on performance curves 520. Optimization systems 500-1 / 500-2 are examples of systems implemented as computer programs on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.
[0122] Both the first optimization system 500-1 and the second optimization system 500-2 determine the values of the mapping parameters 126 by first determining the corresponding optimal model size 532 and the optimal amount of training data 534 for a given amount of computational budget 312. The optimal size 532 / 534 is computationally optimal for its corresponding computational budget 312. The optimization system 500-1 / 500-2 then interpolates (and / or extrapolates) these data points to fit the mapping parameters 126 of the allocation map 120, which establishes a continuous mapping from the computational budget 120 to the allocation tuple 130. However, the two optimization systems 500-1 / 500-2 may differ in the way they determine the optimal size 532 / 534 themselves. The first optimization system 500-1 fixes the trial model size 332 and generates the curve by varying the trial data size 334. In contrast, the second optimization system 500-2 varies the trial model size 332 and generates the curve while fixing the total computation to the computational budget 312 (i.e., the "equal computational budget" curve). The first optimization system 500-1 and the second optimization system 500-2 may work separately or in conjunction to determine the values of the mapping parameters 126. For example, the results of the two optimization systems 500-1 / 500-2 may be averaged, used for different types of machine learning models 102, used for different ranges of trial sizes, etc. The details of the first optimization system 500-1 are summarized below, followed by the second optimization system 500-2.
[0123] First Optimization System (FOS)
[0124] FOS 500-1 determines a corresponding performance curve 522.i for each test model size 332.i. Test Model Size Performance curve Defines the possible computational budget To predict performance measures In this case, the prediction performance measure is the one that is used when the computational budget is equal to the possible Total calculation Training with experimental model sizes The prediction performance of the experimental machine learning model 302. Similarly, the constraint Defines the size of the given test model From the possible computational budget The possible amount of training data (and vice versa) of the curve equation.
[0125] FOS 500-1 can be interpolated with the test model size The corresponding performance metric for the trial allocation tuple 330.ij to determine the performance curve 522.i of the test model size 332.i. In other words, FOS 500-1 measures the performance Size of the test model The associated test data size Interpolation. FOS 500-1 may interpolate performance metric 350 using a variety of different curve fitting techniques such as power law fitting, linear regression, polynomial regression, polynomial interpolation, and the like.
[0126] FOS 500-1 then calculates the budget for each given 312.kDetermine the optimal model size 532.k and the optimal amount of training data 534.k. To this end, FOS 500-1 calculates a budget for each given 312.k Determine the optimal performance curve The optimal performance curve achieves the optimal performance metric for a given computational budget 312.k. That is, when When evaluated at , it reaches the minimum value among all performance curves 522:
[0127] FOS 500-1 then select the associated test model size As the optimal model size 532.k for a given computational budget 312.k. FOS 500-1 may then use, for example, the constraint The optimal data size 534.k is determined based on the optimal model size 532.k and the corresponding computational budget 312.k. Typically, can be any suitable function that characterizes the model size , training data volume and the computation required to train a machine learning model with model size 102 on the amount of training data For example, in some embodiments, the function is assumed or approximated as ,in is a constant, such as In other embodiments, the test 300 and / or optimization 500 system may, for example, use interpolation and other data fitting techniques described herein based on the total computation expended during training of the test machine learning model 302. Determined empirically .
[0128] FOS 500-1 then uses the optimal model size 532, the optimal data size 534, and the given computational budget 312 to fit the values of mapping parameters 126, e.g., to , and The triplet of and For example, FOS 500 - 1 may fit the values of mapping parameters 126 using any of the curve fitting techniques described herein.
[0129] Second Optimization System (SOS)
[0130] SOS 500-2 determines a corresponding performance curve 524.k for each given computational budget 312.k. Performance curve Defines the possible model sizes from To predict performance measures In this case, the prediction performance measure is the one that is used when the computational budget is equal to the given Total calculation When training, the possible model size The prediction performance of the experimental machine learning model 302. Similarly, the constraint Defines a given computational budget From the possible model size The possible amount of training data (and vice versa). Thus, performance curve 524 corresponds to an "equal computational budget" curve, since the corresponding computational budget 312.k of each curve 524.k is fixed.
[0131] SOS 500-2 can calculate the budget by interpolation and The corresponding performance metric for the trial allocation tuple 330.ij to determine the performance curve 524.k for a given computational budget 312.k. In other words, SOS 500-2 is proportional to the experimental model size Interpolation Performance Metrics , and select the test data size , so that the total calculation is fixed to the calculation budget The SOS 500 - 2 may interpolate the performance metric 350 using a variety of different curve fitting techniques such as power law fitting, linear regression, polynomial regression, polynomial interpolation, and the like.
[0132] Then, SOS 500-2 calculates the budget for each 312.kDetermine the optimal model size 532.k and the optimal amount of training data 534.k. To this end, SOS 500-2 selects the optimal model size 532.k as the model size that optimizes the corresponding performance curve 524.k for a given computational budget 312.k, such that corresponds to the minimum value.
[0133] Then, SOS 500-2 can use the constraint The optimal data size 534.k is determined based on the optimal model size 532.k and the corresponding computational budget 312.k. As mentioned above with respect to FOS 500-1, SOS 500-2 may assume The functional form of is determined empirically.
[0134] Then, SOS 500-2 uses the optimal model size 532, the optimal data size 534, and the given computational budget 312 to fit the values of mapping parameters 126, for example, to , and The triplet of and For example, SOS 500-2 may fit the values of mapping parameters 126 using any curve fitting technique described herein.
[0135] Figure 6 6 is a flow chart of an example process 600 for determining values for a set of allocation mapping parameters based on an optimal model size and an optimal amount of training data for a given computational budget. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, an optimization system appropriately programmed according to the present specification, such as Figure 5 The optimization systems 500 - 1 and 500 - 2 in FIG. 5 may perform process 600 .
[0136] The optimization system determines, for each of a plurality of computational budgets, an optimal model size and an optimal amount of training data corresponding to the computational budget based on the performance metrics corresponding to the plurality of trial allocation tuples ( 610 ).
[0137] The optimization system determines values for a set of allocation mapping parameters based on an optimal model size and an optimal amount of training data corresponding to each of the plurality of computational budgets ( 620 ).
[0138] In some embodiments, step 620 is completed by step 622, which is performed as follows:
[0139] The optimization system fits values of a set of allocation mapping parameters based on an optimal model size and an optimal amount of training data corresponding to each of a plurality of computational budgets ( 622 ).
[0140] Fig. 7A is a flow chart of an example process 700 for determining an optimal model size and an optimal amount of training data for a given computational budget based on a performance curve. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, an optimization system appropriately programmed according to the present specification, such as Figure 6 The first optimization system 500-1 in can perform process 700.
[0141] The optimization system determines a corresponding performance curve for each of a plurality of trial model sizes based on performance metrics corresponding to the plurality of trial allocation tuples (710). The performance curves for the trial model sizes define a continuous mapping from possible computational budgets to predictive performance metrics, wherein the predictive performance metrics corresponding to the possible computational budgets define predictive performance of a trial machine learning model of the trial model size that is trained using an amount of computational resources that satisfies a threshold defined by the possible computational budgets.
[0142] In some embodiments, step 710 is completed by step 712, which is performed as follows:
[0143] The optimization system determines performance curves for the trial model sizes by interpolating performance metrics for the trial allocation tuples corresponding to the trial model sizes ( 712 ).
[0144] The optimization system uses the performance curves to determine an optimal model size and an optimal amount of training data corresponding to each computational budget (720).
[0145] In some implementations, step 720 is accomplished by steps 722-726, which are as follows: For each of the plurality of computational budgets:
[0146] The optimization system determines an optimal performance curve from among the performance curves that achieves an optimal performance metric for the computational budget ( 722 ).
[0147] The optimization system determines the optimal model size as the trial model size corresponding to the optimal performance curve ( 724 ).
[0148] The optimization system determines an optimal amount of training data based on the computational budget and the optimal model size (726).
[0149] Figure 7BAn example of generating a set of allocation mapping parameters using a performance curve that defines a continuous mapping from possible computational budgets to a predicted performance metric is shown. Specifically, Figure 728 shows an example of a performance curve that maps possible computational budgets to a predicted performance metric that the system generated by training a range of experimental model sizes from 75 million to 10 billion parameters. In Figure 728, the horizontal axis represents the possible computational budgets and the vertical axis represents the predicted performance metric, in which case the predicted performance metric is characterized as training loss, for example, such that a lower training loss indicates better performance. The system determines an optimal performance curve, for example, by determining, for each computational budget, a performance curve that represents the optimal performance metric for that computational budget (in this case, the lowest value of the computational budget). The system then uses the optimal performance curve to generate allocation mapping parameters that define a mapping from possible computational budgets to target model sizes (represented by a line in Figure 730) and a mapping from possible computational budgets to a target amount of training data (represented by a line in Figure 732). Specifically, the data points in Figure 730 are compared to the values in Figure 732. contrast The pairs of phases are used to fit the Similarly, the data points in Figure 732 are contrast The pairs of phases are used to fit the This fit then determines the appropriate distribution map = .
[0150] Fig. 8A 8 is a flow chart of another example process 800 for determining an optimal model size and an optimal amount of training data for a given computational budget based on a performance curve. For convenience, process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, an optimization system appropriately programmed according to the present specification, such as Figure 6 The second optimization system 500 - 2 may perform process 800 .
[0151] The optimization system determines a corresponding performance curve for each of a plurality of computational budgets based on performance metrics corresponding to the plurality of trial allocation tuples (810). The performance curve for the computational budget defines a continuous mapping from possible model sizes to predictive performance metrics, wherein the predictive performance metrics corresponding to the possible model sizes define predictive performance of trial machine learning models having the possible model sizes trained using an amount of computational resources that satisfies a threshold defined by the computational budget.
[0152] In some embodiments, step 810 is completed by step 812, which is performed as follows. The optimization system determines a performance curve for the computational budget by interpolating the performance metrics of the trial allocation tuples corresponding to the computational budget, wherein the trial allocation tuple corresponds to the computational budget if training the trial machine learning model on the trial amount of training data defined by the trial allocation tuple using the trial model size defined by the trial allocation tuple will use an amount of computing resources that meets a threshold defined by the computational budget.
[0153] The optimization system uses the performance curves to determine an optimal model size and an optimal amount of training data corresponding to each computational budget (820).
[0154] In some implementations, step 820 is accomplished by steps 822 and 824, which are performed as follows: for each of the plurality of computational budgets:
[0155] The optimization system determines an optimal model size as a model size that optimizes a performance curve corresponding to the computational budget (822).
[0156] The optimization system determines an optimal amount of training data based on the computational budget and the optimal model size (824).
[0157] Figure 8B An example of generating a set of allocation mapping parameters using a corresponding performance curve for each of a plurality of possible computational budgets is shown. The performance curve for a computational budget defines a continuous mapping from possible model sizes to a prediction performance metric, where the amount of training data used during training is selected so that the total amount of computation used during training matches the computational budget. In this case, the computational budget is arrive FLOPs. Specifically, in graph 826, the horizontal axis represents possible model sizes and the vertical axis represents a predicted performance metric, in this case, characterized by training loss (e.g., such that lower training loss indicates better performance). The system then uses the performance curve to generate allocation mapping parameters that define a mapping from possible computational budgets to target model sizes (represented as a line in graph 828) and a mapping from possible computational budgets to a target amount of training data (represented as a line in graph 830). Specifically, the data points in graph 828 are similar to contrast The pairs of phases are used to fit the line represented by the graph 828. Similarly, the data points in graph 830 are used to fit the line represented by graph 830. of contrast The fit then determines the appropriate distribution mapping .
[0158] Fig. 9A and 9B An example optimization system 500-3 is shown that can determine values for allocation mapping parameter set 126 using performance estimation function 540. Optimization system 500-3 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.
[0159] Third Optimization System (TOS)
[0160] TOS 500-3 uses a different approach than FOS 500-1 and SOS 500-2. TOS 500-3 does not generate performance curves, but directly uses the performance estimation function 540 to estimate the performance function The performance estimation function 540 is configured to process the input model size defined and the amount of training data input to generate a predictive performance metric. Given a machine learning model 102 with an input model size And in the input training data The prediction performance metric characterizes the prediction performance of the machine learning model 102 on the machine learning task 104. Similar to the mapping parameters 126 of the allocation mapping 120, the performance estimation function 540 is composed of the parameter set 542 parameterization, these parameters specify its functional form. In some embodiments, for example, when the machine learning model 102 is an LLM, the performance estimation function 540 can be approximated as:
[0161] In this case, is a set of parameters 542 of the performance estimation function 540, which determines right and The functional dependency of . Eq. The first term captures the loss of the ideal generative process to the data distribution. The second term takes into account the loss of the model size The last term considers machine learning models that have not been trained to convergence.
[0162] refer to Fig. 9A , TOS 500-3 first measures the performance of the test allocation tuple 330 350 is compared with the predicted performance metric generated by the performance estimation function 540 to determine the value of the parameter 542. Specifically, the TOS 500-3 uses the performance estimation function 540 to process the trial model size 332.i and the trial data size 334.j of each trial allocation tuple 330.ij to generate the corresponding predicted performance metric Then, TOS 500-3 uses the error metric 550 to compare the difference between the observed and predicted performance measures:
[0163] In some implementations, the error metric 550 is the Huber loss, which corresponds to:
[0164] Huber loss ( ) are usually robust to outliers, which makes them well suited for predicting performance.
[0165] Then, TOS 500-3 estimates the parameters of the performance estimation function 540. 542 Optimize 902 the error metrics 500 to determine their corresponding values.
[0166] refer to Fig. 9B , TOS 500-3 uses unknown performance functions Alternative known performance estimation functions Then, TOS 500-3 determines the value of mapping parameter 126 so that the target model size 132 and target data size 134Optimize each input computation budget 512 to allocate mapping 120 performance estimation function In other words, the target size 132 / 134 corresponds to the extreme value of the performance estimation function 550 for each input computation budget 512:
[0167] Note that the above equation is subject to the constraint that the total computation = equal to the constraint of the input computation budget 512. TOS 500-3 may implement a computation function of the form However, as mentioned with respect to FOS 500-1 and SOS 500-2, TOS 500-3 may use the total computational power consumed during the training of the trial machine learning model 302.ij to determine empirically (e.g., by interpolation) .use , TOS 500-3 may estimate the value of mapping parameter 126 as follows: in Represents a given computational budget The target model size is Represents a given computational budget The target amount of training data is . In this case, is the reference equation The allocation map parameters 126 of the allocation map 120 described above correspond to the same parameters 542 of the performance estimation function 540, but determine and right The functional dependency of .
[0168] Fig.10 1 is a flow chart of an example process 1000 for determining the value of a set of allocation mapping parameters using a performance estimation function. For convenience, process 1000 will be described as being performed by a system of one or more computers located in one or more locations. For example, an optimization system appropriately programmed according to the present specification, such as the third optimization system 500-3 of FIG. 9 , can perform process 1000.
[0169] The optimization system determines a set of parameters of a performance estimation function configured to process data defining: (i) an input model size, and (ii) an input amount of training data to generate a predictive performance metric that characterizes a predictive performance of a machine learning model trained on the input amount of training data having the input model size on a machine learning task (1010). The optimization system fits values of the set of parameters of the performance estimation function based on the performance metric corresponding to the plurality of trial allocation tuples.
[0170] In some embodiments, step 1010 is completed by step 1012, which is performed as follows:
[0171] The values of a set of parameters of the system fitting performance estimation function are optimized to minimize, for each trial assignment tuple, an error metric between: (i) a performance metric corresponding to the trial assignment tuple, and (ii) a predicted performance metric generated by using the performance estimation function to process a trial model size defined by the trial assignment tuple and a trial amount of training data (1012).
[0172] The optimization system determines values for a set of allocation mapping parameters using the performance estimation function (1020).
[0173] In some implementations, step 1020 is completed by step 1022, which is performed as follows:
[0174] The optimization system determines values for a set of allocation mapping parameters so that each input computational budget is mapped to a target model size and a target amount of training data that optimizes a performance estimation function subject to training a machine learning model with a target model size on the target amount of training data using an amount of computational resources given by the input computational budget.
[0175] Fig.11A and 11B An example of experimental results comparing the performance of: (i) a "computationally optimal" machine learning model generated by the training system 300 described in this specification, and (ii) an alternative machine learning model ("Gopher"). The computationally optimal machine learning model requires the same computational budget as the alternative machine learning model during training, but has 4 times fewer model parameters and is trained on 4 times more training data. Fig.11A The improvement (measured in bits per byte) of a computationally optimal machine learning model on a set of language modeling tasks is shown compared to alternative machine learning models. Fig. 11B The relative improvement (expressed as a percentage) of the computationally optimal machine learning model on a set of language understanding tasks compared to alternative machine learning models is shown. It can be appreciated that the computationally optimal model generated by the training system 300 described in this specification significantly outperforms the alternative models.
[0176] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0177] The embodiments and functional operations of the subject matter described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules, which are encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiving device for execution by a data processing device.
[0178] The term "data processing apparatus" refers to data processing hardware and covers all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0179] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to a related program, or in multiple coordinated files, such as files storing one or more modules, subroutines, or portions of code. A computer program may be deployed for execution on one computer located at one site or on multiple computers distributed across multiple sites and interconnected by a data communications network.
[0180] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.
[0181] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or a combination of a dedicated logic circuit and one or more programmed computers.
[0182] Computers suitable for executing computer programs may be based on general or special purpose microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented or incorporated into a dedicated logic circuit. Typically, a computer will also include or be operably coupled to receive data from or transmit data to or to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0183] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
[0184] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a monitor display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display), for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.
[0185] The data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.
[0186] Machine learning models can be implemented and deployed using a machine learning framework such as the TensorFlow framework.
[0187] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0188] A computing system may include a client and a server. The client and the server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to a user device, for example, for displaying data to a user interacting with the device and receiving user input from the user, the device acting as a client. The data generated at the user device, such as the result of the user interaction, may be received from the device at the server.
[0189] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that can be claimed, but should be interpreted as descriptions of the features of specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be cut out from the combination, and the claimed combination may be directed to a sub-combination or a variant of the sub-combination.
[0190] Similarly, although operations are depicted in a particular order in the figures and recited in the claims, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0191] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes shown in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A method performed by one or more computers, the method comprising: obtaining data defining a computing budget, the computing budget characterizing an amount of computing resources allocated for training a machine learning model to perform a machine learning task; processing the data defining the computational budget using an allocation map according to a set of allocation map parameters to generate an allocation tuple, the allocation tuple defining: (i) a target model size for the machine learning model, and (ii) a target amount of training data for training the machine learning model, wherein selecting the model size of the machine learning model as the target model size and training the machine learning model on the target amount of training data is predicted to optimize the performance of the machine learning model on the machine learning task subject to the constraint that the amount of computing resources used to train the machine learning model meets a threshold defined by the computing budget; Instantiating the machine learning model, wherein the machine learning model has the target model size; Obtaining the target amount of training data for training the machine learning model; and The machine learning model having the target model size is trained on the target amount of training data.
2. The method according to claim 1, wherein: The value of the allocation mapping parameter set is determined by an operation, and the operation includes: identifying a plurality of trial allocation tuples, wherein each trial allocation tuple defines: (i) a trial model size for the machine learning model, and (ii) a trial amount of training data used to train the machine learning model; determining, for each trial allocation tuple of the plurality of trial allocation tuples, a performance metric characterizing performance of a trial machine learning model on the machine learning task, the performance metric resulting from selecting a model size of the trial machine learning model to be the trial model size and training the trial machine learning model on the trial amount of training data; and The values of the set of allocation mapping parameters are determined based on the performance metrics corresponding to the plurality of trial allocation tuples.
3. The method according to claim 2, wherein: Determining the values of the allocation mapping parameter set based on the performance metrics corresponding to the plurality of trial allocation tuples comprises: determining, for each of a plurality of computational budgets, an optimal model size and an optimal amount of training data corresponding to the computational budget based on the performance metrics corresponding to the plurality of trial allocation tuples; and The values of the set of allocation mapping parameters are determined based on the optimal model size and the optimal amount of training data corresponding to each of the plurality of computational budgets.
4. The method according to claim 3, wherein: Determining the values of the set of allocation mapping parameters based on the optimal model size and the optimal amount of training data corresponding to each of the plurality of computational budgets includes: The values of the set of allocation mapping parameters are fitted based on the optimal model size corresponding to each of the plurality of computational budgets and the optimal amount of training data.
5. The method according to any one of claims 3 to 4, wherein: For each of the plurality of computational budgets, determining the optimal model size and the optimal amount of training data corresponding to the computational budget comprises: determining a corresponding performance curve for each of a plurality of trial model sizes based on the performance metrics corresponding to the plurality of trial allocation tuples, wherein the performance curve for the trial model size defines a continuous mapping from possible computational budgets to predictive performance metrics, wherein the predictive performance metric corresponding to a possible computational budget defines the predictive performance of a trial machine learning model of the trial model size trained using an amount of computational resources that satisfies a threshold defined by the possible computational budget; and The optimal model size and the optimal amount of training data corresponding to each computational budget are determined using the performance curve.
6. The method according to claim 5, wherein: The performance curves used to size the test model include: The performance curve for the experimental model size is determined by interpolating the performance metrics of experimental allocation tuples corresponding to the experimental model size.
7. The method according to any one of claims 5 to 6, wherein: Determining the optimal model size and the optimal amount of training data corresponding to each computational budget using the performance curve includes, for each computational budget of the plurality of computational budgets: Determining from among the performance curves an optimal performance curve that achieves an optimal performance metric for the computational budget; determining the optimal model size as the test model size corresponding to the optimal performance curve; as well as The optimal amount of training data is determined based on the computational budget and the optimal model size.
8. The method according to claim 3, wherein: For each of the plurality of computational budgets, determining the optimal model size and the optimal amount of training data corresponding to the computational budget comprises: determining a respective performance curve for each of the plurality of computational budgets based on the performance metrics corresponding to the plurality of trial allocation tuples, wherein a performance curve for a computational budget defines a continuous mapping from possible model sizes to predictive performance metrics, wherein the predictive performance metric corresponding to a possible model size defines a predictive performance of a trial machine learning model having the possible model size trained using an amount of computational resources that satisfies a threshold defined by the computational budget; and The optimal model size and the optimal amount of training data corresponding to each computational budget are determined using the performance curve.
9. The method according to claim 8, wherein: The performance curves used to determine the computational budget include: determining the performance curve for the computational budget by interpolating performance metrics of trial allocation tuples corresponding to the computational budget, Wherein, a trial allocation tuple corresponds to the computational budget if training a trial machine learning model having the trial model size defined by the trial allocation tuple on the trial amount of training data defined by the trial allocation tuple will use an amount of computing resources that meets a threshold defined by the computational budget.
10. A method according to any one of claims 8 to 9, wherein Determining the optimal model size and the optimal amount of training data corresponding to each computational budget using the performance curve includes, for each computational budget of the plurality of computational budgets: determining the optimal model size as a model size that optimizes the performance curve corresponding to the computational budget; as well as The optimal amount of training data is determined based on the computational budget and the optimal model size.
11. The method according to claim 2, wherein: Determining the values of the allocation mapping parameter set based on the performance metrics corresponding to the plurality of trial allocation tuples comprises: Determining a set of parameters of a performance estimation function, the performance estimation function being configured to process data defining: (i) an input model size, and (ii) an input amount of training data, to generate a predictive performance metric characterizing a predictive performance of a machine learning model having the input model size and trained on the input amount of training data on the machine learning task, comprising: fitting values of the set of parameters of the performance estimation function based on the performance metrics corresponding to the plurality of trial allocation tuples; and The value of the set of allocation mapping parameters is determined using the performance estimation function.
12. The method according to claim 11, wherein: Determining the value of the allocation mapping parameter set using the performance estimation function includes: Determining the values of the allocation mapping parameter set so that each input computational budget is mapped to a target model size and a target amount of training data that optimizes the performance estimation function subject to the constraint of using an amount of computational resources given by the input computational budget to train a machine learning model with the target model size on the target amount of training data.
13. A method according to any one of claims 11 to 12, wherein Fitting the values of the set of parameters of the performance estimation function based on the performance metrics corresponding to the plurality of trial allocation tuples includes: The values of the set of parameters of the performance estimation function are fitted to minimize, for each trial allocation tuple, an error metric between: (i) the performance metric corresponding to the trial allocation tuple, and (ii) a predicted performance metric generated by using the performance estimation function to process the trial model size defined by the trial allocation tuple and the trial amount of training data.
14. The method according to claim 13, wherein: The error metric includes the Huber loss.
15. A method according to any one of claims 2 to 14, wherein For each of the plurality of trial allocation tuples, determining the performance metric corresponding to the trial allocation tuple comprises: A trial machine learning model having the trial model size is trained on the trial amount of training data using a learning rate schedule selected based on the trial amount of training data.
16. A method according to any preceding claim, wherein: The allocation map causes the target model size and the target amount of training data to increase at substantially the same rate in response to an increase in the computational budget.
17. A method according to any preceding claim, wherein: The machine learning task includes a language modeling task.
18. A method according to any preceding claim, wherein: The machine learning model includes a neural network model.
19. The method according to any preceding claim, further comprising: receiving a model input to the machine learning model; as well as The model input is processed using the machine learning model to generate a model output according to the training values of the set of model parameters of the machine learning model.
20. The method of claim 19, wherein: The machine learning model comprises a multimodal model in which one or both of the model input and the model output comprises an image or audio, and The multimodal model is configured to process the model input comprising at least one of visual words representing pixels of a still or moving image, data representing an audio waveform, and text words representing a text sequence to generate the model output comprising text words, images, or audio representing the model input.
21. A method according to any preceding claim, for adapting the machine learning model to specific computing hardware, wherein: The machine learning model includes a neural network model and the specific computing hardware includes a plurality of neural network accelerators, and the method includes: determining an energy budget for training the machine learning model, wherein the energy budget defines a total number of floating point operations used to train the machine learning model; determining the computational budget according to the energy budget; determining a hardware specification of the specific computing hardware on which the machine learning model is to be trained, wherein the hardware specification defines a number of the neural network accelerators in the specific computing hardware; determining the target model size of the machine learning model using the method of any preceding claim, wherein the target model size defines the number of trainable parameters of the machine learning model; determining the target amount of training data for training the machine learning model using the method of any preceding claim, wherein the target amount of training data defines the number of training tokens to be used to train the model; and The machine learning model having the defined number of trainable parameters is trained on the specific computing hardware using the defined number of training tokens.
22. A method performed by one or more computers, comprising: receiving a model input to the machine learning model; as well as processing the model input using the machine learning model to generate a model output according to training values of a set of model parameters of the machine learning model; The machine learning model has been generated by an operation, wherein the operation includes: Obtaining data defining a computing budget, the computing budget representing an amount of computing resources allocated to train the machine learning model to perform the machine learning task; processing the data defining the computational budget using an allocation map to generate an allocation tuple according to a set of allocation map parameters, the allocation tuple defining: (i) a target model size for the machine learning model, and (ii) a target amount of training data for training the machine learning model, wherein selecting the model size of the machine learning model as the target model size and training the machine learning model on the target amount of training data is predicted to optimize the performance of the machine learning model on the machine learning task subject to the constraint that the amount of computing resources used to train the machine learning model meets a threshold defined by the computing budget; Instantiating the machine learning model, wherein the machine learning model has the target model size; Obtaining the target amount of training data for training the machine learning model; and The machine learning model having the target model size is trained on the target amount of training data.
23. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 22.
24. A system comprising: one or more computers; as well as One or more storage devices, the one or more storage devices being communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1 to 22.
