Hardware-aware progressive training of machine learning models

Abstract

Hardware-aware progressive training of machine learning models is provided. A model is trained according to different values ​​of hardware-level and model-level performance settings in a training process specified in a training schedule. The hardware-level performance settings can enable / disable / modify the hardware characteristics of the computing resources used for training at various points during training. The model-level performance settings can take on a wide variety of values ​​to adjust characteristics of the model being trained or the training process at different stages of training. Complementary values ​​of the hardware-level and model-level performance settings are identified / applied to generate a training schedule that improves training speed in early stages of training while improving model quality in later stages of training.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 899,728, filed August 31, 2022, and claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 252,743, filed October 6, 2021, the disclosure of which is hereby incorporated by reference herein. [Background technology]

[0002] background A neural network is a machine learning model that includes one or more layers of nonlinear operation that predict an output for a received input. In addition to an input layer and an output layer, some neural networks include one or more hidden layers. The output of each hidden layer can be input to another hidden layer or to an output layer of the neural network. Each layer of a neural network can generate a respective output from a received input according to values ​​for one or more model parameters for that layer. The model parameters can be weights and / or bias values ​​determined through a training process that cause the neural network to generate an accurate output when evaluated using a performance or loss function.

[0003] Increasing the speed of the training process is essential to improving machine learning models. There are several platform / hardware optimizations that can achieve a trade-off between training speed and quality. However, because the quality of machine learning models is so important, hardware techniques are not applied to speed up the training process unless there is a loss in quality, which leads to many performance optimization opportunities being unavailable. Summary of the Invention

[0004] Quick Overview Aspects of the present disclosure provide hardware-aware progressive training of machine learning models. Progressive learning or training is a technique for training a machine learning model by adjusting the model or the training process for training the model while training the model. The progressive training system can generate and apply different values ​​of both model-level and hardware-level performance settings at different stages of the training process to improve the speed at which the progressive training system trains the model while maintaining model quality according to a predetermined minimum threshold.

[0005] The model-level performance settings correspond to the characteristics of the machine learning model being trained or the parameters of the training process applied. The training system can adjust to different values ​​of the model-level performance settings during training independent of the computing resources used to train the model. The hardware-level performance settings correspond to the hardware features of the computing resources used to train the machine learning model. The hardware-level performance settings can take different values ​​to enable, disable, or modify different hardware features during training applied by the training system.

[0006] The training system utilizes existing hardware features to adjust both hardware-level and model-level performance settings during training of the machine learning model at different stages of the training process. The training system can identify and apply complementary values ​​for the hardware-level and model-level performance settings to generate a training schedule that improves model training speed in early stages of training while maintaining or improving model quality in later stages of training.

[0007] Aspects of the present disclosure provide for improving training speed by using available computing resources and their respective available hardware features, such as hardware parallelism, operand numerical precision, and variable levels of intra- and inter-device communication, to improve the speed at which models are trained compared to progressive training alone. The training system can be scaled as needed to take advantage of the hardware features of the computing resources of the computing platforms of the connected devices to further improve the speed at which the training process is performed.

[0008] The training system can generate and store training schedules that are later queried for reuse in training other machine learning models or previously trained models. The training system can use previously generated training schedules, e.g., portions of training schedules focused on improving model quality before increasing training speed, to retrain models on new training data.

[0009] Aspects of the present disclosure also provide for exploring neural architectures that can be modified during training according to a training schedule, e.g., with less computational overhead than modifying other candidate architectures and / or to make greater use of hardware-aware progressive training to achieve increased training speed compared to other architectures.

[0010] One aspect of the disclosure relates to a system including one or more processors configured to receive a request to train a machine learning model; receive, by the one or more processors, a training schedule that specifies multiple values ​​for one or more hardware-level performance settings and one or more model-level performance settings; train the machine learning model in accordance with the training process, the one or more hardware-level performance settings, and the one or more model-level performance settings that are set to different ones of the multiple values ​​of the training schedule at different times during the training; and, in response to receiving the request, transmit the trained machine learning model to one or more computing devices.

[0011] One aspect of the disclosure relates to a method, the method including receiving, by one or more processors, a request to train a machine learning model, the one or more processors configured to train the machine learning model in accordance with one or more hardware-level performance settings and one or more model-level performance settings, the method further including receiving, by the one or more processors, a training schedule specifying a plurality of values ​​for the one or more hardware-level performance settings and the one or more model-level performance settings; training, by the one or more processors, the machine learning model in accordance with the training process and the one or more hardware-level performance settings and the one or more model-level performance settings that are set to different ones of the plurality of values ​​of the training schedule at different times during the training; and in response to receiving the request, transmitting, by the one or more processors, the trained machine learning model to one or more computing devices.

[0012] One aspect of the disclosure relates to one or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more processors configured to train a machine learning model according to one or more hardware-level performance settings and one or more model-level performance settings, cause the one or more processors to perform operations including receiving a request to train a first machine learning model; receiving a training schedule specifying a plurality of values ​​for the one or more hardware-level performance settings and the one or more model-level performance settings; training the first machine learning model according to the training process and the one or more hardware-level performance settings and the one or more model-level performance settings that are set to different ones of the plurality of values ​​of the training schedule at different times during training; and transmitting the trained first machine learning model to one or more computing devices in response to receiving the request.

[0013] Aspects of the disclosure may include one or more of the following features. In some examples, an aspect of the disclosure includes all of the following features in combination.

[0014] The one or more model-level performance settings may include one or more of an input data size for input data to the machine learning model, one or more model hyperparameters that specify the size or shape of the machine learning model, and one or more training process hyperparameters that modify the training process performed by the one or more processors to train the machine learning model.

[0015] The one or more hardware-level performance settings may include settings for regulating intra-processor or inter-processor data communications of one or more processors.

[0016] The one or more processors may include multiple processors logically or physically grouped into multiple groups, and the one or more hardware-level performance settings may include settings for the speed of inter-processor data communication within the different groups.

[0017] The one or more hardware-level performance settings may include settings for adjusting the numerical precision of operations performed by the one or more processors while training the machine learning model according to the training process.

[0018] During training of the machine learning model, the one or more processors may be further configured to set the one or more hardware-level performance settings and the one or more model-level performance settings to a first value of a plurality of values ​​of the training schedule, and at a first time after initiation of training of the machine learning model, adjust the one or more hardware-level performance settings and the one or more model-level performance settings to a second value of the plurality of values ​​different from the first value.

[0019] Upon receiving the training schedule, the one or more processors may be further configured to generate a training schedule using a training schedule machine learning model, where the training schedule machine learning model is trained to generate the training schedule from one or more input parameters that at least partially describe one or more of the machine learning models, the machine learning tasks, and the computing resources available for training the machine learning models, and where each example training schedule is trained using one or more training examples of the training schedule, labeled with respective data that at least partially describes one or more respective input parameters used to generate the example training schedule, the training speed, and the training process and model quality of the respective machine learning models trained according to the example training schedule.

[0020] The machine learning model is a neural network having a neural architecture selected from a plurality of candidate neural architectures, the selection of the neural architecture being based at least in part on a comparison of estimated respective training speeds of the neural networks and respective model qualities, and the neural network is trained according to a training process and a respective training schedule, the neural network having each candidate neural architecture of the plurality of candidate neural architectures.

[0021] Upon receiving the training schedule, the one or more processors may be further configured to send a query including data at least partially describing one or more of the machine learning model, the machine learning task, and / or computing resources available for training the machine learning model to one or more memory devices that store a plurality of candidate training schedules, and in response to the query, receive a training schedule from the plurality of candidate training schedules.

[0022] One aspect of the disclosure relates to a method that includes performing, by one or more processors, a neural architecture search across a plurality of candidate neural architectures to identify a target neural architecture, the method including estimating at least a training speed and a model quality of a first neural network having a first candidate neural architecture of the plurality of candidate neural architectures and trained according to a training process and one or more hardware-level performance settings and one or more model-level performance settings that are set to different values ​​of a first plurality of values ​​during training, and selecting the first candidate neural architecture as the target neural architecture based at least on a comparison of the estimated training speed and estimated model quality of the first neural network to respective estimated training speeds and respective estimated model qualities of one or more second neural networks, each having a second candidate neural architecture and trained according to a training process and one or more hardware-level performance settings and one or more model-level performance settings that are set to different values ​​of a respective second plurality of values ​​during training.

[0023] The method may further include training, by the one or more processors, the first neural network according to a third plurality of values ​​of the training schedule, and transmitting, by the one or more processors, the trained first neural network to one or more computing devices. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a block diagram of an exemplary training system according to an aspect of the present disclosure. [Diagram 2] 1 is a flowchart of an example process for hardware-aware progressive training of machine learning models according to aspects of the present disclosure. [Figure 3A] 1 is a flowchart of an example process for training a machine learning model to generate a training schedule for hardware-aware progressive training, according to aspects of the present disclosure. [Figure 3B] 1 is a flowchart of an example process for querying and applying pre-generated training schedules from one or more memory devices that store multiple training schedules, according to an aspect of the disclosure. [Figure 4] 1 is a flowchart of an example process for exploring neural architectures according to aspects of the present disclosure. [Diagram 5] FIG. 1 is a block diagram of an exemplary computing environment for implementing an exemplary training system according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Detailed Description overview Aspects of the present disclosure provide hardware-aware progressive training of machine learning models. Hardware-aware progressive training refers to applying a wide variety of different values ​​to both model-level and hardware-level performance settings during training of a machine learning model that are adjusted to different values ​​over the course of training. The training system can generate and apply a training schedule that specifies multiple values ​​of model-level and hardware-level performance settings to be applied at different points during training. As described herein, a training system configured for hardware-aware progressive training can improve the speed at which the training system trains a model at early points in the training process, as well as improve the model quality of the trained model at later points in the training process, compared to other approaches in which hardware-aware progressive training is not applied.

[0026] The hardware-level performance settings may include settings for adjusting the performance of the computing resources used to train the machine learning model. Values ​​for the hardware-level performance settings may be adjusted to enable, disable, or modify specific hardware features available on the computing resources. The computing resources may be any of a wide variety of combinations of computing and memory devices that may be part of a computing platform, for example. The computing platform may logically organize how the devices communicate with each other, and that organization may also be modified by different values ​​of the corresponding hardware-level performance settings.

[0027] These hardware features may be selectively applied by the training system to adjust the performance of the computing resources in performing operations as part of the training process. For example, hardware features applied according to different values ​​of corresponding hardware-level performance settings may cause the computing resources to perform operations more quickly, as measured in processing cycles, clock time, etc., at the expense of accuracy rates in performing those operations. Other values ​​for the hardware-level performance settings may cause the computing resources to perform operations more accurately, such as different numerical calculations, at the expense of additional processing cycles, processing / memory utilization, and / or time, etc. As a result, the trained models have improved model quality, as measured, for example, by model precision or recall rates.

[0028] Model-level performance settings applied at different values ​​by the training system modify the machine learning model or the training process itself. Model-level performance settings do not affect the hardware or hardware features used by the training system during training, but depending on the values ​​used for those settings, they may affect the quality of the resulting trained model and the speed at which the model is trained. Hardware-aware progressive training achieves a more effective use of the available configurations of both model-level and hardware-level features available on the platform when training the model, in order to reach higher training speeds and maintain or improve model quality at different stages of training that cannot be reached by progressive training alone.

[0029] The training system can train the machine learning model over multiple phases. A training phase can be defined as several training steps, each of which represents a complete forward and backward pass that updates model parameter values ​​based on the calculated error. The number of training steps in a training phase can be variable, e.g., from thousands to millions. The number of training steps can be variable, e.g., based on the total number of training steps for all of the training phases and / or the size of the training dataset. In some examples, a phase can be defined as a period of time that is shorter than a particular model performance milestone achieved, such as the total training time to train the model, the number of epochs or times the entire training set is processed by the model, and / or any threshold based metric for measuring a threshold recall or model accuracy.

[0030] For example, the training system can apply values ​​for model-level performance settings that correspond to smaller network size, smaller input size, lower regularization and / or lower regularization, etc., which may result in faster training at the expense of model quality. The training system can apply model-level performance settings with different values ​​that correspond to larger network size, larger input size, higher regularization, and / or higher regularization, which may result in slower training due to performance overhead, but higher model quality.

[0031] Training speed can be measured, for example, in terms of the number of processing cycles required to train a machine learning model across an epoch of training data, by how long it takes to process an individual training example or a mini-batch of training examples, and / or by the number of processing cycles required to complete one or more stages of training. Model quality can be measured, for example, according to how well a machine learning model performs the task it was trained to perform. Exemplary metrics for measuring model quality can include recall, loss between model predictions and corresponding ground truth labels, model accuracy, and / or model precision in performing a machine learning task.

[0032] During training, the training system applies different values ​​for both hardware-level and model-level performance settings and adjusts those values ​​at different points during training to achieve different tradeoffs between training speed and model quality. Exemplary points at which the training system applies different values ​​include the beginning of different stages of training defined, for example, according to time, number of training iterations, or meeting minimum milestones for model quality. Other examples include time-based intervals, such as every minute or hour that passes during training.

[0033] Based on the training schedule as described herein, the training system first applies values ​​to the performance settings to adjust the training of the model to emphasize training speed over model quality in order to learn high-level patterns and their relationships between training examples and their labels at a high training speed. As the training progresses, the training system gradually adjusts the values ​​of the performance settings to emphasize model quality improvement with speed overhead according to a rate of change that can be specified in the training schedule. When the training reaches its final stage, the training system applies values ​​of the hardware-level and model-level performance settings to emphasize model quality with little or no importance given to reduce performance overhead, resulting in a slowdown in the training speed.

[0034] The training system can generate a training schedule with complementary values ​​for various hardware-level and model-level performance settings. The complementary values ​​for model-level performance settings allow certain hardware features to be applied more efficiently, for example resulting in fewer processing cycles to perform operations as part of the training process implementation, or allowing optimization processes to improve model quality. For example, values ​​of model-level performance settings for enabling quadratic optimization methods during training complement values ​​for hardware-level performance settings that correspond to performing operations with lower numerical precision, for example using less than 64-bit floating point or integer precision.

[0035] The training system can identify complementary values ​​of the performance settings by the training system as part of generating the training schedule. For example, the training system can implement a training schedule machine learning model that is trained to generate a training schedule from one or more input parameters that at least partially describe one or more of the machine learning model to be trained on a set of computing resources, the machine learning task, and the set of computing resources available for training the model. In some examples, the training system can explore a space of candidate training schedules according to different optimization parameters or search criteria, as described herein.

[0036] Examples of complementary values ​​include values ​​for lower resolution, weaker regularization, and smaller models paired with hardware-level performance settings for local node communication and gradient accumulation and lower precision computations. In later stages of training, higher resolution, stronger regularization, and larger models may be paired with hardware-level performance values ​​for global communication and gradient accumulation, as well as higher precision computations.

[0037] As better training schedule performances are identified, e.g., by observing higher training speeds and / or higher model quality at different points during training, those training schedules can be provided to, as further examples, retrain the training schedule machine learning model or to update search criteria for a search for a training schedule given a set of input parameters. In general, a higher performing training schedule includes complementary hardware-level and model-level performance settings over a lower performing training schedule.

[0038] Aspects of the present disclosure provide at least the following technical advantages: Machine learning models can be trained faster, e.g., in less clock time and / or using fewer processing cycles, relative to other models that have not been trained using hardware-aware progressive training. At later stages of training, model quality can be qualified by gradually adjusting performance settings to favor model quality at the expense of performance overhead. Improved model quality of trained machine learning models can improve the ability of computing devices to deploy the models in inference, for example, because responses to queries or requests to process data on the model can be generated more accurately.

[0039] Training can be performed more efficiently, e.g., using more available features to accelerate operations as part of implementing the training process, versus not using a training schedule as described herein. The training system is configured to generate a training schedule with complementary values ​​to reduce or avoid conflicting values ​​of hardware-level and model-level performance settings that may inhibit training.

[0040] The training schedule applied and generated by the training system is adjusted according to the available hardware features of the training process and the computing resources designed for training the model using the given training schedule. For example, a computing platform may include a wide variety of different computing devices available for training the machine learning model, with the different devices varying in terms of available hardware features and / or data processing capabilities.

[0041] Because the training system can apply a training schedule having hardware-level performance settings based on the particular hardware characteristics and processing power available by the allocated computing resources, the training system can more efficiently use the allocated computing resources for training a particular machine learning model. The training system can apply the same training schedule to the same set of computing resources at different scales so as not to add additional processing overhead to the platform operation to scale computing resources up or down during or between training sessions.

[0042] The overhead of adjusting model-level and hardware-level performance settings generates little or negligible overhead for training and running machine learning models. As a result, changes can often be applied to both model-level and hardware-level performance settings to vary the tradeoff between model quality and training speed. Despite the large number of possible combinations of model-level and hardware-level performance settings, aspects of the present disclosure provide for exploring the space of candidate training schedules to identify combinations of model-level and hardware-level performance settings for improving or maintaining model quality at a higher training speed compared to other approaches where hardware-aware progressive training is not applied.

[0043] Exemplary System 1 is a block diagram of an exemplary training system 100 according to aspects of the disclosure. Training system 100 can be implemented on one or more computing devices in one or more physical locations. Training system 100 is shown in FIG. 1 as part of a computing platform 101. Computing platform 101 can be a collection of computing devices that communicate with one or more other computing devices, such as computing device 105, over a network.

[0044] The training system 100 includes a training engine 110, and may further include a training schedule engine 115 and a training schedule library 120. In some examples, the training system 100 may further include a neural architecture search engine 125.

[0045] The training system 100 is configured to receive a request to train a machine learning model, for example, from a computing device 105. As an example, the computing device 105 can send the request to the training system 100 via some interface, such as, for example, an API or a web interface on a browser or a mobile application presented on a display of the computing device 105.

[0046] The computing device 105 may be a device configured to automatically communicate with a user's computing device operated by a user and / or the training system 100. For example, the computing device 105 may be configured to receive and deploy a trained machine learning model. The computing device 105 may be further configured to receive requests from other computing devices (not shown) to process inputs by the deployed model to generate respective output data. The other computing devices may be connected to the computing device 105 separately or as part of a network connecting the platform 101 with the computing device 105.

[0047] The request from the computing device 105 may specify input parameters that at least partially describe the machine learning model, the machine learning task, and / or the computing resources available to train the model. The input parameters describing the machine learning model may include a model type, such as a neural network, a support vector machine, a regression model, etc. The input parameters may further include specific characteristics of the desired machine learning model, such as a neural network having a particular width or depth.

[0048] The input parameters may further specify the type of machine learning task that the machine learning model is trained to perform, such as a regression or classification task. Exemplary machine learning tasks are provided herein, and generally, a machine learning task may be defined for the approximation of a function between a set of inputs and corresponding outputs that are learned by a machine learning model trained to perform the task. The input parameters may further specify the subtype of the machine learning task that the machine learning model is trained to perform, such as binary classification, multi-class classification, linear regression, logistic regression, etc.

[0049] The training system 100 may be configured to automatically select a type of machine learning model when a task is specified in the input parameters, but is not configured to select a model type. For example, the training system 100 may be part of an automated machine learning (AutoML) system (not shown in FIG. 1). The AutoML system may be configured to automatically select a machine learning model to perform based on input parameters that, optionally, among other input parameters, specify a task to be performed. Even if the input parameters specify a model type, in some examples, the AutoML system implementing the training system 100 may be configured to suggest one or more model types based on other received parameters. As described in more detail with respect to FIG. 4, the training system 100 implements a neural architecture search (NAS) engine 125 that, in some examples, is configured to identify neural architectures for training neural networks having those architectures and that may be trained using hardware-aware progressive training.

[0050] A neural architecture refers to a set of values ​​that describe the shape or topology of a neural network. Exemplary values ​​that may be part of a neural architecture include, for example, the number of layers the architecture has, the width of each layer, the number of nodes or neurons in each layer, the type of operation performed in each layer given a set of inputs, and the type of activation function applied for one or more of the layers of the network. Each neural network shall have its own neural architecture.

[0051] The input parameters may also specify the computing resources on which the training system 100 should train the machine learning model. The computing resources 130 of the computing platform 101 may include a wide variety of different computing devices including a wide variety of different types and configurations of processors and memory devices, as described herein with reference to FIG. 5. The computing resources 130 may include several computing devices with various hardware features to improve data processing and storage on the computing devices. These hardware features may be enabled, disabled, or modified according to different values ​​of hardware-level performance settings adjusted by the training system 100.

[0052] The input parameters can specify how much, what kind, and / or which specific computing resources should be used by the training system 100 in training the machine learning model. For example, the computing device 105 may be associated with a user to which a portion of the computing resources 105 is assigned. In other examples, the platform 101 may provide more or less computing resources, measured, for example, in the amount of time available, the number of processing cycles, or more or fewer devices with different processing speeds or capabilities. Processing capabilities can be measured, for example, in clock speed, data bandwidth, cache memory size, etc. For example, the request may specify the use of a graphics processing unit (GPU) to accelerate the training of the machine learning model, as compared to the use of other less specialized devices such as a central processing unit (CPU).

[0053] The request may also specify training data or a location of the training data to be used to train the machine learning model. For example, the training data may be stored on one or more computing devices of the platform 101, which may be the same as or different from the device implementing the training system 100. The training data may include, for example, one or more training examples of inputs that the model is trained to process and generate respective outputs. Some or all of the training examples may include labels of ground truth outputs that correspond to the labeled examples.

[0054] The training engine 110 receives a request from the computing device 105 and a training schedule that specifies values ​​for hardware-level and model-level performance settings for training the machine learning model according to the request. As described in more detail with reference to FIGS. 3A-3B, the training engine 110 can receive the training schedule, for example, from a training schedule engine 115 configured to generate a training schedule according to aspects of the present disclosure. In another example, the training engine 110 receives the training schedule by querying a training schedule library 120 that stores a collection of pre-generated training schedules.

[0055] The training engine 110 implements a training process for training the machine learning model over a training period. The training process may include any set of operations for training the machine learning model that may be repeated one or more times over the training period. The training process may be variable depending on, for example, the type of model to be trained and / or the nature of the machine learning task the model is to be trained to perform. Exemplary processes may be based on supervised, unsupervised, or semi-supervised learning techniques. For example, the training engine 110 may be configured to train the machine learning model as a neural network using backpropagation with gradient descent and further updating one or more weights or model parameter values ​​for the machine learning model according to the calculated gradients and, optionally, one or more other parameters. As described herein, some model-level performance settings set to different values ​​may cause the training engine 110 to modify the training process to train the model.

[0056] The training engine 110 may also be configured to perform various optimization processes as part of training, including, for example, adaptive moment estimation (Adam) optimization, stochastic or mini-batch gradient descent, gradient descent with momentum, as well as processes to reduce overfitting in the trained model, for example using dropout.

[0057] Other training processes based on different model architectures, such as models based on clustering or support vector machines, can also be applied by training engine 110. In addition, other types of training processes, such as processes based on unsupervised or semi-supervised techniques, can also be performed by training engine 110 to train machine learning models according to aspects of the present disclosure.

[0058] The training period may be defined according to one or more termination criteria, which may be provided, for example, as an additional input parameter as part of the received request or may be pre-determined. Training engine 110 stops training when the termination criterion is met. This criterion may be, for example, a maximum number of iterations of the training process performed by training engine 110, a maximum amount of elapsed time from the start of training, meeting a minimum model quality performance threshold by the trained model, and / or failure of a minimum predetermined improvement to model quality after a certain number of iterations or elapsed time.

[0059] The training system 100 can train the machine learning model over multiple phases. A training phase can correspond to several training steps, each of which represents a complete forward and backward pass of updating model parameter values ​​based on the calculated error. The number of training steps in a training phase can be variable, e.g., from thousands to millions. The number of training steps can be variable, e.g., based on the total number of training steps for all of the training phases and / or the size of the training dataset. In some examples, a phase can be defined as a period of time less than a particular model performance milestone achieved, such as the total training time to train the model, the number of epochs or times the entire training set is processed by the model, and / or any threshold based on a metric to measure a threshold recall or model accuracy.

[0060] At each stage, the training engine 110 can apply different values ​​for hardware-level and model-level performance settings to adjust the training process during that stage. The hardware-level and model-level performance settings can take on a range of values ​​with variable trade-offs between training speed and model quality of the trained machine learning model. The training engine 110 can be configured to perform a combination of hardware-level and model-level training optimizations together and adjust values ​​for both hardware-level and model-level performance parameters to achieve different balances between training speed and model quality of the resulting trained model. The training schedule can specify the ratio at which values ​​are adjusted for various hardware-level and model-level performance settings. For example, if the values ​​are numeric and start at one end of a range of values ​​that emphasizes training speed over model quality, the training schedule can specify the ratio at which values ​​for a particular performance setting are adjusted to transition to a value that emphasizes model quality over training speed, or vice versa.

[0061] Early in the training, the training schedule can specify hardware-level and model-level performance settings that emphasize higher training speed at the expense of model quality. The training schedule can include several intermediate values ​​of both hardware-level and model-level performance settings to transition the training process performed by the system to emphasize model quality over training speed. The training schedule specifies points at which the intermediate values ​​should be applied to the performance settings, and the training system is configured to apply the values ​​for those settings at the specified points. These points can be intervals according to the start of a subsequent training phase and / or other conditions such as time. For example, the training schedule can specify different values ​​for the performance settings at intervals of every minute. Later in the training, the training schedule can specify values ​​or schemes for hardware-level and model-level performance settings that emphasize higher model quality at the expense of lower training speed.

[0062] The range of values ​​for the various hardware-level and model-level performance settings are variable according to at least the types of performance settings available during training. For example, one model-level performance setting is the learning rate for training a machine learning model. Adjustments to the learning rate may be made in very small increments initially, such as from 0.1 to 0.01. After a certain number of stages or training steps, the learning rate may be decreased by an amount, such as 10 times its current value.

[0063] Another exemplary model-level performance setting is regularization. For performance settings such as regularization, where the performance settings include different types or optimization of classification, as opposed to adjusting a numerical value, the value for the performance setting can correspond to the type of scheme enabled by the performance setting. For model regularization, such as data augmentation, the method for augmentation may vary from a single distortion to more advanced blurring and distortion depending on different model-level performance setting values.

[0064] The range of values ​​for the various different hardware-level and model-level performance settings may be integers. As another example, a hardware-level performance setting may be a communication radius for transmitting data such as gradients between chips, nodes, or other devices that train machine learning models. Initially, the communication radius may be small, e.g., 2×2, for communicating between local devices adjacent to each other. This communication radius may be adjusted to increase, e.g., up to 16×16 or more, for communicating with hundreds or thousands of chips over different hardware interconnects within and / or between data centers.

[0065] The training engine 110 is configured to cause the computing resources 130 to perform operations to train the machine learning model according to current values ​​of hardware-level and model-level performance settings.

[0066] For example, the training engine 110 may generate a program or sequence of instructions that, when executed by the computing resource 130, causes the computing resource 130 to perform operations according to values ​​for performance settings specified in the program or sequence of instructions. In some examples, the training engine 110 is configured to enable, disable, or modify the execution of hardware features via one or more control signals to the computing resource device. For example, the training engine 110 may cause different hardware features to be enabled by an operating system or other software or firmware controlling the computing resource 130. In other examples, the training engine 110 may send a signal directly over a bus or communication channel over which the device is configured to receive the control signal to enable or disable the hardware feature.

[0067] Some examples of hardware features that may be adjusted by different values ​​of the hardware-level performance settings include enabling / disabling inter- or internal data communication between computing devices, the level of numerical precision that the computing devices apply to perform their respective operations as part of the training process, and / or enabling / disabling hardware parallelism on the computing devices. In some examples, inter- or internal data communication may be further adjusted, such as the rate, volume, or type of data sent between devices.

[0068] Hardware-level performance settings may include settings for adjusting software-defined or virtually defined clusters of computing devices with logical paths between those computing devices. Exemplary operations performed by computing resources 130 during training may include computing dot products, matrix multiplications of vectors of input values ​​and matrices or tensors of weights of neural network layers, computing activation functions, performing convolution operations, pooling multiple values ​​of feature maps, etc.

[0069] Model-level performance settings may include model hyperparameters, such as the size of the machine learning model or the topology or shape of the neural network, including the size of the input the model receives. Model-level performance settings may also include training process hyperparameters to modify the training process used by the training engine in training the machine learning model, such as a learning rate or batch size. Training process hyperparameters may also include parameters whose values ​​control the application of various optimization processes that can be performed as part of the training process to further improve the model, such as a quadratic optimization method or process for how much of the model's functional parts are regularized or how much of the data is normalized. Examples of training process hyperparameters may also include, for example, a learning rate or mini-batch size when the training process is mini-batch gradient descent.

[0070] In the case of model-level performance settings, the training engine 110 can send signals interpretable by the computing resources 130 to adjust the model-level performance settings in accordance with the training schedule throughout the training period. For example, the training engine 110 can generate a program or sequence of instructions that specifies adjustments to the model and / or training process during training, at which points or stages the adjustments should be made in accordance with the model-level performance settings of the training schedule.

[0071] The training engine 110 can generate a training schedule by exploring adjustments to values ​​for hardware-level and model-level performance settings for hardware-level or model-level features available on the platform on which the system is implemented. As part of the generation, the training engine 110 can identify model-level and hardware-level performance settings that are complementary in achieving higher training speed or model quality depending on the point during training at which the settings are applied.

[0072] For example, different values ​​of hardware-level performance settings for local-only communication of nearby computing devices in a cluster may be paired with different values ​​of model-level performance settings where training engine 110 applies batch normalization or cross-replica gradient summation to accelerate training at the expense of model quality in the early stages of training. The devices of computing resources 130 may be logically and / or physically organized as clusters or groups of computing resources, with interconnections between at least some of the devices in the cluster to facilitate inter-device communication. Hardware-level performance settings that training engine 110 can adjust during training may include settings for adjusting communication overhead between devices in the cluster.

[0073] As yet another example, a value for a hardware-level performance setting for higher numerical precision during training may be paired with a value for a model-level performance setting, which causes the training engine 110 to apply any of a wide variety of quadratic optimization methods for better model quality at the expense of training speed.

[0074] As yet another example, a hardware-level performance setting for enabling parallel computing on a particular type of accelerator, such as a GPU or TPU, may be paired with a particular model-level performance setting for selecting the activation function used during training of a particular neural network. For example, ReLU may be selected as the activation function when parallel computing is selected for faster training at low model quality, but swish may be selected as the activation function later during training for improved model quality at the expense of reduced training speed due to reduced hardware implementation parallelism.

[0075] Due to the large space of model architectures and hardware settings, systems such as the training system 100 described herein may allow hardware settings to be combined with progressive training. For example, simply combining hardware-level and model-level progressive training may cause catastrophic quality loss, making the model quality too low to be useful. As another example, applying lower regularization at the model level and lower precision at the hardware level at the beginning of training may make the initial quality loss too low to recover, even if regularization and numerical precision are significantly increased later during training.

[0076] In some examples, the model may be retrained according to a training schedule or a portion of a training schedule previously used by the training engine in training the model. Retraining may include performing a number of iterations of the training process using new training data. An exemplary retraining may include backpropagation with gradient descent and updating model weights for a previously set neural network from an earlier training. Instead of reusing the same training schedule from an earlier stage of training, the training engine 110 may apply values ​​of hardware-level and model-level performance settings of a previously used training schedule for a later stage or point during training. In this way, values ​​for performance settings that correspond to the current performance of the model (already trained) may be used by the training engine 110 to emphasize improvement of model quality over training speed.

[0077] One exemplary case in which a portion of the training schedule may be used as part of the retraining is when retraining a generative machine learning model, such as a model for an online search engine. The model may be optionally retrained to take into account new training data and / or model-level optimizations that may be deployed after deployment of the generative machine learning model. The training system may reuse the training schedule previously used to initially train the generative machine learning model, but initiate the retraining according to a point or stage where model quality is emphasized over training speed.

[0078] The training schedule library 120 is a collection of pre-generated training schedules stored, for example, on one or more memory devices as part of a queryable database. This training schedule library 120 may be populated with training schedules generated by a training system, as described in more detail with reference to FIG. 2. In some examples, the training schedule engine 115 adds the generated training schedule to the library 120 and tags it with metadata that at least partially describes the input parameters received as part of a request to train a model using the generated training schedule. In other examples, the training schedule engine 115 may populate the training schedule library 120 with one or more training schedules for commonly received machine learning models requested to be trained by the system 100. As described in more detail with reference to FIG. 3B, the training engine 110 may query the training schedule library 120 to identify previously generated stored training schedules for machine learning models that are the same or similar to the model currently being requested by the engine 110 to be trained.

[0079] In some examples, the training system 100 may further include a neural architecture search (NAS) engine 125. As described in more detail with reference to Figure 4, the NAS engine 125 is configured to search for neural architectures for neural networks that benefit from training according to a training schedule as described herein.

[0080] For example, the training system 100 can receive input parameters for training a machine learning model that specify a machine learning task to perform without specifying a particular model type. In another example, the training system 100 can receive a request to generate a neural network based on a neural network architecture identified by the NAS engine 125.

[0081] Exemplary Methods 2 is a flow chart of an example process 200 for hardware-aware progressive training of machine learning models. A training system, such as the training system 100 of FIG.

[0082] The training system receives a request to train a machine learning model, according to block 210. The request may include various types of data or metadata, including one or more input parameters. These input parameters may include the input parameters described herein with reference to FIG. 1 that at least partially describe one or more of the machine learning model, the machine learning task, and the computing resources available to train the machine learning model.

[0083] The training system receives a training schedule that specifies a plurality of values ​​for one or more hardware-level performance settings and one or more model-level performance settings, according to block 220. For example, the training system may generate the training schedule as described herein with reference to Figures 1 and 3A. As another example, the training system may query one or more memory devices that store a plurality of pre-generated training schedules as described herein with reference to Figures 1 and 3B.

[0084] The training system trains the machine learning model according to a training process, one or more hardware-level performance settings, and one or more model-level performance settings that are set to different ones of a plurality of values ​​in the training schedule at different times during training, according to block 230. As described herein with reference to Figure 1, the training system is configured to apply different values ​​of both the hardware-level and model-level performance settings at various points during training. The training schedule can specify those points, for example, as steps or other defined intervals, and the training schedule can further specify the rate at which values ​​are changed from one end of a range to the other.

[0085] The training system transmits the trained machine learning model to one or more computing devices according to block 240. The one or more computing devices may be, as an example, the devices that originally requested that the machine learning model be trained. In other examples, the one or more computing devices may be pre-determined to receive the trained machine learning model, for example, as part of a model deployment on a device on the edge of a network or another device of a computing platform.

[0086] 3A is a flowchart of an example process 300A for training a machine learning model to generate a training schedule for hardware-aware progressive training. For purposes of explanation, the trained machine learning model is referred to as a training schedule machine learning model.

[0087] The training system receives one or more training examples of the training schedule according to block 310. Each example training schedule may be labeled with respective data at least partially describing one or more respective input parameters used to generate the example training schedule, respective training rates, and respective model qualities of respective models trained using the example training schedule. This training data may be generated manually, automatically, or by a combination of both approaches.

[0088] For example, the training system can store metadata about the training schedule generated according to the received input parameters and record its training speed and model quality after training the model. Because the training speed and model quality are variable throughout training, the training system can store separate values ​​representing the speed and quality, respectively, at different intervals where values ​​from the training schedule are applied to the performance settings. Additionally or alternatively, the training system can calculate a function of the separate training speed and model quality values, for example as an average or sum.

[0089] Using one or more training examples, the training system trains a machine learning model, i.e., a training schedule machine learning model, to generate a training schedule from one or more input parameters, according to block 320. The input parameters are input parameters that can be received as part of a request to train a model, as described herein with reference to Figures 1-2. The training system can train the training schedule machine learning model in a variety of different ways, for example, using some form of gradient descent and backpropagation with model weight updates. A loss function or performance function for training the training schedule machine learning model can be a function of how close the training speed or model quality at various points during the training period is to the training speed or model quality of the ground truth at those same points during training.

[0090] In another example, the training system may be configured to search for a training schedule according to an optimization technique over a set of candidate training schedules. The search may be defined to identify a training schedule that has the highest model quality and training speed throughout the course of training, subject to various constraints that may be set according to the input parameters. For example, the constraints may be over a particular subset of performance settings of hardware and performance levels available for a given training process and set of computing resources used in training the model using the identified training schedule.

[0091] FIG. 3B is a flowchart of an example process for querying and applying pre-generated training schedules from one or more memory devices that store multiple training schedules according to an aspect of the disclosure.

[0092] The training system, in accordance with block 330, sends a query to one or more memory devices that store a plurality of candidate training schedules, the query including data that at least partially describes one or more of the machine learning model, the machine learning task, and / or computing resources available for training the machine learning model. As described herein with reference to FIG. 1, the training system may include a training engine configured to receive input parameters as part of a request to train a model, and to query a training schedule library in the memory device for previously generated training schedules tagged with at least some of the input parameters.

[0093] The training system receives a training schedule from a plurality of candidate training schedules in response to the query, according to block 340. For example, the received training schedule may be a training schedule having metadata that is identical or most similar to the input parameters as in the query. The input parameters may be compared to a predetermined similarity measure corresponding to one or more input parameters.

[0094] FIG. 4 is a flowchart of an example process for exploring neural architectures according to an embodiment of the disclosure.

[0095] Aspects of the present disclosure also provide a training system configured to explore a set of candidate neural network architectures for a target architecture to which hardware-aware progressive training can be applied. For example, the training system can identify a target architecture to which all or most of the hardware features for a specified set of computing resources can be applied during training at different values ​​for the tradeoff between training speed and model quality. As part of adjusting performance settings during training, the training system can incur a performance overhead over the operations performed to cause the computing resources to train the model according to the adjusted values. As another example, the training system can identify a target architecture for which model-level performance settings are adjustable with minimal performance overhead relative to other candidate architectures.

[0096] The training system explores neural architectures that benefit from continuous adjustment of hardware-level and model-level performance settings during training. For example, neural architectures that are scalable in model size, measured for example by the number of neural network layers and / or the number of nodes in each layer, or input size with it, and that can be trained on corresponding computing resources that can be scaled to accommodate increased model or input sizes, will benefit more during training, as measured for example in higher training speeds and model quality using a training schedule that varies performance settings, as described herein.

[0097] According to block 410 of process 400, the training system estimates at least the training speed and model quality of a first neural network having a first candidate neural architecture of a plurality of candidate neural architectures and trained using hardware-aware progressive learning. This estimation may be part of measuring the performance of the candidate neural architecture within a search space of neural architectures. This search space may include a wide variety of different candidate architectures that may be filtered or adjusted based on different input parameters provided. For example, if the training system receives input parameters specifying the model type to be a convolutional neural network, the training system may search a search space of neural architectures that include at least one convolutional layer.

[0098] The training system selects the first candidate neural architecture based at least on a comparison of the estimated training speed and the estimated model quality of the first neural network with the estimated training speed and the estimated model quality of each of the one or more second neural networks. Each second neural network has a respective candidate neural architecture according to block 420. The second neural networks may be trained according to hardware-aware progressive learning as described herein to identify the respective training speeds and model qualities. Additionally or alternatively, the training system may estimate the training speeds and model qualities.

[0099] Selection by the training system may be part of multiple iterations of selecting a candidate neural architecture and comparing it to the current best known architecture. The search may be augmented by using at least the training speed and model quality from the hardware-aware progressive training as indicators of the performance of different candidate models. Any of a wide variety of neural architecture search processes may be applied based at least on its training speed and model quality, such as random search over a number of iterations or until a candidate neural architecture that meets a threshold performance value is found.

[0100] Once the first candidate neural architecture is identified as the target neural architecture, the training system may proceed to train a neural network having the target neural architecture, e.g., as described herein with reference to Figures 1-2.

[0101] Aspects of the present disclosure may provide at least the following technical advantages: Generating neural networks having neural architectures selected from a NAS as described herein allows for improved utilization of hardware-aware progressive training as described herein, where the neural architectures are tunable to the computing resource environment in which they are trained, allowing increased access to those hardware features, in contrast to neural architectures that are not identified as described herein, which may be incompatible with those hardware features to accelerate the operation of the training process being performed.

[0102] Exemplary Computing Environment 5 is a block diagram of an exemplary environment 500 for implementing the training system 100. The system 100 can be implemented on one or more devices having one or more processors at one or more locations, such as a computing platform 101 having one or more server computing devices 515 and one or more memory devices 530. The user computing devices 512 and the server computing devices 515 can be communicatively coupled to the memory devices 530 via a network 560. The memory devices 530 can be a combination of volatile and non-volatile memory and can be in the same or different physical location as the computing devices 512, 515. For example, the memory devices 530 can include any type of non-transitory computer-readable medium that can store information, such as a hard drive, a solid-state drive, a tape drive, an optical storage device, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a writeable memory, and a read-only memory.

[0103] The server computing device 515 may include one or more processors 513 and a memory 514. The memory 514 may store information accessible by the processor 513, including instructions 521 executable by the processor 513. The memory 514 may also include data 523 that may be retrieved, manipulated, or stored by the processor 513. The memory 514 may be some type of non-transitory computer-readable medium capable of storing information accessible by the processor 513, such as volatile or non-volatile memory. The processor 513 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0104] The available computing resources for the platform 101 may include a processor 513, and / or one or more of a memory 514 or a memory device 530. As described herein, the computing resources for the platform 101 may be configured to implement one or more hardware features during data processing that may be enabled or modified according to one or more hardware-level performance settings. The training system 100 is configured to train a machine learning model according to an aspect of the present disclosure on the computing resources of the platform 101.

[0105] The instructions 521 may include one or more instructions that, when executed by the processor 513, cause the processor 513 to perform an action defined by the instructions. The instructions 521 may be stored in an object code format or other format that is processed directly by the processor 513, including interpretable scripts or collections that are interpreted on demand or independent of pre-compiled source code modules. The instructions 521 may include instructions for implementing the training system 100 according to aspects of the disclosure. The training system 100 may be executed using the processor 513 and / or using other processors located remotely from the server computing device 515.

[0106] Data 523 may be retrieved, stored or modified by processor 513 according to instructions 521. Data 523 may be stored in a computer register, a relational or non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, Proto, or XML document. Data 523 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, data 523 may include information sufficient to identify related information or information used to calculate related data, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memories, including other network locations.

[0107] The user computing device 512 may also be configured similarly to the server computing device 515, with one or more processors 516, memory 517, instructions 518, and data 519. The user computing device 512 may also include a user output 526 and a user input 524. The user input 524 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0108] The server computing device 515 may be configured to transmit data to the user computing device 512, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 526. The user output 526 may also be used to display an interface between the user computing device 512 and the server computing device 515. The user output 526 may alternatively or additionally include one or more speakers, transducers or other audio output, a haptic interface, or other tactile feedback that provides non-visual and non-audible information to a user of the computing device 512.

[0109] Although FIG. 5 illustrates the processors 513, 516 and memories 514, 517 as being present within the computing devices 515, 512, the components described herein, including the processors 513, 516 and memories 514, 517, may include multiple processors and memories that are in different physical locations and can operate not within the same computing device. For example, some of the instructions 521, 518 and data 523, 519 may be stored on a removable SD card and others in a read-only computer chip. Some or all of the instructions and data may be stored in a location that is physically remote from the processors 513, 516 but still accessible. Similarly, the processors 513, 516 may include a collection of processors capable of performing simultaneous and / or sequential operations. The computing devices 515, 512 may each include one or more internal clocks that provide timing information usable for timing operations and programs executed by the computing devices 515, 512.

[0110] The server computing device 515 may be configured to receive requests to process data from the user computing device 512. For example, the platform 101 may be configured to provide a wide variety of services to users via various user interfaces and / or APIs that present platform services. One or more services may be a machine learning framework or set of tools for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 512 may receive and transmit data specifying target computing resources to be allocated to train and deploy a neural network to perform a particular machine learning task.

[0111] For example, the server computing device 515 may be configured to receive a request that specifies, for example, a set of training data, a type of model to be trained, such as a deep neural network, a recurrent neural network, and a convolutional neural network, and a type of machine learning task that the model is trained to perform. The request can, optionally, specify more or fewer parameters, as described herein.

[0112] The devices 512, 515 may be capable of direct and indirect communication over the network 560. The devices 515, 512 may set up listening sockets that may accept connection initiations for sending and receiving information. The network 560 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using one or more company-proprietary communication protocols. The network 560 may support a wide variety of short-range and long-range connections. The short-range and long-range connections may be achieved over different bandwidths, such as 2.402 GHz to 2.480 GHz, 2.4 GHz, and 5 GHz, or using a wide variety of communication standards, such as standards for wireless broadband communication. The network 560 may additionally or alternatively support wired connections between the devices 512, 515, including through various types of Ethernet connections.

[0113] It should be understood that aspects of the present disclosure can be implemented according to a wide variety of different configurations and qualities of computing devices, including paradigms for serial or parallel processing, or over a distributed network of multiple devices. In some implementations, aspects of the present disclosure may be performed on a single device or any combination of devices.

[0114] Example Machine Learning Tasks As described herein, aspects of the present disclosure provide for hardware-aware progressive training of machine learning models that perform respective machine learning tasks, examples of which are set forth below.

[0115] As an example, the input to the machine learning model to be trained may have the form of an image or video. The machine learning model can be trained to extract, identify, and generate features as part of processing a given input, for example as part of a computer vision task. A machine learning model trained to perform this type of machine learning task may be trained to generate an output classification from a set of different possible classifications. Additionally or alternatively, the machine learning model may be trained to output a score corresponding to an estimated probability that an identified object in an image or video belongs to a particular class.

[0116] As another example, the input to the machine learning model may be a data file corresponding to a particular format, such as an HTML or XML file, a word processing document, or formatted metadata obtained from other types of data, such as metadata for an image file. A machine learning task in this context may be to classify, score, or otherwise predict some characteristic about the received input. For example, a machine learning model may be trained to predict the probability that a received input contains text related to a particular subject. Also, as part of performing a particular task, a machine learning model may be trained to generate text predictions, for example, as part of a tool for auto-completion of text in a document as the document is being composed. A machine learning model may also be trained to predict the translation of text in an input document into a target language, for example, as a message is being composed.

[0117] Other types of input documents may be data related to the characteristics of a network of interconnected devices. These input documents may include activity logs as well as records regarding access rights to different computing devices to access different sources of potentially sensitive data. Machine learning models may be trained to process these and other types of documents to predict ongoing and future security breaches to the network. For example, machine learning models may be trained to predict intrusions into a network by malicious actors.

[0118] As another example, the input to the machine learning model may be audio input, including streamed audio, pre-recorded audio, and audio as part of a video or other source or medium. Machine learning tasks in the audio context may include speech recognition, including separating audio from other identified audio sources and / or enhancing characteristics of identified audio for audibility. Machine learning models may be trained to predict accurate translations of input audio into a target language, for example in real time, as part of a translation tool.

[0119] In addition to data inputs, including various types of data described herein, machine learning models can also be trained to process features corresponding to a given input. A feature is a value, such as a numerical or categorical value, that is related to some characteristic of the input. For example, in the context of an image, an image feature may relate to the RGB values ​​for each pixel in the image. A machine learning task in the context of an image / video may be to classify the content of an image or video, for example, for the presence of different people, places, or things. Machine learning models can be trained to extract and select relevant features for processing to generate an output for a given input, and further trained to generate novel features based on learned relationships between various characteristics of the input data.

[0120] Aspects of the disclosure may be embodied as one or more computer programs or combinations of one or more of these computer programs, in digital circuitry, in a computer-readable storage medium, which may be non-transitory, for example, as one or more instructions executable by one or more computing devices and stored on one or more tangible memory devices.

[0121] In this specification, the phrase "configured to" is used in different contexts in relation to a computer system, hardware, or a part of a computer program, engine, or module. When a system is described as being configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed thereon that, when operated, causes the system to perform one or more operations. When any hardware is described as being configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive inputs, act on the inputs, and generate outputs corresponding to the one or more operations. When a computer program, engine, or module is described as being configured to perform one or more operations, this means that the computer program, engine, or module includes one or more program instructions that, when executed by one or more computing devices, such as one or more processors, cause the one or more computing devices to perform the one or more operations.

[0122] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it should be understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring that the components be separated. The described components, modules, programs, and engines may be integrated as a single system or may be part of multiple systems.

[0123] Unless otherwise stated, the above-described alternative embodiment examples are not mutually exclusive and may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features described above can be utilized without departing from the subject matter defined by the claims, the above description of those examples should be interpreted as examples, not limitations, of the subject matter defined by the claims. In addition, the presentation of examples described herein, as well as phrases expressed as "such as," "including," and the like, should not be interpreted as limiting the subject matter of the claims to any particular example, but rather, the examples are intended to illustrate only one of many possible embodiments. Moreover, the same reference numbers in different drawings may identify the same or similar elements.

Claims

1. One or more processors, Receiving a request to train a machine learning model; receiving a training schedule that specifies a plurality of values ​​for one or more hardware-level performance settings and one or more model-level performance settings; training the machine learning model according to a training process, one or more hardware-level performance settings, and one or more model-level performance settings that are set to different ones of the plurality of values ​​of the training schedule at different times during training; and in response to receiving the request, transmitting the trained machine learning model to one or more computing devices.

2. the one or more model-level performance settings: An input data size for input data to the machine learning model; and One or more model hyperparameters that specify the size or shape of the machine learning model; and and one or more training process hyperparameters that modify the training process performed by the one or more processors to train the machine learning model.

3. 2. The system of claim 1, wherein the one or more hardware-level performance settings include settings for regulating intra-processor or inter-processor data communications of the one or more processors.

4. the one or more processors include a plurality of processors logically or physically grouped into a plurality of groups; the one or more hardware level performance settings include settings for speed of data communication between processors in different groups; The system of claim 3.

5. 4. The system of claim 3, wherein the one or more hardware-level performance settings include settings for adjusting numerical precision of operations performed by the one or more processors while training the machine learning model according to the training process.

6. 4. The system of claim 3, wherein the one or more hardware-level performance settings include settings for enabling or disabling hardware parallelism among the one or more processors while training the machine learning model according to the training process.

7. During training of the machine learning model, the one or more processors: setting the one or more hardware-level performance settings and the one or more model-level performance settings to a first value of the plurality of values ​​of the training schedule; 2. The system of claim 1, further configured to: adjust, at a first time point after initiation of the training of the machine learning model, the one or more hardware-level performance settings and the one or more model-level performance settings to a second value of the plurality of values ​​that is different from the first value.

8. Upon receiving the training schedule, the one or more processors are further configured to generate a training schedule using a training schedule machine learning model, the training schedule machine learning model comprising: to generate a training schedule from one or more input parameters that at least partially describe one or more of the machine learning model, the machine learning task, and computing resources available for training the machine learning model; 2. The system of claim 1, wherein each example training schedule is trained using one or more training examples of training schedules that are labeled with respective data at least partially describing one or more respective input parameters used to generate the example training schedule, a training rate, and a model quality of the training process and a respective machine learning model trained according to the example training schedule.

9. 2. The system of claim 1 , wherein the machine learning model is a neural network having a neural architecture selected from a plurality of candidate neural architectures, the selection of the neural architecture being based at least in part on a comparison of estimated respective training speeds of neural networks to respective model qualities, and the neural network is trained according to the training process and a respective training schedule, the neural network having each candidate neural architecture of the plurality of candidate neural architectures.

10. Upon receiving the training schedule, the one or more processors: sending a query including data at least partially describing one or more of the machine learning model, the machine learning task, and computing resources available for training the machine learning model to one or more memory devices storing a plurality of candidate training schedules; The system of claim 1 , further configured to: receive the training schedule from the plurality of candidate training schedules in response to the query.

11. 1. A method comprising: receiving, by one or more processors, a request to train a machine learning model, the one or more processors configured to train the machine learning model according to one or more hardware-level performance settings and one or more model-level performance settings; The method further comprises: receiving, by the one or more processors, a training schedule that specifies a plurality of values ​​for the one or more hardware-level performance settings and the one or more model-level performance settings; training, by the one or more processors, the machine learning model according to a training process and the one or more hardware-level performance settings and the one or more model-level performance settings that are set to different ones of the plurality of values ​​of the training schedule at different times during training; transmitting, by the one or more processors, the trained machine learning model to one or more computing devices in response to receiving the request; A method comprising:

12. the one or more model-level performance settings: An input data size for input data to the machine learning model; and One or more model hyperparameters that specify the size or shape of the machine learning model; and and one or more training process hyperparameters that modify the training process performed by the one or more processors to train the machine learning model.

13. 12. The method of claim 11, wherein the one or more hardware-level performance settings include settings for regulating intra-processor or inter-processor data communications of the one or more processors.

14. the one or more processors include a plurality of processors logically or physically grouped into a plurality of groups; the one or more hardware level performance settings include settings for speed of data communication between processors in different groups; The method of claim 13.

15. 14. The method of claim 13, wherein the one or more hardware-level performance settings include settings for enabling or disabling hardware parallelism among the one or more processors while training the machine learning model according to the training process.

16. Receiving the training schedule includes generating, by the one or more processors, a training schedule using a training schedule machine learning model, the training schedule machine learning model comprising: to generate a training schedule from one or more input parameters that at least partially describe one or more of the machine learning model, the machine learning task, and computing resources available for training the machine learning model; 12. The method of claim 11, wherein each example training schedule is trained using one or more training examples of training schedules that are labeled with respective data at least partially describing one or more respective input parameters used to generate the example training schedule, a training rate, and a model quality of the training process and a respective machine learning model trained according to the example training schedule.

17. 12. The method of claim 11, wherein the machine learning model is a neural network having a neural architecture selected from a plurality of candidate neural architectures, the selection of the neural architecture being based at least in part on a comparison of estimated respective training speeds of neural networks with respective model qualities, and the neural network is trained according to the training process and a respective training schedule, the neural network having each candidate neural architecture of the plurality of candidate neural architectures.

18. receiving the training schedule, sending, by the one or more processors, a query including data at least partially describing one or more of the machine learning model, the machine learning task, and computing resources available for training the machine learning model to one or more memory devices storing a plurality of candidate training schedules; and receiving, by the one or more processors, the training schedule from the plurality of candidate training schedules in response to the query.

19. Training the machine learning model setting, by the one or more processors, the one or more hardware-level performance settings and the one or more model-level performance settings to a first value of the plurality of values ​​of the training schedule; 12. The method of claim 11, further comprising: at a first time point after initiation of the training of the machine learning model, adjusting, by the one or more processors, the one or more hardware-level performance settings and the one or more model-level performance settings to a second value of the plurality of values ​​that is different from the first value.

20. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more processors configured to train a machine learning model according to one or more hardware-level performance settings and one or more model-level performance settings, cause the one or more processors to: Receiving a request to train a first machine learning model; receiving a training schedule that specifies a plurality of values ​​for the one or more hardware-level performance settings and the one or more model-level performance settings; training the first machine learning model according to a training process and the one or more hardware-level performance settings and the one or more model-level performance settings that are set to different ones of the plurality of values ​​of the training schedule at different times during training; and in response to receiving the request, transmitting the trained first machine learning model to one or more computing devices.

Images