Systems and methods for improved optimization of machine-learned models

Abstract

Generally, the present disclosure is directed to systems and methods for improved optimization of machine-learned models. In particular, the present disclosure provides stochastic optimization algorithms that are both faster than widely used algorithms for fixed amounts of computation, and are also able to scale up substantially better as more computational resources become available. The stochastic optimization algorithms can be used with large batch sizes. As an example, in some implementations, the systems and methods of the present disclosure can implicitly compute the inverse hessian of each mini-batch of training data to produce descent directions.

Description

Technical field [0001] The present disclosure relates generally to machine learning. More specifically, the present disclosure relates to systems and methods for improving the optimization of machine learning models, such as, for example, deep neural networks. Background technique [0002] The progress of machine learning (eg, deep learning) is slowed by the number of days or weeks required to train large models. Natural solutions that use more hardware are limited by diminishing returns and lead to inefficient use of additional resources. [0003] The current state of training deep neural networks is that simple mini-batch optimizers (such as stochastic gradient descent (SGD) and momentum optimizers) and diagonal natural gradient methods are most used in practice. As the availability of distributed computing increases, the total real time (wall-time) for training large models has become a substantial bottleneck, and methods to reduce the total real time without sacrificing model...

AI Technical Summary

Problems solved by technology

In practice, however, reducing variance by increasing the batch size usually results in a speedup sublinear to the batch size, as well as reduced generalization

Images