Systems and methods for evaluating a loss function or a gradient of a loss function via dual decomposition

Abstract

Systems and methods for evaluating a loss function or a gradient of the loss function. In one example embodiment, a computer-implemented method includes partitioning a weight matrix into a plurality of blocks. The method includes identifying a first set of labels for each of the plurality of blocks with a score greater than a first threshold value. The method includes constructing a sparse approximation of a scoring vector for each of the plurality of blocks based on the first set of labels. The method includes determining a correction value for each sparse approximation of the scoring vector.The method includes determining an approximation of a loss or a gradient of a loss associated with the scoring function based on each sparse approximation of the scoring vector and the correction value associated with the sparse approximation of the scoring vector.

Description

technical field [0001] The present disclosure generally relates to evaluating loss functions or gradients of loss functions. More specifically, the present disclosure relates to computer systems and methods for efficiently evaluating loss functions and / or their gradients for problems with large output spaces via dual decomposition of loss functions. Background technique [0002] Large output spaces are ubiquitous in several machine learning problems today. Such machine learning problems can include, for example, extreme multiclass or multilabel classification problems with many classes, language modeling with large vocabularies, or metric learning with large pairwise distance constraints. In all such problems, the main bottleneck in training the model is the evaluation of the loss function and its gradient. Loss functions for such problems typically need to enumerate all possible outputs and may require linear running time in the number of outputs used for evaluation. Thi...

AI Technical Summary

Problems solved by technology

In all such problems, the main bottleneck in training the model is the evaluation of the loss function and its gradient

Loss functions for this kind of problem typically need to enumerate all possible outputs and may require linear running time in the number of outputs used for evaluation

This can be a significant bottleneck in iterative methods such as gradient descent for training said model, since each step can be operation-intensive

Images