Regularization of recurrent machine-learned architectures

Abstract

A modeling system trains a recurrent machine-learned model by determining a latent distribution and a prior distribution for a latent state. The parameters of the model are trained based on a divergence loss that penalizes significant deviations between the latent distribution the prior distribution. The latent distribution for a current observation is a distribution for the latent state given a value of the current observation and the latent state for the previous observation. The prior distribution for a current observation is a distribution for the latent state given the latent state for the previous observation independent of the value of the current observation, and represents a belief about the latent state before input evidence is taken into account.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS [0002] This application claims the benefit of and priority to US Provisional Application No. 62 / 778,277, filed on December 11, 2018, the entire contents of which are incorporated herein by reference. Background technique [0003] The present invention relates generally to recursive machine learning models, and more particularly to regularization of recursive machine learning models. [0004] Modeling systems typically use recursive machine learning models such as recurrent neural networks (RNNs) or long short-term memory models (LSTMs) to generate sequential predictions. Recursive machine learning models are configured to generate subsequent predictions based on latent states for the current prediction, sometimes incorporating an initial sequence of actual inputs. The current latent state represents contextual information about predictions generated up to the current prediction, and is generated based on the latent state for t...

AI Technical Summary

Problems solved by technology

However, as the complexity and size of the model increases, the parameters are often difficult to train, which can lead to overfitting the model to the dataset or losing contextual information that might be useful for generating predictions

Although regularization methods have been applied to reduce model complexity, training recurrent machine learning models to preserve important contextual information and control sensitivity to continuous input data remains a challenging problem

Images