Systems and methods for regularizing neural networks

Abstract

The present disclosure relates generally to machine learning. More specifically, the present disclosure relates to systems and methods for regularizing a neural network by decorrelating neurons or other parameters of the neural network during training of the neural network, thereby promoting innovation of these parameters with respect to each other.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 660,617, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to machine learning. More specifically, the present disclosure relates to systems and methods for regularizing a neural network by decorrelation of neurons and / or other components or parameters of the neural network during training of the neural network. Background Art

[0004] Neural networks have provided a huge breakthrough in big data prediction, as well as improving the predictive accuracy and power of machine learning. However, due to the very large number of parameters that neural networks attempt to learn in order to achieve such performance, they can significantly overfit the training data, potentially causing poor generalization to data not observed during training.

[0005] Neural networks also suffer from potential instabilities and non-reproducibility in training. With respect to non-reproducibility, for example, it is possible that independently training the same network structure twice on the same data can still result in two very different neural networks, and potentially produce different predictions on data not observed during training. This can occur in highly parallelized and distributed training systems that do not guarantee access to the training examples in the same order. If the network then controls which subsequent training examples are seen, then the network can evolve very differently with covariate shifts. These deviations can arise from different random initializations, parallelization, and / or ordering of the training examples.

[0006] It has been shown that there is a lot of redundancy in neural networks. Redundancy can lead to the overfitting and other problems mentioned above. In particular, the network can be well overparameterized, with some parameters being predictable by other parameters. This particularly leads to overfitting with limited training examples, because the learning (regret) penalty is paid for having more parameters than necessary, and convergence speed and generalization ability are slowed down.

[0007] As a result, neural networks suffer from overfitting, potentially poor generalization to unseen data, instability, and non-reproducibility when retrained on the same data but with potentially different initialization, parallelization, and order of training examples. These issues make it difficult, and sometimes impossible, to deploy large-scale systems using neural networks, resulting in huge losses due to the inability to exploit the significant advantages and strengths of neural networks in providing more accurate predictions. Current solutions require repetition in training and deployment, causing over-utilization of CPU and memory, and hindering the ability to deploy more systems due to lack of resources. Summary of the invention

[0008] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0009] One example aspect of the present disclosure is directed to a computer-implemented method for training a neural network. The method includes obtaining, by one or more computing devices, data describing the neural network. The neural network includes multiple layers of neurons. The method includes training the neural network by backpropagating a loss function through the neural network by the one or more computing devices. The loss function describes the performance of the neural network relative to a set of training examples. Backpropagating the loss function through the neural network by the one or more computing devices includes, for each of one or more of a plurality of layers of the neural network: determining, by the one or more computing devices, a gradient of the loss function relative to neurons included in the layer. For at least the layer, the loss function includes an innovation loss term that provides a loss value for each of one or more neurons or units included in the layer based on the ability of one or more other neurons in the layer to predict a value of such a neuron. Backpropagating the loss function through the neural network by the one or more computing devices includes, for each of one or more of a plurality of layers of the neural network: modifying, by the one or more computing devices, neurons included in the layer based at least in part on the gradient of the loss function including the innovation loss term to decorrelate the neurons included in the layer.

[0010] Another example aspect of the present disclosure is directed to a computer-implemented method for training a neural network. The method includes obtaining, by one or more computing devices, data describing the neural network. The neural network includes multiple layers of neurons. The method includes determining, by the one or more computing devices, a benefit score for each of one or more neurons included in one or more layers of a plurality of layers based at least in part on an error in predicting a value of such a neuron by one or more other neurons included in the same layer. The method includes modifying, by the one or more computing devices, a weight associated with at least one of the one or more neurons based at least in part on the benefit score determined for the at least one neuron.

[0011] In some implementations, modifying, by the one or more computing devices, a weight associated with at least one of the one or more neurons based at least in part on the benefit score includes regularizing, by the one or more computing devices, the weight to zero when the error is less than a threshold.

[0012] In some implementations, the method further includes randomly reinitializing one or more links associated with the neuron after regularizing the weights to zero by the one or more computing devices.

[0013] In some embodiments, for each of one or more neurons, the error in predicting the value of the neuron by one or more other neurons included in the same layer includes an error in estimating a full activation weight of such neuron based on one or more regularized activation weights of one or more other neurons included in the same layer.

[0014] In some embodiments, modifying, by one or more computing devices, a weight associated with at least one of the one or more neurons includes modifying, by the one or more computing devices, a weight associated with at least one of the one or more neurons to a regularized value that is equal to the dot product of a sigmoid function or a hyperbolic tangent function applied to the benefit score of such neuron and the value of such neuron.

[0015] In some embodiments, for each of the one or more neurons, the benefit score comprises: the cumulative squared error over all batches and the current batch; the average per-batch per-example error; or a sliding or decaying exponential window error over a set of recent batches.

[0016] Another example aspect of the present disclosure is directed to a computer-implemented method for training a neural network. The method includes obtaining, by one or more computing devices, data describing the neural network. The neural network includes a plurality of neurons connected by a plurality of links, respectively. The method includes training the neural network by backpropagating a loss function through the neural network by the one or more computing devices. The loss function describes the performance of the neural network relative to a set of training examples. Backpropagating the loss function through the neural network by the one or more computing devices includes, for each of one or more neurons, links, or biases of the neural network: determining, by the one or more computing devices, a gradient of the loss function relative to the one or more neurons, links, or biases of the neural network. For at least one or more neurons, links, or biases of the neural network, the loss function includes an innovation loss term, the innovation loss term providing a loss value for each of the one or more neurons, links, or biases based on the ability of one or more other neurons, links, or biases to predict the value of such neurons, links, or biases. Backpropagating the loss function through the neural network by the one or more computing devices includes modifying, by the one or more computing devices, the one or more neurons, links, or biases of the neural network based at least in part on the gradient of the loss function including the innovation loss term to decorrelate the one or more neurons, links, or biases of the neural network.

[0017] In some implementations, one or more other neurons, links, or biases of the neural network may be included in the same layer of the neural network as the one or more neurons, links, or biases that they are attempting to predict.

[0018] In some implementations, at least one of one or more neurons, links, or biases of a neural network is included in a different layer of the neural network than the one or more neurons, links, or biases that such at least one neuron, link, or bias attempts to predict.

[0019] In some implementations, for each training batch from the layer, at least one of one or more neurons, links, or biases of the neural network are randomly selected.

[0020] In some implementations, an example is to mini-batch training batches to compute the innovation decorrelation loss.

[0021] In some embodiments, input layer embeddings are randomly or deterministically initialized to non-zero values ​​that promote decorrelation of layers and towers connected to them.

[0022] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0023] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A detailed discussion of embodiments for one of ordinary skill in the art is set forth in the specification, which refers to the accompanying drawings, in which:

[0025] Figure 1A A block diagram of an example computing system for training a neural network is depicted according to an example embodiment of the present disclosure.

[0026] Figure 1B A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0027] Figure 1C A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0028] Figure 2A-2C An example neural network according to an example embodiment of the present disclosure is depicted.

[0029] Reference numbers repeated across multiple drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION

[0030] Overview and Introduction

[0031] Example aspects of the present disclosure are directed to systems and methods for regularizing a neural network by decorrelating neurons or other parameters or components of the neural network during training of the neural network. Specifically, the present disclosure provides a new scheme for decorrelating correlated neurons in a layer (e.g., an output layer or a hidden layer) in a neural network. In some embodiments, the systems and methods of the present disclosure can identify layer neurons that provide less increased prediction value in a layer. Neurons that can be well predicted by other neurons in the layer add less predicted value to the network. Such neurons can be forced to move in a direction that increases their increased value. This can be done by injecting an additional loss into the layers of the neural network so that the loss gradients on these neurons push the neurons away from predicting each other.

[0032] Thus, in some embodiments, the schemes described herein are based not only on pairs of neurons in a decorrelated layer, but also on innovations that ensure that each neuron in a layer of a neural network is pushed toward information that exceeds that of other neurons in the layer. As an example, this can be done by ensuring that the minimum mean square error of the value of a given neuron predicted by some or all other neurons in the layer is maximized. Variations on these example schemes can be implemented by applying additional per-layer regularizer losses that enforce such constraints. As an example, constraints can be applied in the form of L2 regularization and in the form of Lagrangian dual variable optimization. Thus, the present disclosure provides multiple strategies for measuring the innovation of neurons and applying related losses to maximize them. Neurons may, but do not necessarily, include nonlinear activations. For example, a neuron may include and implement a linear or nonlinear activation function to provide a neuron output based on an input value provided to the neuron as an input.

[0033] In addition, the present disclosure also demonstrates that in addition to the example regularization types and Lagrangian dual variable techniques, additional techniques similar to Minimum Description Length (MDL) regularization can be implemented to instill neuron innovation. Specifically, the innovation of a neuron (e.g., the additional dimension it brings beyond the value predicted by other neurons for it) can be used to enlarge or reduce the weight of the neuron. In one example, if a neuron is innovative, its weight will not be reduced (or can be enlarged), while if it is not innovative, its weight will be reduced, allowing other neurons to take over and, in some cases, forcing the neuron to be regularized eventually.

[0034] Therefore, the present disclosure provides systems and methods that can be used to regularize neural networks, thereby improving network reproducibility, generalization, and stability. As an example technical effect and benefit, resource savings can be achieved by eliminating the need for ensemble training, thereby saving RAM and CPU usage in deployment, ensuring faster training, and also potentially improving prediction accuracy. As another example technical effect and benefit, neural networks trained according to the systems and methods of the present disclosure exhibit reduced prediction bias and overfitting, resulting in better generalization to unseen data.

[0035] More specifically, consider a layer of a neural network (e.g., a hidden layer of a neural network). It is assumed that a given layer holds all the information learned by the network because it acts as a bottleneck for information to go from input to output. Due to over-parameterization, some activated neurons may be correlated with other neurons. In other words, a subset of activated neurons in a layer may be used to predict other subsets. If this is the case, spending learning loss on predictable neurons may result in slower convergence. In turn, this may lead to overfitting and poor generalization, as well as non-reproducible and unstable networks. Specifically, such redundancy may increase the diversity of solutions, making the network less reproducible. This problem can be addressed from two directions: decorrelation; or forced correlation. The present disclosure provides systems and methods that employ decorrelation schemes.

[0036] Some techniques add a layer loss applied to the activations of all neurons in a layer in a neural network, where the layer loss penalizes layers for pairs of neurons with high covariance. Thus, the layer loss operates on a pair-wise basis. Backpropagation of this loss reduces the covariance between such pairs. The overall layer objective is set to the sum of the squares of all covariances (or the square of the Frobenius norm of the covariance matrix) omitting the elements on the diagonal. An estimate of the covariance is obtained over a batch of training examples. Backpropagation applies the gradient on this additional loss. While this scheme has merit, it does not fully solve the problem for two reasons:

[0037] First, covariance may not be the right measure of how well a given neuron explains another neuron, because it omits the mean of the two neurons. Since there is typically no other knowledge in the layer for predicting the mean, some relationships between neurons in the layer may be lost by looking only at the covariance. For example, two neurons can have (very) equal expected values ​​- one with a large variance and the other with a small variance - but in both cases the variance is much smaller than the mean. Furthermore, the covariance between neurons is very small. One neuron may still predict most of the energy of the other neuron, but the correlation based on covariance would mean that this is not the case. The covariance constraint does not impose any constraints on the mean, so the means can vary greatly. For example, if the sum of the means of two neurons is 1, then their means can be any two values ​​that sum to 1. When only the covariance constraint is imposed, the possibility of such different multiple solutions is not eliminated. Furthermore, since activations in neural networks are nonlinear, it cannot be expected that an activated neuron will have a mean of 0 in a batch. In fact, for ReLU, this is impossible unless the neuron is not activated for any example in the batch. Only the covariance constraint implicitly assumes 0 mean.

[0038] Second, covariance or correlation only gives the relationship between two elements. Penalizing the sum of the squares of the covariance omits the relationships between other neurons. For example, if there are three linearly dependent neurons, to predict the third neuron, both the first and second neurons will be penalized without considering their relationship, rather than splitting the penalty between them. This can potentially exaggerate the penalty for some neurons while underestimating the penalty for others.

[0039] Aspects of the present disclosure address both of these issues. Specifically, in some embodiments, instead of using covariance, the systems and methods of the present disclosure use correlation because there is no estimator for the mean in the layer. One goal, then, is to essentially force the innovation of each neuron in the layer (e.g., the activation of each neuron) to be maximized relative to the information from other neurons in the layer. For example, consider that the information in a given layer with n neurons is given by points in n-dimensional Euclidean space, and squared error is an effective metric.

[0040] In order to maximize the innovation of an activated neuron (e.g., where the activation can also be linear) relative to its peers, the systems and methods of the present disclosure can try to predict the activated neuron based on other activated neurons. The best predictor in terms of minimizing the mean squared error on an example training batch is the conditional mean estimator. However, due to the difficulty in calculating it, in some embodiments, the present disclosure may consider the best linear predictor instead. Because there is no other information in the layer to optimize the loss outside the layer, in some embodiments, a homogeneous linear estimator can be used. In other words, since the final target only "sees" the activated neurons of the layer, nothing else should be used to determine whether the neuron is innovative relative to all other information units that propagate to the final target.

[0041] The first example strategy provided by the present disclosure is to use a homogeneous linear minimum mean square error (MMSE) estimator to predict each neuron in the layer. Given that the estimates of statistical correlation and mean are accurate, the estimator gives a good estimate of each activation based on other activations (given computationally tractable constraints). According to aspects of the present disclosure, the goal in some embodiments is to make the error of the estimator as large as possible; the larger the error, the better the innovation of the neuron. This solution requires multiple inversions of large matrices. However, if this is intractable, the error minimization can be replaced by a potentially suboptimal regression-based solution proposed as a second example strategy.

[0042] The third example strategy provided by the present disclosure regards the n neurons of the k examples in the training batch as an n-dimensional vector of neuron values ​​for the k examples. The j-th vector represents the state of the j-th neuron on the k examples of the training batch. Each neuron vector can be subtracted from its projection on the other n-1 vectors to produce a prediction error of the vector predicted by the other n-1 vectors. In other words, an error vector orthogonal to the other vectors can be generated. The norm of the vector is the prediction error of the batch. Similarly, the square of the norm can be maximized by, for example, back-propagating the gradient of the inverted value of the square of the norm. Alternatively, the norm of the projection of the vector to other vectors can be minimized.

[0043] In the following sections, the disclosure describes the details of the strategies presented above. First, the disclosure demonstrates an example of how to apply decorrelation of neuron pairs. Next, the disclosure demonstrates three example strategies for calculating maximum innovation-based mean square error (MSE)-based objectives for network layers. Then, the disclosure demonstrates two example schemes for applying this objective to layers of the network: a class regularization scheme for enforcing a maximum MMSE (innovation) constraint on a batch, and a variant using Lagrangian-constrained optimization. Next, it is described how to utilize a squared error innovation estimator to apply an online benefit-based (class MDL regularization) scheme on a layer. Finally, some practical complexity issues are addressed, and several ideas on how to apply this technology in a practical system are presented.

[0044] Therefore, this paper provides a scheme based on the innovation or mean squared error of predicting a layer neuron by other neurons in the layer. An underlying idea is that if a neuron can be predicted from other neurons in the layer with a small margin of error, then the neuron is likely to be redundant. A method is proposed to learn about relevant neurons by pushing the gradient of a neuron to increase its innovation over other neurons. Alternatively, it is proposed to regularize a neuron if it is not innovative using a scheme based on a reward score, where the score is closely related to the innovation of the neuron.

[0045] Although the part discussed herein focuses on neurons in hidden layers, the innovative regularization techniques described herein can be performed in any layer. It may be sufficient to perform the technique on the top layer and / or the bottom layer, but there is no restriction not to include more layers except for the increased computational cost. For example, the computational cost of performing the technique may make it desirable to implement it only on the top layer to reduce complexity. However, applying it to other layers may also be beneficial. Therefore, unless otherwise specified, any reference to a layer in this disclosure includes input layers, hidden layers, and output layers within its scope (e.g., layers including layers composed of single neurons, each of which is toward a component (e.g., a tower) of a complete network).

[0046] In addition, although the discussion focuses on neural networks, the schemes herein (e.g., schemes utilizing MSE decorrelation) are more general than just for neural networks. They can also be used to find correlations between features in linear models. Specifically, in some embodiments, they can be applied only to a subset of features present in a batch update of a linear model, rather than to the entire feature set.

[0047] In addition, although the scheme is usually described on the activated neurons of the layer, it can be applied to pre-activated neurons, links, biases, or other parameters or aspects of the neural network. Since it is desired to measure the innovation of the layer at the point closest to the final output, and because the impact on the final goal is the ultimate concern, it is reasonable to apply constraints after activation. However, if many neurons are not activated, it may not be very beneficial, and it is also possible to apply a pre-activation scheme. Similarly, in addition to or instead of nonlinear activation, the scheme described herein can be applied to linear activation. In addition, the scheme can also be applied to other components and links or any parameters in the network. Specifically, in some embodiments, it can be applied to different neurons across layers. However, the scheme mainly considered in the present disclosure is to apply it within a complete layer, because the layer should capture all information from the layers and neurons below it and relay this (compressed) information to the target.

[0048] In some embodiments, the proposed method can be specifically applied to the layer closest to the output in the overall model structure. An example of such a constellation includes several independent networks, each of which has a neuron at the output. An additional innovation loss can be applied to the layer of output neurons of the components that combine the whole, but force each independent overall component to be different from each other and innovate on other networks.

[0049] In some examples, where the input is represented by features encoded into embeddings, innovation of neurons or network components can be encouraged by enforcing different random initializations on the input embeddings to replicate the overall components of the network.

[0050] The systems and methods of the present disclosure provide multiple technical effects and benefits. As an example technical effect, the systems and methods of the present disclosure enable the generation of neural networks or other machine learning models that exhibit less overfitting and better generalization, stability, and reproducibility for unseen data. Therefore, the systems and methods of the present disclosure enable the generation of machine learning models that exhibit superior performance relative to the prior art. In addition, the systems and methods of the present disclosure can enable faster convergence of neural networks because, in some embodiments, neurons are pushed to innovate on their peers in each round of training.

[0051] As another example technical effect, the system and method of the present disclosure enable improved utilization of processing and memory resources. More specifically, some prior art techniques that attempt to solve overfitting, poor generalization, instability and / or non-reproducibility rely on the performance of the overall solution. These overall solutions require duplication (which can be large-scale) of multiple networks in training and deployment. This training and deployment of repeated or redundant networks greatly increases the consumption of both processing and memory resources. Therefore, by providing a solution that solves overfitting, poor generalization, instability and / or non-reproducibility without relying on or less relying on the performance of the overall solution, the system and method of the present disclosure releases a large amount of processing and memory resources, otherwise the large amount of processing and memory resources will be dedicated to such an overall solution. Alternatively, if a similar overall structure is maintained, the innovation encouragement method can rely on the overall to promote better reproducibility.

[0052] Decorrelation

[0053] Consider a layer of a neural network with n neurons. The network is trained on batches of k examples at a time. For some examples, let Z represent an n-dimensional random column vector representing the activations of the neurons at the end of the layer.

[0054] The correlation matrix of the layer is given by

[0055] C z =E[ZZ T ]

[0056] The superscript T represents the transpose operator. We can use the correlation matrix C in the current training batch z Empirical estimates of elements i, j

[0057]

[0058] In order for these estimates to be valid, we should have sufficiently more examples in the batch than the number of neurons in the layer, i.e., k>>n. Throughout this disclosure, the following notation conventions will be used. The subscripts denote the index j of an activated neuron (or layer neuron), where 1≤j≤n. Multiple (two) subscripts i,j represent the relationship between two such neurons. The superscript l, where 1≤l≤k, represents the index of an example in the training batch.

[0059] According to aspects of the present disclosure, a constraint may be imposed on the sum of squares of correlations in a layer by adding a layer loss, such as the following example layer loss:

[0060]

[0061] where ||C z || F Represents C zThe Frobenius norm of , and diag denotes the diagonal elements of the matrix. Note that this loss can be normalized by the number of neurons n. Adding the regularizer weight λ and propagating the gradient of the loss with the overall objective superimposed down from the activations of neurons in a given layer can enforce a regularization that will reduce the magnitude of the correlation between pairs of activations in the layer. This addresses the fact that similar schemes using covariance cannot address the correlation of the means of the activations.

[0062] The above scheme can be extended so that the correlations are continuously updated throughout training, rather than being recomputed for each batch. One implementation of this can use a form of exponential decay windowing for the batches, which multiplies the previously calculated values ​​by a small positive constant (e.g., less than 1), and adds the calculated correlations for the current batch to another constant less than 1.

[0063] Alternative solutions

[0064] The decorrelation method presented above uses decorrelation or de-covariance between layer or component outputs for the training batch. Other losses can produce similar effects to decorrelation. A loss that can be called an anti-distillation loss can be used to push the outputs away from each other, such as an L2 loss,

[0065]

[0066] Other losses that could force the predictions to be different are also possible.

[0067] The correlation and covariance losses can be modified to eliminate the double counting caused by component outputs being correlated outside of a pair. Reducing the correlation between outputs a and b and between outputs a and c may double count the correlation between components of a and b and c. To eliminate this double counting, the component Z can be i Perform Gram-Schmidt orthogonalization to generate the basis of the batch vector. Not in vector Z i Increase the decorrelation loss between, but for j < i, it can be in Z i and B j Add decorrelation loss between j are the basis vectors produced at Gram-Schmidt orthonormalization step j. Applying the loss only to these correlations will ensure that each additional step attempts to eliminate only the projections (or correlations) that were not eliminated by the previous steps.

[0068] Instilling innovation in layers

[0069] MMSE Estimator

[0070] To solve (under feasible linear constraints) the maximization of the innovation of each activated neuron, we start with the solution given by Z -j The n-1 dimensional vector representing other neurons derives the j-th layer neuron Z j The linear MMSE estimator of . Similarly, in order for the estimation of correlation to be effective, the batch size should be large enough relative to the number of neurons, that is, k>>n. Let C z；-j Represents from C z The (n-1)×(n-1) dimensional correlation matrix is ​​obtained by discarding the j-th row and j-th column.

[0071] b j =E[Z j Z -j ]

[0072] is the n-1 dimensional cross-correlation vector between the jth neuron and all other neurons in the layer. Note that b j is equal to C with the jth element omitted z The jth column of ,. Then, according to Z -j Z j The (homogeneous) linear MMSE estimator of is given by

[0073]

[0074] in

[0075] a j =(C z；-j ) -1 b j .

[0076] For example, in estimating Z for example l in a batch j The error at time is given by

[0077]

[0078] where lowercase letters are used to represent instantiation values ​​of random variables. It can be estimated per example, where the superscript l represents the index of the example in the batch. The average batch error is given by

[0079]

[0080] The mean batch squared error is given by

[0081]

[0082] Squared mean batch squared error

[0083]

[0084] The expected MSE can also be calculated directly from the correlation. It is given by

[0085]

[0086] Using this expression, we do not need to calculate the error for each example in the batch. Instead, we estimate the correlation from the batch, invert the matrix C z；-j , and use this equation to calculate the MSE for the batch.

[0087] To avoid possible singularities, if our neurons are not activated during a batch or are activated for a very small fraction of the batch, we can exclude them from the prediction and additional propagation losses.

[0088] Find the inverse innovation loss:

[0089] The loss of a layer in a training batch is the sum of the losses over neurons and examples in the batch.

[0090]

[0091] The loss can be normalized per example and per layer of neurons. However, it is possible that this may not be necessary. In any case, the normalization or lack thereof can focus on a regularization scale factor λ that can be added to the loss when combined with the target loss of the network. The loss can be decomposed per neuron per example, as well as per neuron only.

[0092] The innovation loss aims to drive innovation in each neuron in the network's maximizing layer. This means that the (minimum) mean squared prediction error should be maximized. To superimpose this constraint with the network loss being minimized, the concept of error can be transformed into a loss that can be minimized. Applying a negative batch mean squared error loss would satisfy this requirement. However, such a loss is not convex (it is concave). Therefore, in the case of an MSE of 0, the gradient of the loss is 0. Around 0, it has a very small magnitude. The area where the MSE is close to 0 is where we need large gradients to push neurons to innovate faster on other neurons.

[0093] In some embodiments, we can have a loss that inverts the MSE, without negating the MSE. This loss does provide large gradients in the 0 region. We can dampen this loss by adding a positive term β to the denominator, capping the gradient at 0 MSE, i.e. without this term the gradient is infinite. For the jth neuron on the lth example, this results in the following loss

[0094]

[0095] For numerical convenience, a factor of 1 / 2 is added to cancel out the 2 terms in the gradient. (It is ultimately offset by a regularization factor λ which will be added.) The layer loss is given by

[0096]

[0097] The gradient of the per-neuron per-example loss with respect to (with respect to) the ith neuron in example e is given by

[0098]

[0099] Note that this gradient has the loss for each neuron j in example l with respect to any neuron in the layer (including the jth neuron). We use a j；v To represent the n-1 dimensional vector a j The difference between the last two regions is only due to the removal of z -j The j-th component in is shifted upward.

[0100] We note that the gradient pushes the weights of non-innovative neurons away from the estimator based on the correlation of other neurons. However, it also pushes the neurons in the estimator in the opposite direction away from the estimated neurons. This may potentially weaken other neurons. In some embodiments, it may be more desirable to allow only the predicted neurons to be moved without moving the predicting neurons. This can be applied by stopping the propagation of the gradient on the i-th neuron, where i≠j. However, it may not be necessary. The alternative method based on the beneficial score (MDL regularization) presented below can solve this problem.

[0101] If we have very correlated neurons (in the extreme case, equal neurons) i and j, and the gradient of the innovation loss of one neuron i on the two will negate the gradient of the innovation loss of the other neuron j on the two, and they may never move out of this correlated state. To solve such problems, in some embodiments, we can prioritize among the neurons. One solution to this is to avoid such problems by assigning different regularization coefficients to the losses of each neuron. However, we can randomly assign the inverse of the ratio of the correlations between neurons, which may produce the same problem. This can be solved by randomly perturbing the regularization coefficients between batches. This topic will be discussed in more detail when we describe how to use regularization to apply layer losses on the network. Similarly, the scheme based on beneficial scores (MDL regularization) described at the end can be more robust to problems like this.

[0102] Replaceable innovation losses:

[0103] As an alternative to inverting the MSE loss, decreasing functions of the MSE with positive target values ​​can also be used, as long as they meet the convexity requirements and achieve maximum gradients in the region of 0 MSE. Unfortunately, exponential decays with respect to the squared error suffer from the same problem as the negated MSE (0 gradient of non-squared error with respect to 0 squared error). However, functions that decay with respect to the square root of the MSE meet these requirements and can be used. Two such losses are the inverse losses with respect to the L1 norm of the error given by

[0104]

[0105] Here, again, β is used to dampen the function to avoid infinite gradients at 0. Alternatively, the Laplace loss

[0106]

[0107] The parameter α can be used.

[0108] Batch squared error:

[0109] The loss L used to predict neuron j from other neurons j is factored over the examples in the batch. However, the statistical relevance of this scheme is estimated by all examples in the batch. It is also reasonable to consider the loss aggregated over the examples. Unlike linear regression, the gradient is no longer linear in the error term. Computing the expected error or squared error over the batch and using the expected value in the update will result in a different batch gradient.

[0110] Using the expected error with a damped inverse square loss (instead of applying the expectation of the loss over the batch as before), the aggregated loss over all neurons is given by:

[0111]

[0112] The loss per neuron is

[0113]

[0114] Relative to the example The differential of neuron j

[0115]

[0116] Sum all examples in a batch

[0117]

[0118] The resulting gradient for all examples in the batch is equal to the expected gradient of the error over the batch. When we factorize over examples, the sum of the gradients over the batch is the expected value of the same function

[0119]

[0120] For positive and sufficiently large x and for sufficiently small β, the magnitude of this function is convex. Therefore, by Jensen's inequality, the gradients computed per example can result in larger gradient steps (if they are all in the same direction). Similarly, for sufficiently large negative values, the same happens in the opposite direction.

[0121] For some i≠j, the gradient with respect to j is multiplied by a factor -a in the case of i<j and i>j respectively. j；i or -a j；i-1 , L j Relative to The gradient of is represented similarly to the per-example gradient.

[0122] Using the expected squared error with a damped inverse squared loss, the aggregated loss for all neurons is given by

[0123]

[0124] The loss per neuron is

[0125]

[0126] Relative to the example The differential of neuron j is,

[0127]

[0128] Sum all examples in a batch

[0129]

[0130] Due to EX 2 ≥(EX) 2 , so the magnitude of the gradient of the batch MSE is smaller than the magnitude of the mean error of the batch, which (under the conditions mentioned) is smaller than the aggregation of the gradients of the examples over the batch.

[0131] Similarly, for some i≠j, the gradient with respect to j is multiplied by a factor -a in the case of i<j and i>j respectively j；i or -a j；i-1 , L j Relative to The gradient of is represented similarly to the gradient of each example.

[0132] If we compute the MSE directly or innovate via projection as presented later, then using the gradient of the expected mean squared error can be a natural fit.

[0133] Gradient Descent Regression Estimator

[0134] An ensemble of n MMSE estimators in a layer of n neurons requires inverting n (n-1) × (n-1) dimensional correlation matrices C z；-j In some cases, this may be computationally intensive. As an alternative, the least squares linear regression problem can be solved using the vector a during the training batch. j The squared loss

[0135]

[0136] This can be done by performing multiple iterations of batch gradient descent on the batch. It can be done for all n neurons. While we can iterate enough times to guarantee convergence to a solution, we can also perform a small fixed number of iterations for some potentially suboptimal solution. This solution can then be used to compute the loss and gradients propagated down from that layer instead of the MMSE. In minimizing the loss relative to a j After the error is , the remaining steps will now minimize the inverted error loss as presented for MMSE. The incomplete regression may lead to some suboptimal minima, on which the error maximization is now performed. The incomplete minimization in the first step may exaggerate the innovation of the neuron. Although suboptimal, it can still perform a sufficient level of regularization, but with a more feasible and less computationally intensive scheme.

[0137] If we can use O(n 2 ), then we can perform several gradient descent iterations in a given batch to improve the estimator a j Then, we store the coefficients a j , and warm-start the next iteration in the next batch using the stored values. This can improve the results and make the vector a j This is close to the value of MMSE, but at the expense of additional storage space.

[0138] Batch innovation through projection

[0139] For projection-based schemes, we should have k ≥ n, i.e., we should have at least as many examples in a batch as there are neurons in the layer. Otherwise, the dimension of the space spanned by the neurons of a layer is bounded by a value smaller than the number of neurons in the layer, k.

[0140] To present the projection-based scheme, we need to transpose the previously defined notation between the n neurons and k examples of the representation layer dimensions. Let

[0141]

[0142] is a column vector representing the k values ​​of neuron j for the k training examples in a batch. Define a k×n dimensional matrix

[0143] X=[x1,x2,...,x n ]

[0144] As the concatenation of the column vectors of layer neuron values ​​on all layer neurons. Let

[0145] B -j =[b -j；1 ,b -j；2 ,...,b -j；n-1 ]

[0146] is a k×(n-1)-dimensional matrix whose columns are the basis vectors spanning the space spanned by the columns of X with the jth column omitted. B can be obtained by performing a Gram-Schmidt orthogonalization on the columns of X excluding the jth column. -j Column B -j The actual columns of will depend on the order in which the Gram-Schmidt procedure is performed on the columns of X, but this will not affect the actual projection error we are interested in. The complexity of this procedure will be discussed at the end of this section.

[0147] Now, for all j, 1≤j≤n, we need to calculate the vector x j Projection on the -j The error in the space spanned by . This error is given by

[0148]

[0149] in <x j , b -j；i > is the inner product between two vectors. e j The lth component is the error component of example 1. The error e j Given a vector x that cannot be represented by the other n-1 vectors i , i≠j explains (or predicts) x j In other words, the error is the vector of differences between the values ​​of the jth layer neurons and their predictions by the other n-1 neurons in the layer, over the k training examples in the batch.

[0150] The MSE of neuron j on a batch is given by e jThe normalized (e.g., per-example) squared norm of gives

[0151]

[0152] Due to the projection process over the full batch of examples, expressing the loss objective in terms of batch mean squared error seems to be a natural approach here. Therefore, we define the layer loss as

[0153]

[0154] This can be done by following the linear steps of the derivation of the loss, including generating the basis matrix B for each j -j ) to derive relative to gradient.

[0155] We can also define the per-example loss using the equation for the similar loss in the MSE strategy. j The error corresponding to each element in can be inserted into the equation. Similarly, the gradient can be obtained from executing is derived in a linear step.

[0156] Matrix B -j The generation of requires that each of the n-1 vectors from X be incrementally projected onto B -j The projection is subtracted from the vector, and the residual is normalized to B -j In the new column of j Projecting onto the basis requires another n-1 projections (multiplications). Each projection is O(k) multiplications. For the complete process, we have O(1+2+3+...+n-1)=O(n 2 ) projection / normalization steps. Therefore, the complexity of projecting a vector j into the basis created by the other vectors is O(kn 2 ) multiplications. If we do this for each j, we will need O(kn 3 ) multiplications.

[0157] This complexity can be reduced by sharing some of the basis vectors we create. We can store O(nk log n) elements and perform only O(kn 2 log n) operations, instead of storing O(nk) elements at any given time (O(n) basis, each of the k elements). We store the vector x jDivide into two sets of equal (or nearly equal) size. For each set, generate a basis. Store as the concatenation of the n basis vectors formed by the two sets. Then, each set is again divided in half. Let A be the original upper half and B be the original lower half. The original vectors of A are now split in half into two sets A1 and A2, and those of B are split into sets B1 and B2. For each in set A v we continue the Gram-Schmidt process on the basis vectors formed for set B (individually between the sets). Similarly, this is also done for B v using the basis formed for A. Recursively repeat the process until each remaining set has a single vector. For each of these single vectors, now project its sibling from the previous step onto the basis consisting of the vectors in the set on the path of this single vector to produce an error. This process has O(log n) steps, with O(n 2 ) projections at each step, each taking O(k) operations.

[0158] Alternative projection loss

[0159] Rather than using the projection error and minimizing the loss inversely proportional to the error, we can directly minimize the projection energy of any vector onto the space spanned by other vectors. We can perform Gram-Schmidt orthonormalization on the component x j to generate a basis for the batch of vectors. In addition to adding a decorrelation loss between the vectors x j we will also add a decorrelation loss between x j and the columns of B -j where B -j is the basis matrix defined as above. The loss for x j will be imposed on the energy of the projections from x j onto all columns of B -j . Imposing the loss only on these correlations will ensure that each additional step attempts to eliminate only the projections (or correlations) not eliminated by the previous step. If the loss is imposed on both x j and the columns of B -j and gradient backpropagation is allowed to both of these vectors, then for each j we can reduce the complexity of the process and perform it only on column i such that i < j. This allows constructing a single matrix B to account for all correlations, but may suffer from multiple compensations for the same correlations.

[0160] Applying layer innovation on the network

[0161] We will now show how to impose layer innovation decorrelation constraints on neural network updates during backpropagation. These methods apply to any of the innovation loss calculation strategies presented above.

[0162] Innovation Infusion as Regularization

[0163] One scheme for imposing a layer innovation loss constraint when training a network is a form of L2-like regularization. During backpropagation, when a layer is entered from the layer above, the loss seen by that layer is the sum of the loss from above (the full objective plus potential regularization losses from other layers) and a scaled version of the innovation loss.

[0164]

[0165] in, Applied to any of the above full losses (aggregated over both examples and layer neurons). The coefficient λ is the regularization coefficient.

[0166] As mentioned above, if two layer neurons j and j are equal (or highly correlated) for all examples, then the loss on neuron i will push them away from each other in one direction. The loss on j will push each of them in the opposite direction. This may cause an aggregate gradient that does not push any neuron. To break such symmetry, different scaling factors λ can be used j Regularize the constraints of each neuron. Therefore, the regularization layer loss applied is

[0167]

[0168] If the neurons are scaled versions of each other, then we will choose λ that offsets the ratio between the gradients i and λ j It is still possible (although very unlikely) that this raises similar problems as above. To solve this problem, we can j The batch regularization scaling factor λ is applied to the values. j A fixed mean with an additional low variance (relative to the mean) 0-mean normal random variable may be included, drawn once for each batch and each neuron j.

[0169] Innovation Infusion Using Constrained Optimization

[0170] As an alternative to the regularization scheme, an innovative decorrelation loss can be imposed via Lagrangian constrained optimization. For each element of the constraint (e.g., for each of the n neurons of a layer), we append the dual variable λ j , and the loss superimposed with the loss propagated down the network

[0171]

[0172] Relative to minimized, but with respect to the dual variable λ j We can also add L2 regularization on the dual variable over the training batches.

[0173] Innovative infusion using a Minimum Description Length (MDL)-like regularization scheme

[0174] MDL regularization for linear models uses a helpfulness score that measures the improvement a feature has on the loss. The feature is trained without regularization, but sees all other features that are regularized. The helpfulness score is used to regularize the feature by scaling down its weight. With a large helpfulness, almost no scaling occurs, with a lower helpfulness, the feature can be scaled down. A threshold μ on the helpfulness score can be used to completely reject features whose helpfulness scores are below this threshold.

[0175] The mean squared error calculated for the neurons in a layer can serve as a useful score for an MDL-like regularization scheme for the layer. Using this scheme, neurons are trained without regularization in the backpropagation algorithm. However, to perform innovation regularization, the estimated neuron uses the regularized version of all other neurons. The forward propagation then propagates the regularized version of the neuron to the network. Backpropagation receives the gradients of the regularized version and scales them back to the full version for backpropagation to the layers below.

[0176] β j Represents the helpfulness score of the jth neuron in this layer. To represent the regularized version. For the first two innovative calculation strategies, the full activation weight of the jth neuron is estimated based on the regularized activation weights of all other neurons by the following formula

[0177]

[0178] And the error (for example, example 1) is given by

[0179]

[0180] The regularized weights of the neurons are given by

[0181]

[0182] Where ρ and ξ are parameters, and σ represents the Sigmoid function

[0183]

[0184] We can set the threshold μ so that if β j ≤μ, then we will zj regularized to 0. Note that unlike MDL with linear models, the mean squared error is lower bounded to 0, i.e., we cannot have negative beneficial scores. Therefore, we can reduce the scaling factor using the parameter ξ by choosing a (high) negative value. Alternatively, the sigmoid function can be replaced by another function. A possible alternative is the hyperbolic tangent, i.e.

[0185]

[0186] Where ξ≥0.

[0187] For innovation by projection, we use the regularized weights of other neurons to predict the unregularized weights. This means that the matrix X uses the unregularized version of the neuron weights, while the basis matrix B -j Use the regularized version to build on. The lower complexity version applies B on the vector that causes the regularization -j , but the final projection step is applied to the columns of X. Note that this makes no difference as long as we do not set the threshold vector to 0, since x j All examples of will have the same scaling. If we have β j <μ, the threshold of neurons j is set to 0, which will j All components of are set to 0. This excludes x from the Gram-Schmidt process. j , to find all B with i≤j -i . Hence, it potentially increases the error of other neurons that can be predicted by the jth neuron. This creates a desired effect where we discard neurons that add little value to other neurons, thereby increasing the benefit of other neurons.

[0188] make Represents spread to Then, from Spread to Then the gradient connected to the layer below is given by the chain rule

[0189]

[0190] where σ′(ρβ j +ξ,μ) represents the clipped version of Sigmoid, (assuming we use Sigmoid) for β j ≤μ is clipped to 0.

[0191] The useful score can be the cumulative squared error over all batches and the current batch, or we can normalize it to be an average per batch per example. We can also use a sliding (or decaying exponential) window over a set of recent batches. Since in either case we sum the squared errors over the examples, and we do not apply nonlinearities to the resulting mean squared error, there is no difference between aggregating the per-example squared errors or using the average batch squared error.

[0192] The initial helpfulness score before training starts can be set to some large enough value for all neurons to ensure that they have a chance to train. However, it is possible that random initialization where some neurons are prioritized over others can break the symmetry and let some neurons evolve to be helpful first, keeping the redundant neurons regularized. Setting the initial helpfulness score to 0 for all j will stop training, as it will prevent any backpropagation by setting all neurons to 0.

[0193] The parameter ρ can be tuned by the design of the benefit score. A neuron with a low average or cumulative MSE (depending on the design choice of the benefit score) will have its weights seen by other components of the network above the layer regularized to very small values ​​or to 0 if below a threshold μ. This will stop updates from propagating down to neurons connected to that neuron. However, if these neurons start to deviate from their current values, they may affect other neurons in the current layer to change, so that the MSE of the estimated regularized neuron can grow again, causing that neuron to be effective in updates again.

[0194] If a neuron is regularized to 0 over many batches, the parameters that this neuron presents may not become useful again. It can be beneficial to randomly initialize the neuron's links and its biases, so the network may be able to find different, potentially useful parameters for this neuron. The neuron can now be initialized in a way that is orthogonal to the other neurons on the current batch: randomly generate a batch value for the neuron; generate a linear MMSE estimator for the neuron from the other neurons using the previous two strategies (or derive the estimator from the projection strategy); subtract the MMSE estimate for each example in the batch from the randomly generated value, or, using the projection strategy, just apply it to the neuron to find the residual; and propagate the resulting gradients to the connected links and biases. This can be done over several batches, so the links below adapt to the new values.

[0195] In some embodiments, the last two steps can be replaced by solving equations for the values ​​of the neurons that satisfy the bias and the links directly connected to the neurons below them based on the activations they see below them, and assigning these solutions to warm start these links again.

[0196] For some layers, the relevant neurons that can be predicted by other neurons in the layer represent redundant parameters that do not add predictive value to the network. This can also include parameters that represent features that are not helpful for prediction. This conceptually means that the innovation of neurons can have the meaning of benefit scores. Although in some cases, such benefit scores are similar to the role of MDL regularized benefit scores, the correlation dimension captures more than the conventional MDL score. It also captures the correlation between features. For the top hidden layer connected to the output loss, it is possible to add another dimension of noise reduction improvement to the neural network using the loss that drives the benefit score (such as the loss used in the linear model using MDL).

[0197] The MDL scheme can eliminate bad neurons faster than gradient-based regularization schemes, which require iterative propagation of gradients until neurons are regularized without being useful. If this happens, it can generalize better and better eliminate overfitting.

[0198] As already mentioned, the sigmoid function used to scale the neuron weights as a function of the innovation benefit score is only one option. While it is reasonable for using MDL regularization with linear models, since it takes the benefit score as the logarithmic advantage to indicate that the feature is beneficial to the model, here,

[0199] The concept of probability is slightly divorced from the metric being computed. Besides the already mentioned hyperbolic tangent, other monotonic scaling functions with targets in [0,1] might be better.

[0200] The helpfulness score of the MDL-like scheme is directly related to the objective we are trying to maximize. The larger the objective (the more innovative the neurons), the more helpful it is. By not inverting the innovations, we also do not have different behavior whether we average the squared error of the batch or the objective.

[0201] An additional advantage of the MDL regularization driven strategy is that we do not apply a loss directly on the network. The network can be trained without an additional loss other than the regularizing effect on the regularized neuron itself. Innovation is only used to regularize the neuron in question, without affecting other neurons. This prevents biases that can be formed by unnatural regularization (as we have seen with MDL regularization vs. L1 schemes). It also avoids problems like the one described for two identical neurons, where the innovation targets for the two identical neurons can negate each other because the constraint on one neuron affects the other.

[0202] Practical Constraints and Mini-Batching

[0203] Computing the innovation of the neurons of a layer using any strategy proposed for a given batch may be too computationally intensive. The computational complexity is O(kn 2 ) or larger. If a layer consists of thousands of neurons and the batches of examples consist of thousands of examples, such complexity is not feasible. To address this, we can use mini-batches in either dimension or both. Examples can be mini-batched into smaller batches (although these should be larger than the number of neurons in the batch to meet the dimensionality requirements). We can also divide the neurons in a layer into batches and compute the associated loss for smaller mini-batches of the layer's neurons. These two approaches can also be combined.

[0204] Mini-batching neurons divides the layer into sets that are forced to be uncorrelated, while correlations between neurons from different sets may still exist. This is not ideal, but can be avoided (at least partially) by periodically or randomly shuffling the mini-batch partitions between different batches. The random shuffling will ensure that the innovation constraints are applied evenly on subsets of layer neurons on average. Mini-batching smaller subsets of relative layer neurons and shuffling the partitions between full batches will greatly reduce complexity and make this scheme feasible even in the case of larger layers.

[0205] Example use on a neural network including an ensemble of towers

[0206] One example application of the techniques described herein is a neural network that includes a collection of independent towers (e.g., independent networks that shrink from input to output). Specifically, as an example, Figure 2A A neural network 200 is shown that includes a collection of towers, including towers 202, 204, and 206. Three towers are shown for purposes of explanation, but the network may include any number of independent towers.

[0207] In some embodiments, each tower 202, 204, 206 may receive tower-specific inputs, which may be from an earlier tower-specific portion of the network 200. For example, tower 202 receives tower-specific inputs from portion 252 of the network 200; tower 204 receives tower-specific inputs from portion 254 of the network 200; and tower 206 receives tower-specific inputs from portion 256 of the network 200. Alternatively or additionally, each tower 202, 204, 206 may receive shared inputs, which may be from an earlier portion of the network 200. For example, each tower 202, 204, and 206 receives shared inputs from portion 258 of the network 200.

[0208] Neural network 200 includes neurons organized into multiple layers. As an example, these layers include output layer 218 and multiple hidden layers such as hidden layers 214 and 216. Hidden layer 216 is shown as being located before a set of softmax or sigmoid operations, and thus may be referred to as a logits layer in some cases. As shown in hidden layers 214 and 216, in some cases, neurons may be in the same layer but in different towers (e.g., neurons 241 and 244 are both in hidden layer 214, but in separate towers 202 and 204, respectively).

[0209] In some example applications of the techniques described herein, neurons or other parameters included in one tower may be decorrelated with respect to neurons included in other towers. Decorrelation may be applied at a hidden layer (eg, 214 or 216) or at an output layer 218.

[0210] In some embodiments, the decorrelation technique can be performed by including an innovation loss term in the loss function, which provides each neuron included in one of the towers with a loss value based on the ability of neurons in other towers, rather than other neurons in its own tower, to predict the value of such neuron. The subsequent description refers to decorrelation as any loss that forces a neuron to innovate on other neurons, such as, but not limited to, decorrelation or any of the innovation losses described above.

[0211] As an example, neuron 241 may be decorrelated with neurons 244-248. Furthermore, in some embodiments, decorrelation techniques are not applied to decorrelate neuron 241 from neurons 242 or 243. Thus, an innovation loss may be applied to neuron 241, which provides neuron 241 with a loss value based on the ability of neurons 244-248 in other towers 204, 206, rather than other neurons 242, 243 in its own tower 202, to predict the value of neuron 241. The innovation loss may be applied to some of all neurons 241-248, respectively.

[0212] Similarly, neurons in hidden layer 216 and output layer 218 can be decorrelated with other neurons in other towers in the same layer. As an example, neuron 262 can be decorrelated with neurons 261 and 263. As another example, output 283 can be decorrelated with outputs 281 and 282. Therefore, decorrelation can be applied before the softmax or Sigmoid operation (e.g., at hidden layers 214 and 216) and / or decorrelation can be applied after the softmax or Sigmoid operation (e.g., at output layer 218). If the label is binary, the softmax can be replaced with the Sigmoid function. Otherwise, in some embodiments, decorrelation on softmax can be applied to each label value of softmax (e.g., if there are 4 different label values, decorrelation between neurons can be performed for each of the 4. It can also be performed for 3 of the 4, where the 4th label softmax value comes from the other 3).

[0213] Thus, decorrelation may be applied at one or more different layers of network 200 , including hidden layers such as 214 , logits layers such as hidden layer 216 , and / or output layers such as 218 .

[0214] Figure 2B Another example neural network 300 is shown. The neural network 300 is Figure 2A The network 200 is very similar, except that Figure 2B In the network 300 of FIG. 3 , neurons 261, 262, and 263 of towers 202, 204, and 206 are combined into a single shared neuron 302 before a softmax or sigmoid layer 306 that generates output 304. According to the decorrelation techniques described herein, neurons of hidden layers 214 and / or 216 may be decorrelated. Thus, in some examples, decorrelation may be performed before combining towers into one or more shared neurons (e.g., shared neuron 302). Figure 2B As shown, the combination of towers 202, 204, 206 into shared neuron 302 may occur before softmax layer 306. In some examples, the combination of neurons may be applied as a sum, average, gated mixture of experts, or other forms.

[0215] Figure 2C Another example neural network 400 is shown. Neural network 400 and Figure 2B The network 300 is very similar, except that Figure 2CIn the network 400, neurons 261, 262, and 263 of towers 202, 204, and 206 are combined into a single shared output 402 after the softmax layer 404. Figure 2C As shown, the combination of towers 202 , 204 , 206 into shared neuron 402 may occur after softmax layer 404 .

[0216] Reference again Figure 2A According to another aspect, during training, input values ​​embedded into network 200 may be initialized to values ​​other than zero. For example, input values ​​to portions 252, 254, 256, and / or 258 may be initialized to non-zero values. Specifically, in some embodiments, the input values ​​may be initialized to random values. Initializing input values ​​to non-zero may facilitate the decorrelation techniques described herein because they may provide an initial amount of decorrelation that may be increased during training. In some embodiments, embedded values ​​may be initialized to non-zero values ​​non-randomly, or as values ​​transferred from a previously trained model.

[0217] Example devices and systems

[0218] Figure 1A A block diagram of an example computing system 100 is depicted, in accordance with an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0219] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0220] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0221] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recursive neural network (e.g., a long short-term memory recursive neural network), a convolutional neural network, or other forms of neural networks.

[0222] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine learning model 120.

[0223] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning models 140 may be implemented by the server computing system 130 as part of a web service. Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0224] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0225] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0226] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0227] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0228] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0229] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0230] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, back propagation of errors. In some embodiments, performing back propagation of errors may include performing back propagation truncated over time. The model trainer 160 may perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0231] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Model trainer 160 can perform any decorrelation technique described herein.

[0232] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0233] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0234] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications on the network 180 may be implemented via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0235] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and a collection of training data 162. In such embodiments, the model 120 may be trained and used locally at the user computing device 102. In some such embodiments, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0236] Figure 1B A block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure is depicted. Computing device 10 may be a user computing device or a server computing device.

[0237] Computing device 10 includes multiple applications (e.g., applications 1 to N). Each application includes its own machine learning library and (multiple) machine learning models. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0238] like Figure 1B As shown in , each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, context managers, device state components, and / or additional components). In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to the application.

[0239] Figure 1C Depicted is a block diagram of an example computing device 50 that performs in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0240] The computing device 50 includes a plurality of applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a public API across all applications).

[0241] The central intelligence layer includes multiple machine learning models. For example, Figure 1C As shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in the operating system of the computing device 50 or is implemented by the operating system of the computing device 50 in other ways.

[0242] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing devices 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device (such as, for example, one or more sensors, context managers, device state components, and / or additional components). In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0243] Additional disclosure

[0244] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken by these systems and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or can be distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0245] Although the present subject matter has been described in detail with respect to various specific example embodiments of the present subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. After obtaining an understanding of the foregoing, those skilled in the art can easily produce changes, variations, and equivalents to these embodiments. Therefore, the present subject matter disclosure does not exclude the inclusion of these modifications, variations, and / or increases to the present subject matter, which is apparent to those of ordinary skill in the art. For example, a feature shown or described as a part of an embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, variations, and equivalents.

Claims

1. A computer-implemented method for training a neural network, the method comprising: Obtaining, by one or more computing devices, data describing a neural network, the neural network comprising multiple layers of neurons; as well as Backpropagating a loss function through a neural network by one or more computing devices to train the neural network, wherein the loss function describes a performance of the neural network with respect to a set of training examples, and wherein backpropagating the loss function through the neural network by the one or more computing devices comprises, for each of one or more of a plurality of layers of the neural network: determining, by the one or more computing devices, a gradient of the loss function with respect to neurons included in the layer, wherein, at least for the layer, the loss function includes an innovation loss term that provides a loss value for each of the one or more neurons included in the layer based on an ability of one or more other neurons in the same layer to predict a value of such a neuron; and modifying, by the one or more computing devices, neurons included in the layer to decorrelate neurons included in the layer based at least in part on a gradient of a loss function including the innovation loss term, Wherein, the set of training examples includes text data input to the neural network.

2. The computer-implemented method of claim 1 , wherein at least one of the one or more layers comprises respective portions of two or more separate towers of the neural network, and wherein the innovation loss term provides, for each neuron included in one of the towers, a loss value based on the ability of neurons in other towers, rather than other neurons in its own tower, to predict a value of such neuron, in, Each of the towers is an independent network that shrinks from input to output.

3. The computer-implemented method of claim 1 , wherein: The one or more layers of the neural network include one or more hidden layers of the neural network.

4. The computer-implemented method of claim 1 , wherein: For each of the one or more neurons included in the layer, the innovation loss term pushes the gradient of such neuron to increase its innovation over one or more other neurons included in the layer.

5. The computer-implemented method of claim 1 , wherein: For each of the one or more neurons included in the layer, the loss value provided by the innovation loss term is negatively correlated to an error in predicting a value of such neuron by one or more other neurons included in the layer.

6. The computer-implemented method of claim 5, wherein: For each of the one or more neurons included in the layer, an error in predicting a value of such neuron by one or more other neurons included in the layer includes an error associated with a linear minimum mean square error estimator.

7. The computer-implemented method of claim 6, wherein: For each of one or more neurons included in the layer, the linear minimum mean square error estimator includes an average batch error calculated over a batch of training examples.

8. The computer-implemented method of claim 6, wherein: For each of the one or more neurons included in the layer, the linear minimum mean square error estimator includes an expected mean square error calculated directly from the estimated correlations within the layer.

9. The computer-implemented method of claim 6, wherein: For each of the one or more layers, the innovation loss term provides the loss value based at least in part on an average of a linear least mean square estimator for all neurons included in the layer.

10. The computer-implemented method of claim 6, wherein: For each of the one or more neurons included in the layer, the innovation loss term is decayed relative to a square root of a linear minimum mean square error estimator for such neuron.

11. The computer-implemented method of claim 5, wherein: For each neuron of one or more neurons included in the layer, the innovation loss term provides the loss value based at least in part on an inversion of an error of the prediction.

12. The computer-implemented method of claim 5, wherein: For each of the one or more neurons included in the layer, the error of a value of such neuron predicted by one or more other neurons included in the layer is determined by solving a linear least squares regression problem with respect to the one or more other neurons included in the layer.

13. The computer-implemented method of claim 5, wherein: For each of the one or more neurons included in the layer, the error of predicting a value of such neuron by one or more other neurons included in the layer includes a gradient descent regression estimator.

14. The computer-implemented method of claim 5, wherein: For each of the one or more neurons included in the layer, the error in predicting the value of such neuron by one or more other neurons included in the layer is determined by performing parallel Gram-Schmidt orthogonalization on the neuron example vector and calculating the residual of the resulting vector.

15. The computer-implemented method of claim 5, wherein: For each of one or more neurons included in the layer, the error in predicting the value of such neuron by one or more other neurons included in the layer includes the error in projecting a column vector representing the value of the neuron onto a space spanned by one or more basis matrices including one or more basis vectors.

16. The computer-implemented method of claim 5, wherein: For each of the one or more neurons included in the layer, a squared norm of a projection of a value of such neuron predicted by a basis vector spanned from one or more other neurons included in the layer comprises an additional loss for instilling innovation.

17. The computer-implemented method of claim 1, wherein the one or more neurons included in the layer include all neurons included in the layer.

18. The computer-implemented method of claim 1, wherein: The innovation loss term includes a regularization term that is added to the main loss according to a scaling factor.

19. The computer-implemented method of claim 18, wherein: The regularization term includes an L2 regularization term.

20. The computer-implemented method of claim 1, wherein: For each of the one or more layers, the loss function includes a per-neuron regularization scaling factor that prevents symmetry constraints on neurons.

21. The computer-implemented method of claim 1, wherein: Modifying, by the one or more computing devices, neurons included in the layer based at least in part on a gradient of the loss function includes applying, by the one or more computing devices, the innovation loss term through Lagrangian constrained optimization.

22. The computer-implemented method of claim 1, wherein: For each of the one or more layers, the innovation loss term is applied after activation of neurons included in the layer.

23. A computer-implemented method for training a neural network, the method comprising: Obtaining, by one or more computing devices, data describing a neural network, the neural network comprising multiple layers of neurons; as well as for each of the one or more neurons included in one or more of the plurality of layers, determining, by the one or more computing devices, a benefit score based at least in part on an error in predicting a value of such neuron by one or more other neurons included in the same layer; as well as modifying, by one or more computing devices, a weight associated with at least one neuron of the one or more neurons based at least in part on the benefit score determined for the at least one neuron, Wherein, the input of the neural network is text data.

24. A computer-implemented method for training a neural network, the method comprising: Acquiring, by one or more computing devices, data describing a neural network, the neural network comprising a plurality of neurons respectively connected by a plurality of links; as well as Backpropagating a loss function through a neural network by one or more computing devices to train the neural network, wherein the loss function describes a performance of the neural network relative to a set of training examples, and wherein backpropagating the loss function through the neural network by the one or more computing devices comprises, for each of one or more neurons, links, or biases of the neural network: determining, by the one or more computing devices, a gradient of a loss function with respect to one or more neurons, links, or biases of the neural network, wherein, at least for the one or more neurons, links, or biases of the neural network, the loss function includes an innovation loss term that provides a loss value for each of the one or more neurons, links, or biases based on an ability of one or more other neurons, links, or biases to predict a value of such neuron, link, or bias; and modifying, by the one or more computing devices, one or more neurons, links, or biases of a neural network based at least in part on a gradient of a loss function including the innovation loss term to decorrelate the one or more neurons, links, or biases of the neural network, Wherein, the set of training examples includes text data input to the neural network.

Images