BNN training using mini-batch particle flow

Abstract

Devices, systems, and methods for training a Bayesian Neural Network (BNN) using mini-batch particle streams are discussed herein. A method for training a Bayesian Neural Network (BNN) using batch inputs and operating the trained BNN may include initializing particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN and the particles collectively represent a distribution of the parameters of the BNN, optimizing the particles based on the batches of inputs using a mini-batch training particle stream to result in an optimized distribution of the parameters, determining a predictive distribution using the optimized distribution of the parameters and predictions from each of the NNs, and resulting in marginal distributions representing the predictive distributions.

Description

[Technical Field]

[0001] Priority claims This application claims the benefit of priority to U.S. Patent Application No. 17 / 509,278, filed October 25, 2021, which is incorporated herein by reference in its entirety.

[0002] The embodiments discussed herein relate to devices, systems, and methods for training Bayesian neural networks (NNs) using mini-batch particle flows. [Background technology]

[0003] Most NNs perform point estimates without direct uncertainty metrics or confidence measures. Standard NNs can also perform relatively poorly on open sets. Bayesian NNs (BNNs) learn statistical distributions of weights, providing a statistical environment in which decision uncertainty and confidence can be determined. Existing training methods for BNNs include Hamiltonian Monte Carlo, variational inference (both Monte Carlo and deterministic), stochastic backpropagation, and standard particle filters. These training methods are computationally expensive and require relatively large amounts of data for training. [Brief explanation of the drawings]

[0004] [Figure 1] As an example, we present a block diagram comparing a standard DL architecture with a BNN. [Figure 2] As an example, a diagram of an embodiment of a NN is shown. [Figure 3] As an example, a flow diagram of an embodiment of a NN training procedure using a training particle stream is shown. [Figure 4] As an example, we show a plot of accuracy versus measurement update for a BNN trained on MNIST{0,1} using a training particle stream. [Figure 5] As an example, here is a plot of accuracy versus measurement update for a BNN trained on MNIST{0,1,2,3}. [Figure 6] As an example, a flow diagram of an embodiment of the mini-batch training particle flow technique is shown. [Figure 7] Shows plots of network accuracy with increasing batch updates for batch sizes of 1, 2, and 16, respectively. [Figure 8] FIG. 1 is a block diagram of an example environment including a system for NN training, according to an embodiment. [Figure 9] By way of example, a block diagram of an embodiment of a machine in the exemplary form of a computer system is shown, upon which instructions may be executed to cause the machine to perform any one or more of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0005] Particle flow has recently been modified for use in the context of deep learning (DL). Such particle flow techniques for DL training are called "training particle flow." Training particle flow can train Bayesian neural networks (BNNs). BNNs trained using training particle flow can be called "particle flow BNNs." Particle flow BNN architectures exhibit high prediction accuracy with few training samples and a powerful ability to measure prediction uncertainty using the variance of predictions made by the BNN. A case study using the MNIST class {0,1} demonstrated this to be the case. However, current implementations of particle flow BNNs tend to lack robustness and can have difficulty training more than two classes, as discussed below. See Figures 4 and 5 for examples. The BNN and training particle flow are described. Next, a description of mini-batch particle flow training is provided. Training a BNN using mini-batch particle flow results in a more robust BNN that is easier to train for more than two classes than alternative single-input particle flow training methods.

[0006] FIG. 1 shows, as an example, a block diagram contrasting a standard DL architecture 102 and a BNN 104. Standard DL architectures provide point estimates of model predictions (outputs 108) and network parameters (node weights 112). Such DL architectures 102 do not provide a direct means for quantifying uncertainty. Instead, standard DL architectures 102 rely on indirect methods for estimating uncertainty. Common methods include the use of entropy and confidence scores and functions, as well as application-specific methods. Bayesian DL architectures 104 and statistical methods tend to provide a more natural context for quantifying uncertainty.

[0007] BNNs have been studied since at least 1992 and are a growing field. BNN technology uses Bayes' theorem as a guide to solve for the posterior probability distribution of the weights in a NN (the distribution of nodes). The computational difficulty of solving Bayes' theorem for DL tasks has led to the development of numerous approaches for estimating the posterior distribution of NN weights. Well-known approaches to BNN optimization include Hamiltonian Monte Carlo, Monte Carlo variational inference, deterministic variational inference, and stochastic backpropagation (PBP).

[0008] The output 110 of the BNN 104 is a distribution for each class compared to the output of the NN 102, which provides a score for each class as output 108. The distribution of the output 110 can be a natural consequence of using a distribution ("dist.") instead of a scalar weight as in node 112 to represent the activation function of node 114.

[0009] Deep ensembles and probabilistic regularization methods offer alternative approaches towards estimating uncertainty in DL. While these methods do not optimize Bayes' theorem, they provide a statistical environment for calculating predictive uncertainty at a fraction of the computational cost of BNNs. These statistical methods have been used in a variety of applications to quantify uncertainty.

[0010] A common theme in existing Bayesian NNs and statistical approaches is very large training sets and training for thousands of epochs. However, real-world datasets are typically sparse, and training a general NN with thousands of parameters may be insufficient. Embodiments provide a NN architecture that can quantify uncertainty and perform robustly within the limit of sparse datasets and for open-set problems.

[0011] Embodiments use a modified form of particle flow technology commonly used in particle filters, but modified and reused to train BNNs. The modified form of the particle flow method is referred to herein as "training particle flow." Particle flow is a method for optimizing Bayes' theorem and has (until now) only been used in the context of particle filtering. Numerical experiments with particle flow in the context of particle filtering have shown that particle flow can reduce computational complexity by orders of magnitude compared to standard particle filters and other state-of-the-art algorithms for the same filter accuracy. Furthermore, particle flow can reduce filter error by orders of magnitude compared to extended Kalman filters and other state-of-the-art algorithms for difficult nonlinear non-Gaussian problems.

[0012] While particle methods for optimizing neural networks have recently emerged, optimizing BNNs using particle flows has, to our knowledge, not been done before. Results from BNNs trained to perform classification tasks using MNIST{0,1} demonstrate high prediction accuracy with very few training samples. Furthermore, BNNs trained to perform classification tasks possess a powerful ability to measure prediction uncertainty using the variance of network predictions.

[0013] particle flow Consider a system with an internal state s and measurements m. Bayes' theorem relates the posterior probability of a state given a measurement p(s|m) to the prior distribution p(s) on the state and the likelihood p(m|s) of the measurement given the state according to:

number

[0014] where p(m) = ∫p(m│s)p(s)ds is a proof that acts as a normalizing constant. The measurement m is a quantity that helps characterize what the internal state is like or what the internal state will be like. Given a general Markov process with a sequence of noisy measurements {m}, particle filters provide a way to estimate the system's internal state(s) {s} using Bayes' theorem as a guide.

[0015] Particle flow is a method used in the context of particle filtering to estimate the optimal posterior distribution of the internal state for each measurement. Particle flow optimizes Bayes' theorem by evolving a prior distribution to a posterior distribution along a log homotopy.

number

[0016] Two continuous functions in their respective spaces are said to be homotopic if one can be "continuously deformed" into the other. Homotopy exists between functions that can be deformed in this way.

[0017] where K(λ,m)=∫p(m│s) λ p(s)ds normalizes the posterior distribution p(s,λ│m) for each λ. The scalar homotopy parameter λ = [0,1] evolves the distribution from the prior to the posterior for a particular measurement m. Each particle represents a single realization of the system's internal state s. The flow of particles along the log homotopy is described by the following stochastic differential equation (SDE):

number

[0018] During the ceremony,

number

[0019] The evolution of the posterior distribution of particles follows the Fokker-Planck equation (the diffusion squared matrix is Q ij =Σ k B ik B jk (defined as

number

[0020] Note that the gradients and derivatives written in (4) are in Cartesian coordinates, but (4) can be used in any suitable Cartesian coordinate system by appropriate transformation of the partial derivatives.

[0021] Drift Speed

number

number

number

[0022] During the ceremony,

number

[0023] The geodesic solution assumes a Gromov solution for drift velocity and no diffusion (i.e., Q=0), which does not simultaneously satisfy equations 2 and 4. The zero curvature solution assumes that the particle does not accelerate as it changes λ and there is no diffusion term (i.e., Q=0). The drift velocity in this solution is proportional to the Gromov drift velocity.

[0024] DL and supervised learning tasks DL is a branch of ML that uses a series of layers of nodes to learn higher-order representations of data for supervised, semi-supervised, or unsupervised learning tasks. While the described embodiments focus on supervised learning tasks, the embodiments can be applied to any learning task for which a likelihood function can be defined.

[0025] In supervised learning tasks, deep neural networks (DNNs) can be used to learn the relationship between input and output data for regression or classification. For regression tasks, the NN can predict a dependent variable p that has a causal relationship with the input data. For classification tasks, the network can predict the probability p of a particular class. _j , the word "probability" is a bit of a misnomer here. Classification tasks often use a SoftMax activation function in the output layer to generate a vector whose elements sum to 1. This output represents a set of class probabilities, but these "probabilities" are not necessarily well-calibrated to the actual accuracy of the network. In this sense, the output probabilities can be more accurately understood as the normalized scores for each class. During training, the corresponding truth class or truth value y of the input data is calculated. T You can evaluate the NN predictions against and adjust the network weights using your chosen optimization scheme.

[0026] The "likelihood" function is used in many gradient-based optimization methods in DL. For regression tasks, the NN weights θ = {θ i} and the truth variable y given the network prediction p T The likelihood L of is usually modeled by a normal distribution.

number

[0027] where Σ is the truth y T is the covariance matrix that scales the error between y and the prediction p, and the index k is T This likelihood function is the function of the prediction p and the truth y T We assume a Gaussian white noise discrepancy between the σ and the σ. The corresponding log-likelihood is given by

number

[0028] This is reminiscent of the mean squared error (MSE) when Σ is the identity matrix.

[0029] For classification tasks, a categorical distribution is a function of the predicted class probability p j , j∈[1n クラス ] and the true class y of the input given the NN weights θ T represents the likelihood L of

number

[0030] where y T ={y Tj} is a one-hot encoded vector of the truth classes of the image, or a non-binary vector summing to 1 if you are using soft labels. The corresponding log-likelihood is the negative of the cross-entropy function.

number

[0031] Mapping particle flows for training BNNs 2 shows, by way of example, a diagram of an embodiment of a NN 200. The NN 200 includes L layers of nodes. θ contains the network parameters θ={θ j}={W 1 ,b1 ,W 2 ,b 2 ,…,W L ,b L}. NN200 is a set of N パラメータ parameters, and the network parameters θ are N パラメータ dimensional vector θ∈R Nパラメータ becomes.

[0032] In a typical supervised learning task, we have a set of training data D={X,Y T} to train the NN200, where X={x j} is the set of all inputs, and Y T ={y T,j} is the corresponding set of all truth values given X. Each NN prediction p j is the data

number

number

[0033] In the formula, σ represents the activation function of each layer of nodes, and g represents the affine transformation at the nodes of each layer.

[0034] The goal of a BNN is to use Bayes' theorem to learn the optimal posterior distribution of the network parameters p(θ | D) given the data.

number

[0035] The right-hand side of equation (12) is the posterior probability

number

number

[0036] The embodiment maps the particle flow used in the particle filtering context to the DL context, and defines the network parameters θ and the truth value Y in Eq. (12), respectively. T Equating the internal state {s} and measurements {m} in equation (1) for j}. The training particle flow uses the homotopy color λ to adjust the network parameters {θ j The likelihood of each measurement m given the internal state s is expressed as the likelihood Y of the truth value given the prediction P of the network 200 based on the input data X. T In the DL context, each particle is assigned a network parameter {θ j} represents a single realization of

[0037] The mapping of particle streams to training particle streams used in the particle filtering context is mathematically described in equations (13) through (15).

number

number

[0038] The log homotopy constraint given in equation (3) is logp(θ,λ│{X,Y T})=λlogL(Y T │θ,X)+logp(θ)-logp(λ,│X)Y T (Formula 16)

[0039] Here, a scalar homotopy parameter λ is added to the notation for the posterior distribution and normalization factor to specify the variance of these terms in λ. The corresponding gradients and derivatives written in equation (4) are now with respect to the network parameters for the training particle flow.

number

[0040] Equation 18 shows the mathematical expression of the drift velocity in the training particle stream using the Gromov equation.

number

[0041] The Gromov equation generalizes well to architectures with L ≥ 1 layers, various activation functions, and arbitrary prior and likelihood function forms. However, Equation (18) is incomplete because NN200 uses a linear activation function

number

[0042] Equation 19 provides the mathematical expression for the constant diffusion matrix used in particle flow training. Q=αId (Equation 19)

[0043] where α∈R0 is a positive real number and Id is the identity matrix. A constant diffusion matrix can help provide numerical stability. Additionally, adding a small amount of noise can prevent network training from getting stuck in local minima. A small exponential decay factor can also be added to the diffusion matrix to reduce the effect of noise as the number of weight updates increases. Q=αexp[β(update#)]Id (Formula 20)

[0044] In the formula, β>0 represents the damping rate.

[0045] NN training procedure using training particle flow 3 shows a flow diagram of an embodiment of a NN training procedure using a training particle stream, as an example. The procedure includes initialization 320, training particle stream optimization 322, and prediction 324.

[0046] Initialization 320 involves a user selecting, or a computer automatically instantiating, functional forms for the likelihood and prior in operation 326. Initialization 320 includes sampling the selected prior distribution p(θ) in operation 328. The sampling initializes each realization of the NN optimized in optimization 322. If a multivariate normal distribution is selected as the prior in operation 326, the corresponding Hessian matrix has a simple analytical form. p(θ)~N(θ;μ,Γ) (Eq. 20)

number

[0047] where μ is the mean vector and Γ is the covariance matrix. The mean adds an initial offset to the values of the network parameters. For simplicity and to avoid adding spurious offsets, the mean can be set to 0; however, the offsets can be known and used. The value of Γ characterizes the initial spread and potential correlation of each network parameter. It can be chosen to be large enough to promote fast learning, yet small enough to prevent divergence in the network.

[0048] For the likelihood function, either Equation (7) or (9) can be used depending on the type of supervised learning task. The residual covariance in Equation (7) can be chosen similarly to the previous covariance (e.g., to facilitate learning but prevent divergence). The number of particles, N, can be chosen to be large enough to provide sufficient statistics about the data and avoid divergence of the covariance matrix.

[0049] The particle flow optimization 322 can be explained as follows. training set (x j, y T,j )∈D (operation 330 selects data from the training set): Make the current particle distribution equal to the previous particle distribution. Calculate the covariance of the particle prior distribution, Γ. Repeatedly loop over the scalar homotopy parameter λ = [0, 1] (operation 332) Each λ k ,k=1,2,…,N λ Regarding -1, Calculate the integration step size, δλ=λ k+1 -λ k Each particle {θ i}, for i=1,2,3...,N (operation 334) Use the particle value to input data x j over the network (operation 334) to obtain the prediction. p j =Λ θi (x j) Calculate the gradient and Hessian of the log-likelihood with respect to the network parameters (operation 336). Calculate the drift f and the diffusion matrix Q (operation 338). Update the particle state using numerical integration of the stochastic differential equation (SDE) (Equation 15) (operation 340).

number

[0050] In operation 330, pairs of input and output values are selected from the dataset. In operation 332, operation 322 repeats the discretization step of the homotopy parameter λ. In operation 334, a particle state among the N particle states from operation 328 is selected to pass the data through the selected NN. The result of operation 334 may be a prediction. In operation 336, gradients (derivatives) and Hessian matrices are calculated. Drift and diffusion are determined based on the gradients and Hessian matrices in operation 338. In operation 340, the particle states are updated.

[0051] The particle flow is integrated using numerical integration in equation (15). The scalar homotopy parameter λ = {λ k :k∈[1,N λ The interpolation of} can be based on linear, logarithmic, or adaptive scaling. The number of divisions N of the scalar homotopy parameter λ can strike a balance between integration accuracy and algorithmic efficiency. There are several different methods for the numerical integration of SDEs, but the Euler-Maruyama algorithm can be used for computational efficiency.

[0052] Prediction 324 can provide an output prediction in operation 342. A marginal probability distribution of the output prediction given a new input is determined in operation 344.

number

number

number

[0053] Marginalizing (21) over all network realizations θ yields the new input

number

number

number

[0054] Embodiments can evaluate equation (22) using Monte Carlo sampling of the posterior distribution p(θ|D), where the sampling is done over all grains, children {θ i}.

number

[0055] As can be seen, the prediction distribution is a marginalization of the posterior distribution due to the network predictions, which is the sum of the network parameters.

[0056] Mini-batch particle flow One issue with training particle flows is that they tend to be sensitive to each measurement (particle update). Outliers in the data input can cause the particle distribution to move in the wrong direction during a measurement update, leading to a significant increase in prediction accuracy. In some cases, the BNN during training cannot recover from such detours. Another issue with training particle flows is that training a particle flow BNN takes longer than training a standard NN (non-Bayesian NN), which can process a "mini-batch" of data for each weight update. In contrast, particle flow optimization procedures are formulated to process only one data point at a time, which precludes any kind of batch processing. This is because particle filters typically perform a state transition update at a specific time, followed by a measurement update conditioned on this specific time. The coupling between state transitions and measurements in time necessitates processing measurements one at a time. Therefore, particle flows, which until recently were used only in the context of particle filters, have no reason to process multiple measurements at a time.

[0057] FIG. 4 shows, as an example, a plot of accuracy versus measurement updates for a BNN trained on MNIST{0,1} for the training particle stream BNN described with respect to FIGS. 1-3. In this example, the accuracy of the BNN decreases in accuracy. The BNN recovers from these decreases in the example of FIG. 4.

[0058] As an example, Figure 5 shows a plot of accuracy versus measurement updates for a BNN trained on MNIST{0,1,2,3}. In the example in Figure 5, the accuracy of the BNN drops significantly between measurements 280 and 300. The BNN in the example in Figure 5 does not recover from this drop, and the network parameters remain at values that produce inaccurate predictions.

[0059] Next, we describe a mini-batch training particle flow BNN. This mini-batch particle flow BNN training formulation retains the core training particle flow optimization procedure but modifies the training particle flow framework to accommodate mini-batching of data. The use of mini-batches in stochastic gradient descent-type training is a well-established technique for training NNs. However, the use of mini-batches in training particle flow BNNs, or even particle flow itself, has not been done before, to our knowledge. Results on MNIST{0,1} using mini-batch particle flow show that the ability to process more data in a single update significantly improves the training speed and accuracy of the resulting model. The use of mini-batches in the training particle flow helps avoid the accuracy loss that occurs when using particle updates performed based on a single input. However, modifications to the training particle flow are required to allow the use of mini-batches in the training particle flow. Implementing these modifications reduces the training time of the BNN and improves the accuracy of the trained BNN.

[0060] N mb Data d={x,y T Consider a minibatch of d∈D. mb The joint posterior probability over the independently distributed data samples is:

number

[0061] In the formula, p(θ,λ│{x i ,y T,i}) is the posterior distribution of the i-th sample in the mini-batch, and λ is the scalar homotopy. Here, the same prior distribution of particles p(θ) is assumed for each sample in the mini-batch. Considering the logarithm of the joint posterior probability, we obtain:

number

[0062] Equation 25 can be rewritten as follows:

number

[0063] The mini-batch log-likelihood is

number

number

[0064] Comparing Equation 26, from Equation 16, the drift vector for the entire mini-batch update can be estimated as

number

[0065] This formula can be derived using the general approach described in Appendix A of the reference D.F. Crouse and C. Lewis, “Consideration of Particle Flow Filter Implementations and Biases,” Naval Research Lab, Washington, DC (2019), for a single measurement case.

[0066] It is important to point out that the drift vector of a mini-batch (Equation 27) is not equal to the sum of the drift vectors (Equation 18) over all samples in the mini-batch.

number

[0067] Gradient Operator

number

number

number

number

[0068] NN training procedure using mini-batch training particle flow 6 shows, by way of example, a flow diagram of an embodiment of a mini-batch training particle flow technique. This technique is similar to the training particle flow technique shown in FIG. 3, with minor modifications to particle flow optimization 322, resulting in mini-batch particle flow optimization 658. Particle flow optimization 658 includes selecting a mini-batch of data from a training set in operation 330, iterating through a homotopy discretization step in operation 332, and determining a batch of predictions in operation 334.

[0069] Various Python ML and AI libraries, such as Pytorch® and TensorFlow®, do not store individual gradients for each sample in a mini-batch. Instead, they sum or average gradients by default to increase efficiency and reduce memory usage.

[0070] To accommodate these libraries, the mini-batch particle flow training optimization 658 can be tuned to evolve the mean of the logarithm of the joint posterior probabilities.

number

[0071] In the formula, logp MB (θ)=N mb Since logp(θ), p(θ) is the prior distribution of the particles before the batch update, which physically corresponds to taking the geometric mean of the posterior probabilities of each sample within the mini-batch.

number

[0072] The drift vector determined in operation 664 is:

number

[0073] logL MB The average gradient of the objective function is logL MB This can be computed in most ML libraries by setting , however computing the mean of the Hessian of the mini-batch log-likelihood deserves careful consideration to ensure that the averaging is performed at the appropriate time.

[0074] The training particle stream can compute the Hessian of the log-likelihood using the Gauss-Newton Hessian approximation.

number

[0075] In the formula, p m is the mth component of the network prediction, and r, s are the Hessian exponents. The Hessian mean of the mini-batch log-likelihood contains the mean of the Hessians for each sample in the mini-batch.

number

[0076] This is the Jacobian term,

number

number

[0077] Mini-batch particle flow optimization 658 can be summarized as follows: For each mini-batch d = {x, yT}, d∈D of data in the training set (operation 660 selects a mini-batch of data from the training set) Make the current particle distribution equal to the previous particle distribution. Calculate the covariance Γ of the particle prior distribution, Repeatedly loop over the scalar homotopy parameter λ = [0, 1] (operation 332) Each λ k ,k=1,2,…,N λ Regarding -1, Calculate the integration step size, δλ=λ k+1 -λ k Each particle {θ i}, for i=1,2,3...,N (operation 334), Using the particle values, input data x={x j} over the network (operation 661) to obtain a batch of predictions. {p j}=Λθ i ({x j}) Compute the average gradient and Hessian of the mini-batch log-likelihood with respect to the network parameters (operation 662) Calculate the drift f and the diffusion matrix Q (operation 664). Update the particle state (operation 666) using numerical integration of the stochastic differential equation (SDE) (Equation 15).

number

[0078] Classification results for the digit subset {0,1} from the Modified National Institute of Standards and Technology (MNIST) database. The MNIST database was created by Yann LeCun, Corinna Cortes, and Christopher JC Burges using images from two separate NIST databases. The MNIST database can be accessed at http: / / yann.lecun.com / exdb / mnist / .

[0079] A convolutional neural network (CNN) architecture was instantiated, consisting of two convolutional layers with four filters each, followed by a dense output layer. This network has 286 network parameters. 100 normally distributed particles with initial covariance Γ = 0.04Id were also instantiated. Numerical integration of the flow was performed using N λ The calculation was performed using a logarithmic step size of .times. ... [Table 1]

[0080] Mini-batch training particle flow is MB =1, N MB = 2, and NMB This was implemented to train a BNN with a batch update of =16. When the mini-batch size is >1, the batch contains evenly distributed classes (e.g., a batch size of 16 contains 8 of class 0 and 8 of class 1). From these training examples, the smoothness of the average log-likelihood increases with mini-batch size. Furthermore, the deviation of each particle's log-likelihood from the mean decreases as the batch size increases. This means that individual particles are less susceptible to outliers than when no mini-batches are used (batch size 1).

[0081] Figure 7 shows the N MB =1, N MB = 2, and N MB Figure 1 shows a plot of network accuracy with increasing batch updates for a batch size of ∑ = 16. From this plot, it is clear that a batch size of 16 achieves and maintains the highest accuracy, while a batch size of 1 tends to decrease accuracy. From this study, it is clear that using a mini-batch worth of data for each parameter update improves both the training speed and the robustness of the approach.

[0082] AI is a field concerned with developing decision-making systems to perform cognitive tasks that previously required living actors, such as humans. A neural network (NN) is a computational structure loosely modeled on biological neurons. In general, a NN encodes information (e.g., data or decisions) through weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the basis for many AI applications, such as speech recognition.

[0083] Many NNs are represented as a matrix of weights corresponding to the connections they model. NNs frequently operate by accepting data into a series of input neurons that have many outgoing connections to other neurons. With each pass between neurons, the corresponding weight modifies the input and is passed to an activation function. The result of the activation function is then sent to another neuron further down the NN graph. The process of weighting and processing by the activation function continues until it reaches an output neuron, and the result of the ANN processing constitutes the pattern and value of the output neuron.

[0084] The correct operation of most NNs depends on accurate weights. However, NN designers typically do not know which weights will work for a particular application. NN designers typically select a large number of neuron layers or specific connections between layers, including circular connections. A training process can be used to determine appropriate weights by selecting initial weights. In some instances, the initial weights may be selected randomly. Training data is input into the NN, and the results are compared to an objective function that provides an error indicator. The error indicator is a measure of how inaccurate the NN's results are compared to the expected results. This error is then used to modify the weights. Over many iterations, the weights collectively converge, and the operational data is encoded into the NN. This process is sometimes called optimizing the objective function (e.g., cost function or loss function), thereby minimizing the cost or loss.

[0085] Gradient descent is a common technique for optimizing a particular objective (or loss) function. The gradient of a scalar field (e.g., a vector of partial derivatives) gives the direction of the steepest increase in this objective function. Therefore, adjusting a parameter by a small amount in the opposite direction generally decreases the objective function. After a sufficient number of iterations, the parameters tend to approach a minimum. In some implementations, the learning rate (e.g., step size) is fixed across all iterations. However, small step sizes tend to take longer to converge, while large step sizes can result in oscillations around the minimum or other undesirable behavior. A variable step size is usually introduced to achieve faster convergence without the drawbacks of large step sizes.

[0086] After a forward pass of input data through a neural network, backpropagation provides a parsimonious approach to evaluating the gradient of the objective function with respect to the network parameters. The final output of the network is constructed by combining the operations of each layer. This requires a chain rule to calculate the gradient of the objective function. Backpropagation starts at the final layer and works backward to the input layer, exploiting a recursive relationship between the derivative of the objective with respect to the layer output and the corresponding quantity from the previous layer. This recursive relationship eliminates the redundancy of evaluating the entire chain rule of the derivative of the objective with respect to each parameter. Any well-known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0087] 8 is a block diagram of an example environment including a system for NN training, according to an embodiment. The system can assist in training cybersecurity solutions according to one or more embodiments. The system includes an artificial NN (ANN) 805 that is trained using processing nodes 810. The processing nodes 810 can be central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), application specific integrated circuits (ASICs), or other processing circuits. In an example, multiple processing nodes can be used to train different layers of the ANN 805, or even different nodes 807 within a layer. Thus, a set of processing nodes 810 is configured to perform the training of the ANN 805.

[0088] The set of processing nodes 810 is configured to receive a training set 815 for the ANN 805. The ANN 805 comprises a set of nodes 807 (shown as rows of nodes 807) arranged in layers and a set of inter-node weights 808 (e.g., parameters) between the nodes in the set of nodes. In an example, the training set 815 is a subset of the full training set, where the subset may allow processing nodes with limited storage resources to participate in training the ANN 805.

[0089] The training data may include multiple numerical values representing domains such as words, symbols, other parts of speech, etc. Each training or input 817 value to be classified as the ANN 805 is trained on is provided to a corresponding node 807 in the first or input layer of the ANN 805. The values propagate through the layers and are varied by the objective function.

[0090] As described above, the set of processing nodes is configured to train the neural network to create a trained neural network. For example, once trained, data input to the ANN generates a valid classification 820 (e.g., input data 817 is assigned to a category). The training performed by the set of processing nodes 807 is iterative. In an example, each iteration of training the neural network is performed independently across layers of the ANN 805. Thus, two different layers may be processed in parallel by different members of the set of processing nodes. In an example, different layers of the ANN 805 are trained on different hardware. Members of different members of the set of processing nodes may be located in different packages, housings, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently across nodes in the set of nodes. An example of this is additional parallelism, where individual nodes 407 (e.g., neurons) are trained independently. In an example, the nodes are trained on different hardware.

[0091] FIG. 9 shows, by way of example, a block diagram of one embodiment of a machine in the exemplary form of a computer system 900 upon which instructions may be executed to cause the machine to perform any one or more of the methodologies described herein. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a network switch, or a network bridge, or any machine capable of executing instructions (sequentially or otherwise) that specify actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methodologies described herein.

[0092] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 904, and a static memory 906, which communicate with each other via a bus 908. The computer system 900 may further include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 900 also includes an alphanumeric input device 912 (e.g., a keyboard), a user interface (UI) navigation device 914 (e.g., a mouse), a mass storage device 916, a signal generator 918 (e.g., a speaker), a network interface device 920, and a wireless 930, such as Bluetooth, WWAN, WLAN, and NFC, to enable the application of security controls to such protocols.

[0093] The mass storage device 916 includes a machine-readable medium 922 on which is stored one or more sets of instructions and data structures (e.g., software) 924 that embody or are utilized by any one or more of the methods or functions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904 and / or within the processor 902 during execution thereof by the computer system 900, the main memory 904 and the processor 902 also constituting machine-readable media.

[0094] While the exemplary embodiment illustrates machine-readable medium 922 as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more instructions or data structures. The term "machine-readable medium" should also be interpreted to include any tangible medium that can store, encode, or carry instructions for execution by a machine, causing the machine to perform any one or more of the methods of the present invention, or that can store, encode, or carry data structures utilized by or associated with such instructions. The term "machine-readable medium" should accordingly be interpreted to include, but is not limited to, solid-state memory, and optical and magnetic media. Specific examples of machine-readable media include, by way of example, non-volatile memory, including semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0095] The instructions 924 may also be transmitted or received over a communications network 926 using a transmission medium. The instructions 924 may be transmitted using a network interface device 920 and any one of several well-known transfer protocols (e.g., HTTP). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet, a cellular network, a plain old telephone service (POTS) network, and wireless data networks (e.g., WiFi networks and WiMax networks). The term "transmission medium" shall be interpreted to include any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communications signals or other intangible media for facilitating communication of such software.

[0096] Additional Notes and Examples Example 1 includes a method for training a Bayesian neural network (BNN) using batch inputs and operating the trained BNN, the method including initializing particles such that each particle individually represents a pointwise value of each NN parameter of the NN and the particles collectively represent a distribution of the parameters of the BNN; optimizing the particles based on the batches of inputs using a minimal batch training particle stream to result in optimized distributions of the parameters; determining a predictive distribution using the optimized distributions of the parameters and predictions from each of the NNs; and resulting in marginal distributions representing the predictive distributions.

[0097] In Example 2, Example 1 can further include, wherein the mini-batch training particle stream includes iteratively evolving values of the network parameters based on log homotopy.

[0098] In Example 3, Example 2 can further include, wherein the mini-batch training particle stream includes evolving a mean of the logarithms of the joint posterior probabilities.

[0099] In Example 4, at least one of Examples 2-3 can further include determining, for each batch in the training set, a geometric mean of the posterior probabilities of each input in the batch.

[0100] In Example 5, at least one of Examples 3-4 may further include, for each batch in the training set, evolving the means may include averaging the Hessian matrix of each input in the batch.

[0101] In Example 6, Example 5 can further include, wherein averaging the Hessian matrices includes storing corresponding Hessian and Jacobian terms for each input in the batch.

[0102] In Example 7, Example 6 can further include averaging the Hessian matrix includes determining, for each input in the batch, a product of matrix terms of the Hessian matrix and the Jacobian terms in a Gauss-Newton approximation, producing a product result.

[0103] Example 8 includes a system including a processing circuit and a memory coupled to the processing circuit, the memory including instructions that, when executed by the processing circuit, cause the processing circuit to perform the method of any of Examples 1-7.

[0104] Example 9 includes a non-transitory machine-readable medium containing stored instructions that, when executed by a machine, cause the machine to perform the method of one of Examples 1-8.

[0105] While the embodiments have been described with reference to specific exemplary embodiments, it will be apparent that various modifications and changes can be made to these embodiments without departing from the broader spirit and scope of the present invention. Accordingly, the specification and drawings are to be regarded in an illustrative and not a restrictive sense. The accompanying drawings, which form a part of this specification, show by way of illustration, and not by way of limitation, specific embodiments in which the present subject matter may be practiced. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Since other embodiments may be utilized and derived therefrom, structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the detailed description is not to be construed in a limiting sense, but the scope of various embodiments is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A method for training a Bayesian Neural Network (BNN) using batch inputs and operating the trained BNN, comprising: initializing the particles such that each particle individually represents a pointwise value of a respective NN parameter of a NN, and said particles collectively represent a distribution of parameters of said BNN; optimizing said particles based on batches of inputs using a mini-batch training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and providing a marginal distribution representing the predictive distribution.

2. The method of claim 1 , wherein mini-batch training particle flow comprises iteratively evolving values of the network parameters based on log homotopy.

3. The method of claim 2 , wherein the mini-batch training particle stream comprises evolving an average of the logarithm of the joint posterior probability over the data samples in the mini-batch.

4. The method of claim 2 , wherein the mini-batch training particle flow comprises determining, for each batch in a training set, the geometric mean of the posterior probabilities of each input in the batch.

5. The method of claim 3 , wherein evolving the means comprises, for each batch in a training set, averaging the Hessian matrix of each input in the batch.

6. The method of claim 5 , wherein averaging the Hessian matrices comprises storing corresponding Hessian and Jacobian terms for each input in the batch.

7. Averaging the Hessian matrix for each input in the batch, determining the product of the Hessian terms and the Jacobian terms in a Gauss-Newton approximation, and producing a product; and 7. The method of claim 6, further comprising averaging the results of said products to produce said average Hessian matrix.

8. 1. A non-transitory machine-readable medium containing instructions that, when executed by a machine, cause the machine to perform operations, the operations including: initializing the particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN, and said particles collectively represent a distribution of parameters of the BNN; optimizing said particles based on batches of inputs using a mini-batch training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and yielding marginal distributions that represent the prediction distribution.

9. 10. The non-transitory machine-readable medium of claim 8, wherein mini-batch training particle flow comprises iteratively evolving values of the network parameters based on log homotopy.

10. 10. The non-transitory machine-readable medium of claim 9, wherein the mini-batch training particle stream comprises evolving an average of logarithms of joint posterior probabilities over data samples in a mini-batch.

11. 10. The non-transitory machine-readable medium of claim 9, wherein the mini-batch training particle flow includes determining, for each batch in a training set, a geometric mean of the posterior probabilities of each input in the batch.

12. 11. The non-transitory machine-readable medium of claim 10, wherein evolving the means comprises, for each batch in a training set, averaging a Hessian matrix for each input in the batch.

13. 13. The non-transitory machine-readable medium of claim 12, wherein averaging the Hessian matrices comprises storing, for each input in the batch, corresponding Hessian and Jacobian terms.

14. Averaging the Hessian matrix for each input in the batch, determining the product of the Hessian terms and the Jacobian terms in a Gauss-Newton approximation, and producing a product; and 14. The non-transitory machine-readable medium of claim 13, comprising averaging the results of the products to produce the average Hessian matrix.

15. 1. A system comprising: a processing circuit; a memory coupled to the processing circuitry and containing instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations, the operations including: initializing the particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN, and said particles collectively represent a distribution of parameters of the BNN; optimizing said particles based on batches of inputs using a mini-batch training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and providing marginal distributions that represent the prediction distribution.

16. 16. The system of claim 15, wherein the mini-batch training particle flow comprises iteratively evolving values of the network parameters based on log homotopy.

17. 17. The system of claim 16, wherein the mini-batch training particle stream comprises evolving an average of the logarithm of the joint posterior probability over the data samples in the mini-batch.

18. 17. The system of claim 16, wherein the mini-batch training particle flow includes determining, for each batch in a training set, a geometric mean of the posterior probabilities of each input in the batch.

19. 20. The system of claim 17, wherein evolving the means comprises, for each batch in a training set, averaging the Hessian matrix of each input in the batch.

20. Averaging the Hessian matrix storing, for each input in the batch, corresponding Hessian and Jacobian terms; for each input in the batch, determining the product of the Hessian terms and the Jacobian terms in a Gauss-Newton approximation, and producing a product; and 20. The system of claim 19, further comprising averaging the results of the products to produce the average Hessian matrix.

Images