Training BNNs using mini-batch particle flows

Mini-batch particle flow training addresses the computational inefficiencies and robustness issues of BNNs by allowing mini-batch processing, improving accuracy and efficiency in training BNNs for multiple classes.

JP2024541876AActive Publication Date: 2024-11-13RAYTHEON CO
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024523578
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-10-25
Filing Date
2022-10-25
Publication Date
2024-11-13
Estimated Expiration
2042-10-25

AI Technical Summary

Technical Problem

Existing training methods for Bayesian neural networks (BNNs) are computationally expensive and require large datasets, and current implementations of particle flow BNNs lack robustness and are difficult to train for more than two classes.

Method used

The use of mini-batch particle flow training, which modifies the training particle flow framework to accommodate mini-batch processing, allowing for the evolution of network parameters using mini-batches to improve training efficiency and robustness.

Benefits of technology

Mini-batch particle flow training reduces computational time and enhances the accuracy and robustness of BNNs, enabling them to handle multiple classes effectively with fewer training samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541876000001_ABST
    Figure 2024541876000001_ABST
Patent Text Reader

Abstract

Devices, systems, and methods for training a Bayesian Neural Network (BNN) using mini-batch particle streams are discussed herein. A method for training a Bayesian Neural Network (BNN) using batch inputs and operating the trained BNN may include initializing particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN and the particles collectively represent a distribution of the parameters of the BNN, optimizing the particles based on the batches of inputs using a mini-batch training particle stream to result in an optimized distribution of the parameters, determining a predictive distribution using the optimized distribution of the parameters and predictions from each of the NNs, and resulting in marginal distributions representing the predictive distributions.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Claiming priority This application claims the benefit of priority to U.S. Patent Application No. 17 / 509,278, filed October 25, 2021, which is incorporated by reference in its entirety herein.

[0002] The embodiments discussed herein relate to devices, systems, and methods for training a Bayesian Neural Network (NN) using mini-batch particle flows. [Background technology]

[0003] Most NNs perform point estimates without a direct uncertainty metric or confidence measure. Standard NNs can also perform relatively poorly on open sets. Bayesian NNs (BNNs) learn a statistical distribution of weights, providing a statistical environment in which the uncertainty and confidence of decisions can be determined. Existing training methods for BNNs include Hamiltonian Monte Carlo, variational inference (both Monte Carlo and deterministic), stochastic backpropagation, and standard particle filters. These training methods are computationally expensive and require relatively large amounts of data for training. [Brief description of the drawings]

[0004] [Figure 1] As an example, we present a block diagram comparing a standard DL architecture with a BNN. [Diagram 2] As an example, a diagram of an embodiment of a NN is shown. [Diagram 3] As an example, a flow diagram of an embodiment of a NN training procedure using a training particle stream is shown. [Figure 4] As an example, we show a plot of accuracy versus measurement update for a BNN trained on MNIST{0,1} using a training particle stream. [Diagram 5] As an example, we show a plot of accuracy versus measured update for a BNN trained on MNIST{0,1,2,3}. [Figure 6] As an example, a flow diagram of an embodiment of the mini-batch training particle stream technique is shown. [Figure 7] We show plots of network accuracy with increasing batch updates for batch sizes of 1, 2, and 16, respectively. [Figure 8] FIG. 1 is a block diagram of an example environment including a system for NN training, according to an embodiment. [Figure 9] By way of example, a block diagram of an embodiment of a machine of an exemplary form of a computer system upon which instructions may be executed to cause the machine to perform any one or more of the methods described herein is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Particle flow has recently been modified to be used in the context of deep learning (DL). Such a particle flow approach for DL ​​training is called "training particle flow". Training particle flow can train Bayesian neural networks (BNNs). A BNN trained using training particle flow can be called a "particle flow BNN". Particle flow BNN architectures show high prediction accuracy with few training samples and a strong ability to measure prediction uncertainty using the variance of predictions made by the BNN. A case study using the MNIST class {0,1} showed this to be the case. However, current implementations of particle flow BNNs tend to lack robustness and can have difficulty training more than two classes, as described below. See for example Figures 4 and 5. The BNN and training particle flow are described. Then, a description of mini-batch particle flow training is provided. Training a BNN using mini-batch particle flow results in a more robust BNN that is easier to train more than two classes than alternative single-input particle flow training approaches.

[0006] FIG. 1 shows, as an example, a block diagram of a standard DL architecture 102 versus a BNN 104. Standard DL architectures make point estimates of model predictions (outputs 108) and network parameters (node ​​weights 112). Such DL architectures 102 do not provide a direct means for quantifying uncertainty. Instead, standard DL architectures 102 rely on indirect methods for estimating uncertainty. Common methods include the use of entropy and confidence scores and functions, as well as application-specific methods. Bayesian DL architectures 104 and statistical methods tend to provide a more natural context for quantifying uncertainty.

[0007] BNNs have been researched since at least 1992 and are a growing field. BNN technology uses Bayes' Theorem as a guide to solve the posterior probability distribution of the weights in a NN (the distribution of the nodes 114). The computational difficulty of solving Bayes' Theorem for DL ​​tasks has led to the development of numerous approaches to estimate the posterior distribution of the weights in a NN. Well-known approaches for BNN optimization include Hamiltonian Monte Carlo, Monte Carlo Variational Inference, Deterministic Variational Inference, and Stochastic Backpropagation (PBP).

[0008] The output 110 of the BNN 104 is a distribution per class compared to the output of the NN 102, which provides a score per class as output 108. The distribution of the output 110 may be a natural consequence of using a distribution ("dist.") instead of a scalar weight as in node 112 to represent the activation function of node 114.

[0009] Deep ensembles and probabilistic regularization methods offer alternative approaches towards estimating uncertainty in DL. Although these methods do not optimize Bayes' theorem, they provide a statistical environment for computing prediction uncertainty at a fraction of the computational cost of BNNs. These statistical methods are used in a variety of applications to quantify uncertainty.

[0010] A common theme in existing Bayesian NNs and statistical approaches is very large training sets and training for thousands of epochs. However, real-world datasets are typically sparse, and training a general NN with thousands of parameters may be insufficient. Embodiments provide a NN architecture that can quantify uncertainty and also perform robustly within the limits of sparse datasets and for open set problems.

[0011] The embodiment uses a modified form of particle flow technique that is commonly used in particle filters, but is modified and reused to train BNNs. The modified form of the particle flow method is referred to herein as "training particle flow". Particle flow is a method for optimizing Bayes' theorem, and has (so far) only been used in the context of particle filtering. Numerical experiments of particle flow in the context of particle filtering have shown that particle flow can reduce the computational complexity by orders of magnitude compared to standard particle filters and other state-of-the-art algorithms for the same filter accuracy. Furthermore, particle flow can reduce the filter error by orders of magnitude compared to extended Kalman filters and other state-of-the-art algorithms for difficult nonlinear non-Gaussian problems.

[0012] Recently, particle methods for optimizing NNs have emerged, but optimizing BNNs using particle flows has not been done before to the best of our knowledge. Results from BNNs trained to perform classification tasks using MNIST{0,1} show high prediction accuracy with very few training samples. Furthermore, BNNs trained to perform classification tasks had a strong ability to measure prediction uncertainty using the variance of the network predictions.

[0013] particle flow Consider a system with internal state s and measurements m. Bayes' theorem relates the posterior probability of a state given a measurement p(s│m) to the prior distribution p(s) on the state and the likelihood p(m│s) of the measurement given the state according to:

number

[0014] where p(m) = ∫p(m│s)p(s)ds is a proof that acts as a normalizing constant. The measurement m is a quantity that helps characterize what the internal state is like or what the internal state will be like. Given a general Markov process with a sequence of noisy measurements {m}, particle filters provide a way to estimate the system's internal state(s) {s} using Bayes' theorem as a guideline.

[0015] Particle flow is a method used in the context of particle filtering to estimate the optimal posterior distribution of the internal state for each measurement. Particle flow optimizes Bayes' theorem by evolving a prior distribution to a posterior distribution along a log homotopy.

number

[0016] Two continuous functions in their respective spaces are called homotopic if one can be "continuously deformed" into the other. A homotopy exists between the functions that can be deformed in this way.

[0017] where K(λ,m) = ∫p(m│s). λ p(s)ds normalizes the posterior distribution p(s,λ│m) for each λ. The scalar homotopy parameter λ=[0,1] evolves the distribution from the prior to the posterior for a particular measurement m. Each particle represents a single realization of the system's internal state s. The flow of particles along the log homotopy is described by the following stochastic differential equation (SDE):

number

[0018] During the ceremony,

number

[0019] The evolution of the posterior distribution of particles follows the Fokker-Planck equation (with the diffusion squared matrix Q ij =Σ k B ik B jk The term can be characterized by:

number

[0020] Note that the gradients and derivatives written in (4) are in Cartesian coordinates, but (4) can be used in any suitable Cartesian coordinate system by appropriate transformation of the partial derivatives.

[0021] Drift Speed

number

number

number

[0022] During the ceremony,

number

[0023] The geodesic solution assumes a Gromov solution for drift velocity and no diffusion (i.e. Q=0), which does not simultaneously satisfy equations 2 and 4. The zero curvature solution assumes that the particle does not accelerate as it changes λ and there is no diffusion term (i.e. Q=0). The drift velocity of this solution is proportional to the Gromov drift velocity.

[0024] DL and supervised learning tasks DL is a branch of ML that uses a series of layers of nodes to learn higher-order representations of data for supervised, semi-supervised, or unsupervised learning tasks. Although the described embodiments focus on supervised learning tasks, the embodiments may be applied to any learning task for which a likelihood function can be defined.

[0025] In supervised learning tasks, deep neural networks (DNNs) can be used to learn relationships between input and output data for regression or classification. For regression tasks, NNs can predict a dependent variable p that is causally related to the input data. For classification tasks, the network can predict the probability p of a particular class. _j , the network predicts the corresponding truth class or truth value y of the input data. The word "probability" is a bit of a misnomer here. Classification tasks often use a SoftMax activation function in the output layer to generate a vector whose elements sum to 1. This output represents a set of class probabilities, but these "probabilities" are not necessarily well-calibrated to the actual accuracy of the network. In this sense, the output probabilities can be more accurately understood as a normalized score for each class. During training, the corresponding truth class or truth value y of the input data is calculated. T You can evaluate the NN predictions against and adjust the network weights using your chosen optimization scheme.

[0026] The “likelihood” function is used in many gradient-based optimization methods in DL. For regression tasks, the NN weights θ = {θ i} and the truth variable y given the network prediction p T The likelihood L of is usually modeled by a normal distribution.

number

[0027] where Σ is the truth y T is the covariance matrix that scales the error between y and the prediction p, and the index k is T This likelihood function represents the dimension of the prediction p and the truth y T We assume Gaussian white noise discrepancy between the vectors. The corresponding log-likelihood is given by

number

[0028] This is reminiscent of the mean squared error (MSE) when Σ is the identity matrix.

[0029] For classification tasks, a categorical distribution is the predicted class probability p j , j ∈ [1n クラス ] and the true class y of the input given the NN weights θ T represents the likelihood L of

number

[0030] where y T ={y Tj} is a one-hot encoded vector of the image truth classes, or a non-binary vector summing to 1 if we are using soft labels. The corresponding log-likelihood is the negative of the cross-entropy function.

number

[0031] Mapping particle flows for training BNNs 2 shows, by way of example, a diagram of an embodiment of a NN 200. The NN 200 includes L layers of nodes. θ contains the network parameters θ={θ j}={W 1 ,b1 ,W 2 ,b 2 ,…,W L ,b L}. NN200 is a set of N パラメータ parameters, and the network parameters θ are N パラメータ dimensional vector θ∈R Nパラメータ become.

[0032] In a typical supervised learning task, we have a set of training data D={X,Y T} to train NN200, where X={x j} is the set of all inputs, and Y T ={y T,j} is the corresponding set of all truth values ​​given X. Each NN prediction p j The data

number

number

[0033] In the formula, σ represents the activation function of each layer of nodes, and g represents the affine transformation at the nodes of each layer.

[0034] The goal of a BNN is to use Bayes' theorem to learn the optimal posterior distribution of the network parameters p(θ│D) given the data.

number

[0035] The right-hand side of equation (12) is the posterior probability

number

number

[0036] The embodiment maps the particle flow used in the particle filtering context to the DL context, and defines the network parameters θ and truth values ​​Y T By equating the internal state {s} and measurements {m} in equation (1) for j The training particle stream represents a single realization of the NN’s network parameters {θ j The likelihood of each measurement m given the internal state s is expressed as the likelihood Y of the truth value given the prediction P of the network 200 based on the input data X. T In the DL context, each particle is assigned a network parameter {θ j} represents a single realization of

[0037] The mapping of particle streams to training particle streams used in the particle filtering context is mathematically described in equations (13) through (15).

number

number

[0038] The log homotopy constraint given in equation (3) is logp(θ,λ│{X,Y T})=λlogL(Y T │θ,X)+logp(θ)-logp(λ,│X)Y T (Formula 16)

[0039] Here, a scalar homotopy parameter λ is added to the notation for the posterior distribution and the normalization factor to specify the variance of these terms in λ. The corresponding gradients and derivatives written in equation (4) are now with respect to the network parameters for the training particle flow.

number

[0040] Equation 18 shows the mathematical expression of the drift velocity in the training particle stream using the Gromov equation.

number

[0041] The Gromov equation generalizes well to architectures with L ≥ 1 layers, various activation functions, and arbitrary prior and likelihood function forms. However, equation (18) is not sufficient to justify the use of linear activation functions.

number

[0042] Equation 19 provides the mathematical expression of the constant diffusion matrix used in particle flow training. Q=αId (Equation 19)

[0043] where α ∈ R0 is a positive real number and Id is an identity matrix. A constant diffusion matrix can help provide numerical stability. Additionally, adding a small amount of noise can prevent the network training from getting stuck in local minima. A small exponential decay factor can also be added to the diffusion matrix to reduce the effect of noise as the number of weight updates increases. Q=αexp[β(update#)]Id (Formula 20)

[0044] In the formula, β>0 represents the damping factor.

[0045] NN training procedure using training particle flow 3 shows, by way of example, a flow diagram of an embodiment of a NN training procedure using a training particle stream, which includes initialization 320, training particle stream optimization 322, and prediction 324.

[0046] Initialization 320 involves a user selecting, or a computer automatically instantiating, the functional forms of the likelihood and prior in operation 326. Initialization 320 involves sampling the selected prior distribution p(θ) in operation 328. The sampling initializes each realization of the NN optimized in optimization 322. If a multivariate normal distribution was selected as the prior distribution in operation 326, the corresponding Hessian matrix will have a simple analytical form. p(θ)~N(θ;μ,Γ) (Eq. 20)

number

[0047] where μ is the mean vector and Γ is the covariance matrix. The mean adds an initial offset to the values ​​of the network parameters. For simplicity and to avoid adding spurious offsets, the mean can be set to 0; however, the offsets can be known and used. The value of Γ characterizes the initial spread and potential correlations of each network parameter. The value can be chosen to be large enough to promote fast learning, yet small enough to prevent divergence in the network.

[0048] For the likelihood function, either equation (7) or (9) can be used depending on the type of supervised learning task. The residual covariance in equation (7) can be chosen similarly to the previous covariance (e.g., to facilitate learning but avoid divergence). The number of particles N can be chosen to be large enough to provide sufficient statistics on the data and avoid divergence of the covariance matrix.

[0049] Particle flow optimization 322 can be explained as follows. Training set (x j, y T,j ) ∈ D (operation 330 selects data from the training set): Make the current particle distribution equal to the previous particle distribution. Calculate the covariance of the particle prior distribution, Γ. Repeatedly loop over the scalar homotopy parameter λ = [0, 1] (operation 332) Each λ k , k=1,2,…,N λ Regarding -1, Calculate the integration step size, δλ=λ k+1 -λ k Each particle {θ i}, for i = 1, 2, 3 ..., N (operation 334) Use the value of the particle to input data x j over the network (operation 334) to obtain the prediction. p j =Λ θi (x j) Calculate the gradient of the log-likelihood with respect to the network parameters and the Hessian (operation 336). Calculate the drift f and the diffusion matrix Q (operation 338). Update the particle state using numerical integration of the stochastic differential equation (SDE) (Equation 15) (operation 340).

number

[0050] In operation 330, a pair of input and output values ​​is selected from the data set. In operation 332, operation 322 repeats the discretization step of the homotopy parameter λ. In operation 334, a particle state of the N particle states from operation 328 is selected to pass the data through the selected NN. The result of operation 334 may be a prediction. In operation 336, the gradient (derivative) and Hessian are calculated. Drift and diffusion are determined based on the gradient and Hessian in operation 338. In operation 340, the particle state is updated.

[0051] The particle flow is integrated using numerical integration to integrate equation (15). The scalar homotopy parameter λ = {λ k :k∈[1,N λ The interpolation of} can be based on linear, logarithmic, or adaptive scaling. The number of divisions of the scalar homotopy parameter N λ can be balanced between integration accuracy and algorithmic efficiency. There are several different methods for the numerical integration of SDEs, but the Euler-Maruyama algorithm can be used for computational efficiency.

[0052] Prediction 324 may provide an output prediction in operation 342. A marginalized probability distribution of the output prediction given the new input is determined in operation 344.

number

number

number

[0053] Marginalizing (21) over all network realizations θ gives the new input

number

number

number

[0054] Embodiments can evaluate equation (22) using Monte Carlo sampling of the posterior distribution p(θ│D), where the sampling is done over all grains, children {θ i}.

number

[0055] As can be seen, the predictive distribution is the marginalization of the posterior distribution due to the network predictions, which is the sum over the network parameters.

[0056] Mini-batch particle flow One problem with training particle flows is that it tends to be sensitive to each measurement (particle update). Outliers in the data inputs can push the particle distribution in the wrong direction during a measurement update, leading to a large increase in prediction accuracy. In some cases, the BNN in training cannot recover from such detours. Another problem with training particle flows is that training a particle flow BNN takes longer than training a standard NN (non-Bayesian NN) that can process a "mini-batch" of data for each weight update. In contrast, the optimization procedure in particle flows is formulated to process only one data point at a time, which precludes any kind of batch processing. This is because particle filters typically have a state transition update at a specific time, followed by a measurement update conditioned on this specific time. The coupling between state transitions and measurements in time requires that measurements be processed one at a time. Thus, there is no reason for particle flows, which until recently were only used in the context of particle filters, to process multiple measurements at a time.

[0057] FIG. 4 shows, by way of example, a plot of accuracy versus measurement updates for a BNN trained on MNIST{0,1} for the training particle stream BNN described with respect to FIGS. 1-3. In this example, the accuracy of the BNN decreases in accuracy. The BNN recovers from these decreases in the example of FIG.

[0058] As an example, Figure 5 shows a plot of accuracy versus measurement updates for a BNN trained on MNIST{0,1,2,3}. In the example of Figure 5, the accuracy of the BNN drops significantly between measurements 280 and 300. The BNN in the example of Figure 5 does not recover from this drop, and the network parameters remain at values ​​that produce less accurate predictions.

[0059] Next, we describe a mini-batch training particle-flow BNN. This mini-batch particle-flow BNN training formulation retains the core training particle-flow optimization procedure but modifies the training particle-flow framework to accommodate mini-batching of data. The use of mini-batches in stochastic gradient descent type training is a well-established technique for training NNs. However, the use of mini-batches in training particle-flow BNNs, or even particle-flow itself, has not been done before to the best of our knowledge. Results on MNIST{0,1} using mini-batch particle-flow show that the speed and accuracy of the training of the resulting model is significantly improved because more data can be processed in a single update. The use of mini-batches in the training particle-flow helps to avoid the loss of accuracy that occurs when using particle updates that are performed based on a single input. However, the training particle-flow needs to be modified to allow the use of mini-batches in the training particle-flow. Implementing these modifications reduces the training time of the BNN and improves the accuracy of the trained BNN.

[0060] N mb Data with samples d={x,y T Consider a minibatch of d∈D. For all N mb The joint posterior probability over the independently distributed data samples is:

number

[0061] In the formula, p(θ,λ│{x i ,y T,i}) is the posterior distribution of the i-th sample in the mini-batch, and λ is the scalar homotopy. Here, we assume the same prior distribution of particles p(θ) for each sample in the mini-batch. Considering the logarithm of the joint posterior probability, we obtain

number

[0062] Equation 25 can be rewritten as follows:

number

[0063] The mini-batch log-likelihood is

number

number

[0064] Comparing with Equation 26, from Equation 16, the drift vector across the mini-batch update can be estimated as

number

[0065] This formula can be derived using the general approach described in Appendix A of the following reference: D.F. Crouse and C. Lewis, “Consideration of Particle Flow Filter Implementations and Biases,” Naval Research Lab, Washington, DC (2019), for a single measurement case.

[0066] It is important to point out that the drift vector of a mini-batch (Equation 27) is not equal to the sum of the drift vectors (Equation 18) over all samples in the mini-batch.

number

[0067] Gradient Operator

number

number

number

number

[0068] NN training procedure using mini-batch training particle flow 6 shows, by way of example, a flow diagram of an embodiment of a mini-batch training particle flow technique. This technique is similar to the training particle flow technique shown in FIG. 3, with minor modifications to particle flow optimization 322, resulting in mini-batch particle flow optimization 658. Particle flow optimization 658 includes selecting a mini-batch of data from a training set at operation 330, iterating through a homotopy discretization step at operation 332, and determining a batch of predictions at operation 334.

[0069] Various Python ML and AI libraries, such as Pytorch® and TensorFlow®, do not store individual gradients for each example in a mini-batch: instead, they sum or average the gradients by default to increase efficiency and reduce memory usage.

[0070] To accommodate these libraries, the mini-batch particle flow training optimization 658 can be tuned to evolve the average of the logarithms of the joint posterior probabilities.

number

[0071] In the formula, logp MB (θ)=N mb Since logp(θ), p(θ) is the prior distribution of the particles before the batch update, which physically corresponds to taking the geometric mean of the posterior probabilities of each sample within the mini-batch.

number

[0072] The drift vector determined in operation 664 is:

number

[0073] logL MB The average gradient of the objective function is logL MB This can be computed in most ML libraries by setting , however computing the average of the Hessian of the mini-batch log-likelihood deserves careful consideration to ensure that the averaging is performed at the right time.

[0074] The training particle stream can compute the Hessian of the log-likelihood using a Gauss-Newton Hessian approximation.

number

[0075] In the formula, p m is the mth component of the network prediction, and r,s denote the Hessian exponents. The Hessian average of the mini-batch log-likelihoods contains the average of the Hessians for each sample in the mini-batch.

number

[0076] This is the Jacobian term,

number

number

[0077] Mini-batch particle flow optimization 658 can be summarized as follows: For each mini-batch d = {x, yT}, d ∈ D of data in the training set (operation 660 selects a mini-batch of data from the training set). Make the current particle distribution equal to the previous particle distribution. Calculate the covariance Γ of the particle prior distribution, Repeatedly loop over the scalar homotopy parameter λ = [0, 1] (operation 332) Each λ k , k=1,2,…,N λ Regarding -1, Calculate the integration step size, δλ=λ k+1 -λ k Each particle {θ i}, for i=1,2,3...,N (operation 334), Using the particle values, input data x={x j} over the network (operation 661) to obtain a batch of predictions. {p j}=Λθ i ({x j}) Calculate the average gradient and Hessian of the mini-batch log-likelihood with respect to the network parameters (operation 662) Calculate the drift f and the diffusion matrix Q (operation 664). Update the particle state (operation 666) using numerical integration of the stochastic differential equation (SDE) (equation 15).

number

[0078] Classification results for the digit subset {0,1} from the Modified National Institute of Standards and Technology (MNIST) database. The MNIST database was created by Yann LeCun, Corinna Cortes, and Christopher JC Burges using images from two separate NIST databases. The MNIST database can be accessed at http: / / yann.lecun.com / exdb / mnist / .

[0079] A Convolutional Neural Network (CNN) architecture was instantiated, consisting of two convolutional layers with four filters each, followed by a dense output layer. The network has 286 network parameters. 100 normally distributed particles with initial covariance Γ=0.04Id were also instantiated. Numerical integration of the flow was performed using N λ The calculation was performed using a logarithmic step size of =10. [Table 1]

[0080] The mini-batch training particle flow is MB =1, N MB = 2, and NMB A mini-batch size of 16 was implemented to train a BNN with batch updates of 16. For mini-batch sizes > 1, the batch contains evenly distributed classes (e.g., batch size 16 contains 8 of class 0 and 8 of class 1). From these training examples, the smoothness of the average log-likelihood increases with mini-batch size. Furthermore, the deviation of each particle's log-likelihood from the mean decreases as the batch size increases. This means that individual particles are less affected by outliers than without mini-batches (batch size 1).

[0081] Figure 7 shows the N MB =1, N MB = 2, and N MB Figure 1 shows a plot of network accuracy with increasing batch updates for a batch size of 16 = 16. From the plot, it is clear that a batch size of 16 achieves and maintains the highest accuracy, while a batch size of 1 tends to decrease accuracy. It is clear from this study that using a mini-batch worth of data for each parameter update improves both the training speed and the robustness of the approach.

[0082] AI is a field concerned with the development of decision-making systems to perform cognitive tasks that previously required living actors such as humans. NNs are computational structures loosely modeled on biological neurons. In general, NNs encode information (e.g., data or decisions) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the basis of many AI applications such as speech recognition.

[0083] Many NNs are represented as a matrix of weights that correspond to the connections that are modeled. NNs frequently operate by accepting data into a series of input neurons that have many outgoing connections to other neurons. With each pass between neurons, the corresponding weights modify the input and are passed to an activation function. The result of the activation function is then sent to another neuron further down the NN graph. The process of weighting and processing by the activation function continues until it reaches an output neuron, and the result of the ANN processing constitutes the pattern and value of the output neuron.

[0084] Most NNs depend on accurate weights to work properly. However, NN designers usually do not know which weights will work for a particular application. NN designers usually choose a large number of layers of neurons, or specific connections between layers that include circular connections. A training process can be used to determine the appropriate weights by selecting the initial weights. In some examples, the initial weights may be chosen randomly. Training data is input into the NN, and the results are compared to an objective function that gives an indication of error. The indication of error is a measure of how wrong the NN's results are compared to the expected results. This error is then used to modify the weights. Over many iterations, the weights collectively converge and the operational data is encoded into the NN. This process is sometimes called optimizing the objective function (e.g., a cost function or loss function), whereby a cost or loss is minimized.

[0085] Gradient descent is a common technique for optimizing a certain objective (or loss) function. The gradient of a scalar field (e.g., a vector of partial derivatives) gives the direction of the steepest increase of this objective function. Thus, adjusting the parameters in the opposite direction by a small amount will generally decrease the objective function. After a sufficient number of iterations, the parameters tend to approach a minimum. In some implementations, the learning rate (e.g., step size) is fixed for all iterations. However, small step sizes tend to take longer to converge, while large step sizes may result in oscillations around the minimum or other undesirable behavior. A variable step size is usually introduced to achieve faster convergence without the drawbacks of large step sizes.

[0086] After a forward pass of input data through a neural network, backpropagation provides a parsimonious approach to evaluating the gradient of the objective function with respect to the network parameters. The final output of the network is constructed by combining the operations of each layer. This requires a chain rule to calculate the gradient of the objective function. Backpropagation starts from the final layer and works backwards to the input layer, exploiting a recursive relationship between the derivative of the objective with respect to the layer output and the corresponding quantity from the previous layer. This recursive relationship eliminates the redundancy of evaluating the entire chain rule of the derivative of the objective with respect to each parameter. Any well-known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0087] FIG. 8 is a block diagram of an example environment including a system for NN training, according to an embodiment. The system can assist in training cybersecurity solutions according to one or more embodiments. The system includes an artificial NN (ANN) 805 that is trained using processing nodes 810. The processing nodes 810 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other processing circuitry. In an example, multiple processing nodes can be used to train different layers of the ANN 805, or even different nodes 807 within a layer. Thus, a set of processing nodes 810 is configured to perform the training of the ANN 805.

[0088] The set of processing nodes 810 is configured to receive a training set 815 for the ANN 805. The ANN 805 comprises a set of nodes 807 (shown as rows of nodes 807) arranged in layers and a set of inter-node weights 808 (e.g., parameters) between the nodes of the set of nodes. In an example, the training set 815 is a subset of the complete training set, where the subset may allow processing nodes with limited storage resources to participate in training the ANN 805.

[0089] The training data may include multiple numerical values ​​that represent domains such as words, symbols, other parts of speech, etc. Each training or input 817 value to be classified as the ANN 805 is trained is provided to a corresponding node 807 in the first or input layer of the ANN 805. The values ​​propagate through the layers and are dependent on the objective function.

[0090] As noted above, the set of processing nodes is configured to train the neural network to create a trained neural network. For example, once trained, data input to the ANN generates a valid classification 820 (e.g., input data 817 is assigned to a category). The training performed by the set of processing nodes 807 is iterative. In an example, each iteration of training of the neural network is performed independently between layers of the ANN 805. Thus, two different layers may be processed in parallel by different members of the set of processing nodes. In an example, different layers of the ANN 805 are trained on different hardware. Members of different members of the set of processing nodes may be located in different packages, housings, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently between nodes in the set of nodes. An example of this is additional parallelization where individual nodes 407 (e.g., neurons) are trained independently. In an example, the nodes are trained on different hardware.

[0091] FIG. 9 illustrates, by way of example, a block diagram of one embodiment of a machine in the exemplary form of a computer system 900 upon which instructions may be executed to cause the machine to perform any one or more of the methodologies described herein. In a networked arrangement, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a network switch, or a network bridge, or any machine capable of executing instructions (sequentially or otherwise) that specify actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methodologies described herein.

[0092] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 904, and a static memory 906, which communicate with each other via a bus 908. The computer system 900 may further include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 900 also includes an alphanumeric input device 912 (e.g., a keyboard), a user interface (UI) navigation device 914 (e.g., a mouse), a mass storage device 916, a signal generator 918 (e.g., a speaker), a network interface device 920, and a radio 930, such as Bluetooth, WWAN, WLAN, and NFC, to enable application of security controls to such protocols.

[0093] The mass storage device 916 includes a machine-readable medium 922 on which is stored one or more sets of instructions and data structures (e.g., software) 924 that embody or are utilized by any one or more of the methods or functions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904 and / or within the processor 902 during execution thereof by the computer system 900, with the main memory 904 and the processor 902 also constituting machine-readable media.

[0094] While the machine-readable medium 922 is shown in the exemplary embodiment as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more instructions or data structures. The term "machine-readable medium" should also be construed to include any tangible medium that can store, encode, or carry instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present invention or that can store, encode, or carry data structures utilized by or related to such instructions. The term "machine-readable medium" should accordingly be construed to include, but is not limited to, solid-state memory, and optical and magnetic media. Specific examples of machine-readable media include, by way of example, non-volatile memory, including semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0095] The instructions 924 may further be transmitted or received over a communications network 926 using a transmission medium. The instructions 924 may be transmitted using a network interface device 920 and any one of several well-known transfer protocols (e.g., HTTP). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet, a cellular network, a plain old telephone service (POTS) network, and wireless data networks (e.g., WiFi networks and WiMax networks). The term "transmission medium" shall be interpreted as including any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communications signals or other intangible media for facilitating communication of such software.

[0096] Additional Notes and Examples Example 1 includes a method for training a Bayesian Neural Network (BNN) using batch inputs and operating the trained BNN, the method including initializing particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN and the particles collectively represent a distribution of the parameters of the BNN; optimizing the particles based on the batches of inputs using a minimal batch training particle stream to result in an optimized distribution of the parameters; determining a predictive distribution using the optimized distribution of the parameters and predictions from each of the NNs; and resulting in marginal distributions representing the predictive distributions.

[0097] In Example 2, Example 1 can further include, wherein the mini-batch training particle stream includes iteratively evolving values ​​of the network parameters based on a log homotopy.

[0098] In Example 3, Example 2 can further include, wherein the mini-batch training particle stream includes evolving a mean of the logarithms of the joint posterior probabilities.

[0099] In Example 4, at least one of Examples 2-3 can further include determining, for each batch in the training set, a geometric mean of the posterior probabilities of each input in the batch.

[0100] In Example 5, at least one of Examples 3-4 may further include, where evolving the averages includes, for each batch in the training set, averaging the Hessian matrix of each input in the batch.

[0101] In Example 6, example 5 can further include, where averaging the Hessian includes storing corresponding Hessian and Jacobian terms for each input in the batch.

[0102] In Example 7, Example 6 can further include averaging the Hessian matrix includes determining, for each input in the batch, a product of matrix terms of the Hessian matrix and terms of the Jacobian in a Gauss-Newton approximation, producing a product result.

[0103] Example 8 includes a system including a processing circuit and a memory coupled to the processing circuit, the memory including instructions that, when executed by the processing circuit, cause the processing circuit to perform any of the methods of Examples 1-7.

[0104] Example 9 includes a non-transitory machine-readable medium including instructions stored thereon that, when executed by a machine, cause the machine to perform the method of one of Examples 1-8.

[0105] Although the embodiments have been described with reference to certain exemplary embodiments, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present invention. Accordingly, the specification and drawings should be regarded in an illustrative and not a limiting sense. The accompanying drawings, which form a part of this specification, show by way of illustration, and not by way of limitation, specific embodiments in which the present subject matter may be practiced. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Since other embodiments may be utilized and derived therefrom, structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Accordingly, the detailed description is not to be construed in a limiting sense, and the scope of the various embodiments is defined solely by the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A method for training a Bayesian Neural Network (BNN) using batch inputs and operating the trained BNN, comprising: initializing the particles such that each particle individually represents a pointwise value of a respective NN parameter of a NN, and said particles collectively represent a distribution of parameters of said BNN; optimizing said particles based on batches of inputs using a minimal batch training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and providing a marginal distribution representing the prediction distribution.

2. 2. The method of claim 1, wherein mini-batch training particle streams include iteratively evolving values ​​of the network parameters based on log homotopy.

3. The method of claim 2 , wherein the mini-batch training particle stream comprises evolving a mean of the logarithms of the joint posterior probabilities.

4. 3. The method of claim 2, wherein the mini-batch training particle stream comprises determining, for each batch in the training set, a geometric mean of the posterior probabilities of each input in the batch.

5. The method of claim 3 , wherein evolving the mean comprises, for each batch in the training set, averaging the Hessian of each input in the batch.

6. The method of claim 5 , wherein averaging the Hessian comprises storing, for each input in the batch, corresponding Hessian and Jacobian terms.

7. Averaging the Hessian matrix determining, for each input in the batch, a product of the Hessian terms and the Jacobian terms in a Gauss-Newton approximation, producing a product of the Hessian terms and the Jacobian terms; and 7. The method of claim 6, further comprising averaging the results of said products to produce an average of said Hessian matrix.

8. A non-transitory machine-readable medium containing instructions that, when executed by a machine, cause the machine to perform operations, the operations including: initializing the particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN, and said particles collectively represent a distribution of parameters of the BNN; optimizing said particles based on batches of inputs using a minimal batch training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and yielding a marginal distribution that represents the prediction distribution.

9. 10. The non-transitory machine-readable medium of claim 8, wherein the mini-batch training particle stream comprises iteratively evolving values ​​of the network parameters based on a log homotopy.

10. 10. The non-transitory machine-readable medium of claim 9, wherein the mini-batch training particle stream comprises evolving a mean of logarithms of the joint posterior probabilities.

11. 10. The non-transitory machine-readable medium of claim 9, wherein the mini-batch training particle stream includes determining, for each batch in the training set, a geometric mean of posterior probabilities of each input in the batch.

12. 11. The non-transitory machine-readable medium of claim 10, wherein evolving the mean comprises, for each batch in the training set, averaging a Hessian matrix of each input in the batch.

13. 13. The non-transitory machine-readable medium of claim 12, wherein averaging the Hessian comprises storing, for each input in the batch, corresponding Hessian and Jacobian terms.

14. Averaging the Hessian matrix determining, for each input in the batch, a product of the Hessian terms and the Jacobian terms in a Gauss-Newton approximation, producing a product of the Hessian terms and the Jacobian terms; and 14. The non-transitory machine-readable medium of claim 13, comprising averaging the results of the products to produce the average Hessian matrix.

15. 1. A system comprising: A processing circuit; a memory coupled to the processing circuitry and containing instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations, the operations including: initializing the particles such that each particle individually represents a pointwise value of a respective NN parameter of the NN, and said particles collectively represent a distribution of parameters of the BNN; optimizing said particles based on batches of inputs using a minimal batch training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and providing a marginal distribution representing the prediction distribution.

16. 16. The system of claim 15, wherein the mini-batch training particle stream comprises iteratively evolving values ​​of the network parameters based on log homotopy.

17. 17. The system of claim 16, wherein the mini-batch training particle stream comprises evolving a mean of the logarithms of the joint posterior probabilities.

18. 17. The system of claim 16, wherein the mini-batch training particle flow includes determining, for each batch in the training set, a geometric mean of posterior probabilities of each input in the batch.

19. 20. The system of claim 17, wherein evolving the mean comprises, for each batch in the training set, averaging a Hessian matrix of each input in the batch.

20. Averaging the Hessian matrix storing, for each input in the batch, corresponding Hessian and Jacobian terms; determining, for each input in the batch, a product of the Hessian terms and the Jacobian terms in a Gauss-Newton approximation, producing a product of the Hessian terms and the Jacobian terms; and 20. The system of claim 19, further comprising averaging the results of the products to produce an average Hessian matrix.

Citation Information

Patent Citations

  • System and method for generating adversarial examples

    US20210124999A1

  • Closed loop adaptive particle forecasting

    WO2020237077A1