Particle-flow training of Bayesian neural networks

A modified particle flow technique trains BNNs efficiently with sparse data, enhancing prediction accuracy and uncertainty quantification in Bayesian neural networks.

JP2024541007AActive Publication Date: 2024-11-06RAYTHEON CO
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2024524586
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-10-25
Filing Date
2022-10-25
Publication Date
2024-11-06
Estimated Expiration
2042-10-25

AI Technical Summary

Technical Problem

Existing training methods for Bayesian neural networks (BNNs) are computationally expensive and require large datasets, limiting their effectiveness in sparse data scenarios and open-set detection, while standard neural networks lack direct uncertainty metrics.

Method used

A modified particle flow technique is applied to train BNNs, optimizing Bayes' theorem by evolving a prior distribution into a posterior distribution through a training particle stream, using a logarithmic homotopy constraint and a constant diffusion matrix to reduce computational complexity and enable uncertainty quantification.

Benefits of technology

The method allows BNNs to achieve high prediction accuracy and uncertainty measurement with sparse datasets, effectively handling open-set problems and providing reliable uncertainty metrics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541007000001_ABST
    Figure 2024541007000001_ABST
Patent Text Reader

Abstract

Devices, systems, and methods for training and operating a Bayesian Neural Network (BNN) are discussed herein. The methods may include initializing particles, each individually representing a pointwise value of a NN parameter of a respective NN that collectively represents a distribution of parameters of the BNN, optimizing the particles using a training particle stream to result in an optimized distribution of the parameters, determining a predictive distribution using the optimized distribution of parameters and predictions from each of the NNs, and resulting in marginal distributions representing the predictive distributions.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Claiming priority This application claims the benefit of priority to U.S. Patent Application No. 17 / 509,270, filed October 25, 2021, which is incorporated by reference in its entirety herein.

[0002] The embodiments discussed herein relate to devices, systems, and methods for training a Bayesian Neural Network (NN) using mini-batch particle flows. [Background technology]

[0003] Most NNs perform point estimates without a direct uncertainty metric or confidence measure. Standard NNs can also perform relatively poorly on open sets. Bayesian NNs (BNNs) learn a statistical distribution of weights, providing a statistical environment in which the uncertainty and confidence of decisions can be determined. Existing training methods for BNNs include Hamiltonian Monte Carlo, variational inference (both Monte Carlo and deterministic), stochastic backpropagation, and standard particle filters. These training methods are computationally expensive and require relatively large amounts of data for training. [Brief description of the drawings]

[0004] [Figure 1] As an example, we present a block diagram comparing a standard DL architecture with a BNN. [Diagram 2] As an example, a diagram of an embodiment of a NN is shown. [Diagram 3] As an example, a flow diagram of an embodiment of a NN training procedure using a training particle stream is shown. [Figure 4] As an example, here is a histogram of each prediction made by the BNN before training: [Diagram 5] As an example, we show a histogram of each prediction made by the BNN after 120 training updates using the training particle stream. [Figure 6] As an example, we show the histograms of each prediction made by the BNN using selected Fashion-MNIST samples as input. [Figure 7] As an example, we show histograms representing the distribution of MNIST input to a BNN trained using the training particle stream, and Fashion-MNIST input to a BNN trained using the training particle stream. [Figure 8] FIG. 1 is a block diagram of an example environment including a system for NN training, according to an embodiment. [Figure 9] By way of example, a block diagram of an embodiment of a machine of an exemplary form of a computer system upon which instructions may be executed to cause the machine to perform any one or more of the methods described herein is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The embodiments address the problem of providing predictions and corresponding uncertainties in trained networks with the priority of performing well using sparse datasets for training and open set detection. Deep learning (DL) has been widely adopted in several industries as a means to achieve autonomy. Example industries include self-driving cars, medical diagnostics, bioinformatics, manufacturing, and radar. Although DL has demonstrated state-of-the-art performance in some tasks, quantification of uncertainty remains a global challenge and an active research area. Example tasks include computer vision, speech recognition, and natural language processing. Quantifying uncertainty in decision making is important for any industry that automates, especially in high-stakes situations. Not only can quantification of uncertainty help reduce overconfident decisions, but quantification of uncertainty can also help computers select more realistic human responses. More realistic human responses can be useful for any automated system that interacts with humans or in human-occupied environments.

[0006] FIG. 1 shows, as an example, a block diagram of a standard DL architecture 102 versus a BNN 104. Standard DL architectures make point estimates of model predictions (outputs 108) and network parameters (node ​​weights 112). Such DL architectures 102 do not provide a direct means for quantifying uncertainty. Instead, standard DL architectures 102 rely on indirect methods for estimating uncertainty. Common methods include the use of entropy and confidence scores and functions, as well as application-specific methods. Bayesian DL architectures 104 and statistical methods tend to provide a more natural context for quantifying uncertainty.

[0007] BNNs have been researched since at least 1992 and are a growing field. BNN technology uses Bayes' Theorem as a guide to solve the posterior probability distribution of the weights in a NN (the distribution of the nodes 114). The computational difficulty of solving Bayes' Theorem for DL ​​tasks has led to the development of numerous approaches to estimate the posterior distribution of the weights in a NN. Well-known approaches for BNN optimization include Hamiltonian Monte Carlo, Monte Carlo Variational Inference, Deterministic Variational Inference, and Stochastic Backpropagation (PBP).

[0008] The output 110 of the BNN 104 is a distribution per class compared to the output of the NN 102, which provides a score per class as output 108. The distribution of the output 110 may be a natural consequence of using a distribution ("dist.") instead of a scalar weight as in node 112 to represent the activation function of node 114.

[0009] Deep ensembles and probabilistic regularization methods offer alternative approaches towards estimating uncertainty in DL. Although these methods do not optimize Bayes' theorem, they provide a statistical environment for computing prediction uncertainty at a fraction of the computational cost of BNNs. These statistical methods are used in a variety of applications to quantify uncertainty.

[0010] A common theme in existing Bayesian NNs and statistical approaches is very large training sets and training for thousands of epochs. However, real-world datasets are typically sparse, and training a general NN with thousands of parameters may be insufficient. Embodiments provide a NN architecture that can quantify uncertainty and also perform robustly within the limits of sparse datasets and for open set problems.

[0011] The embodiment uses a modified form of particle flow technique that is commonly used in particle filters, but is modified and reused to train BNNs. The modified form of the particle flow method is referred to herein as "training particle flow". Particle flow is a method for optimizing Bayes' theorem, and has (so far) only been used in the context of particle filtering. Numerical experiments of particle flow in the context of particle filtering have shown that particle flow can reduce the computational complexity by orders of magnitude compared to standard particle filters and other state-of-the-art algorithms for the same filter accuracy. Furthermore, particle flow can reduce the filter error by orders of magnitude compared to extended Kalman filters and other state-of-the-art algorithms for difficult nonlinear non-Gaussian problems.

[0012] Recently, particle methods for optimizing NNs have emerged, but optimizing BNNs using particle flows has not been done before to the best of our knowledge. Results from BNNs trained to perform classification tasks using MNIST{0,1} show high prediction accuracy with very few training samples. Furthermore, BNNs trained to perform classification tasks had a strong ability to measure prediction uncertainty using the variance of the network predictions.

[0013] particle flow Consider a system with internal state s and measurements m. Bayes' theorem relates the posterior probability of a state given a measurement p(s│m) to the prior distribution p(s) on the state and the likelihood p(m│s) of the measurement given the state according to:

number

[0014] where p(m) = ∫p(m│s)p(s)ds is a proof that acts as a normalizing constant. The measurement m is a quantity that helps characterize what the internal state is like or what the internal state will be like. Given a general Markov process with a sequence of noisy measurements {m}, particle filters provide a way to estimate the system's internal state(s) {s} using Bayes' theorem as a guideline.

[0015] Particle flow is a method used in the context of particle filtering to estimate the optimal posterior distribution of the internal state for each measurement. Particle flow optimizes Bayes' theorem by evolving a prior distribution to a posterior distribution along a log homotopy.

number

[0016] Two continuous functions in their respective spaces are called homotopic if one can be "continuously deformed" into the other. A homotopy exists between the functions that can be deformed in this way.

[0017] where K(λ,m) = ∫p(m│s). λ p(s)ds normalizes the posterior distribution p(s,λ│m) for each λ. The scalar homotopy parameter λ=[0,1] evolves the distribution from the prior to the posterior for a particular measurement m. Each particle represents a single realization of the system's internal state s. The flow of particles along the log homotopy is described by the following stochastic differential equation (SDE):

number

[0018] During the ceremony,

number

[0019] The evolution of the posterior distribution of particles follows the Fokker-Planck equation (with the diffusion squared matrix Q ij =Σ k B ik B jk The term can be characterized by:

number

[0020] Note that the gradients and derivatives written in (4) are in Cartesian coordinates, but (4) can be used in any suitable Cartesian coordinate system by appropriate transformation of the partial derivatives.

[0021] Drift Speed

number

number

number

[0022] During the ceremony,

number

[0023] The geodesic solution assumes a Gromov solution for drift velocity and no diffusion (i.e. Q=0), which does not simultaneously satisfy equations 2 and 4. The zero curvature solution assumes that the particle does not accelerate as it changes λ and there is no diffusion term (i.e. Q=0). The drift velocity of this solution is proportional to the Gromov drift velocity.

[0024] DL and supervised learning tasks DL is a branch of ML that uses a series of layers of nodes to learn higher-order representations of data for supervised, semi-supervised, or unsupervised learning tasks. Although the described embodiments focus on supervised learning tasks, the embodiments may be applied to any learning task for which a likelihood function can be defined.

[0025] In supervised learning tasks, deep neural networks (DNNs) can be used to learn relationships between input and output data for regression or classification. For regression tasks, NNs can predict a dependent variable p that is causally related to the input data. For classification tasks, the network can predict the probability p of a particular class. _j , the network predicts the corresponding truth class or truth value y of the input data. The word "probability" is a bit of a misnomer here. Classification tasks often use a SoftMax activation function in the output layer to generate a vector whose elements sum to 1. This output represents a set of class probabilities, but these "probabilities" are not necessarily well-calibrated to the actual accuracy of the network. In this sense, the output probabilities can be more accurately understood as a normalized score for each class. During training, the corresponding truth class or truth value y of the input data is calculated. T You can evaluate the NN predictions against and adjust the network weights using your chosen optimization scheme.

[0026] The “likelihood” function is used in many gradient-based optimization methods in DL. For regression tasks, the NN weights θ = {θ i} and the truth variable y given the network prediction p T The likelihood L of is usually modeled by a normal distribution.

number

[0027] where Σ is the truth y T is the covariance matrix that scales the error between y and the prediction p, and the index k is T This likelihood function represents the dimension of the prediction p and the truth y T We assume Gaussian white noise discrepancy between the vectors. The corresponding log-likelihood is given by

number

[0028] This is reminiscent of the mean squared error (MSE) when Σ is the identity matrix.

[0029] For classification tasks, a categorical distribution is the predicted class probability p j , j ∈ [1n クラス ] and the true class y of the input given the NN weights θ T represents the likelihood L of

number

[0030] where y T ={y Tj} is a one-hot encoded vector of the image truth classes, or a non-binary vector summing to 1 if we are using soft labels. The corresponding log-likelihood is the negative of the cross-entropy function.

number

[0031] Mapping particle flows for training BNNs 2 shows, by way of example, a diagram of an embodiment of a NN 200. The NN 200 includes L layers of nodes. θ contains the network parameters θ={θ j}={W 1 ,b1 ,W 2 ,b 2 ,…,W L ,b L}. NN200 is a set of N パラメータ parameters, and the network parameters θ are N パラメータ dimensional vector θ∈R Nパラメータ become.

[0032] In a typical supervised learning task, we have a set of training data D={X,Y T} to train NN200, where X={x j} is the set of all inputs, and Y T ={y T,j} is the corresponding set of all truth values ​​given X. Each NN prediction p j The data

number

number

[0033] In the formula, σ represents the activation function of each layer of nodes, and g represents the affine transformation at the nodes of each layer.

[0034] The goal of a BNN is to use Bayes' theorem to learn the optimal posterior distribution of the network parameters p(θ│D) given the data.

number

[0035] The right-hand side of equation (12) is the posterior probability

number

number

[0036] The embodiment maps the particle flow used in the particle filtering context to the DL context, and defines the network parameters θ and truth values ​​Y T By equating the internal state {s} and measurements {m} in equation (1) for j The training particle stream represents a single realization of the NN’s network parameters {θ j The likelihood of each measurement m given the internal state s is expressed as the likelihood Y of the truth value given the prediction P of the network 200 based on the input data X. T In the DL context, each particle is assigned a network parameter {θ j} represents a single realization of

[0037] The mapping of particle streams to training particle streams used in the particle filtering context is mathematically described in equations (13) through (15).

number

number

[0038] The log homotopy constraint given in equation (3) is logp(θ,λ│{X,Y T})=λlogL(Y T │θ,X)+logp(θ)-logp(λ,│X)Y T (Formula 16)

[0039] Here, the scalar homotopy parameter (outside 1) TIFF2024541007000022.tif6127 is (outside 2) have been added to the notation for the posterior distribution and normalization factor to specify the variance of these terms in TIFF2024541007000023.tif6127. The corresponding gradients and derivatives written in equation (4) are now with respect to the network parameters for the training particle flow.

number

[0040] Equation 18 shows the mathematical expression of the drift velocity in the training particle stream using the Gromov equation.

number

[0041] The Gromov equation generalizes well to architectures with L ≥ 1 layers, various activation functions, and arbitrary prior and likelihood function forms. However, equation (18) is not sufficient to justify the use of linear activation functions.

number

[0042] Equation 19 provides the mathematical expression of the constant diffusion matrix used in particle flow training. Q=αId (Equation 19)

[0043] where α ∈ R0 is a positive real number and Id is an identity matrix. A constant diffusion matrix can help provide numerical stability. Additionally, adding a small amount of noise can prevent the network training from getting stuck in local minima. A small exponential decay factor can also be added to the diffusion matrix to reduce the effect of noise as the number of weight updates increases. Q=αexp[β(update#)]Id (Formula 20)

[0044] In the formula, β>0 represents the damping factor.

[0045] NN training procedure using training particle flow 3 shows, by way of example, a flow diagram of an embodiment of a NN training procedure using a training particle stream, which includes initialization 320, training particle stream optimization 322, and prediction 324.

[0046] Initialization 320 involves a user selecting, or a computer automatically instantiating, the functional forms of the likelihood and prior in operation 326. Initialization 320 involves sampling the selected prior distribution p(θ) in operation 328. The sampling initializes each realization of the NN optimized in optimization 322. If a multivariate normal distribution was selected as the prior distribution in operation 326, the corresponding Hessian matrix will have a simple analytical form. p(θ)~N(θ;μ,Γ) (Eq. 20)

number

[0047] where μ is the mean vector and Γ is the covariance matrix. The mean adds an initial offset to the values ​​of the network parameters. For simplicity and to avoid adding spurious offsets, the mean can be set to 0; however, the offsets can be known and used. The value of Γ characterizes the initial spread and potential correlations of each network parameter. The value can be chosen to be large enough to promote fast learning, yet small enough to prevent divergence in the network.

[0048] For the likelihood function, either equation (7) or (9) can be used depending on the type of supervised learning task. The residual covariance in equation (7) can be chosen similarly to the previous covariance (e.g., to facilitate learning but avoid divergence). The number of particles N can be chosen to be large enough to provide sufficient statistics on the data and avoid divergence of the covariance matrix.

[0049] Particle flow optimization 322 can be explained as follows. Training set (x j, y T,j ) ∈ D (operation 330 selects data from the training set): Make the current particle distribution equal to the previous particle distribution. Calculate the covariance of the particle prior distribution, Γ. Repeatedly loop over the scalar homotopy parameter λ = [0, 1] (operation 332) Each λ k , k=1,2,…,N λ Regarding -1, Calculate the integration step size, δλ=λ k+1 -λ k Each particle {θ i}, for i = 1, 2, 3 ..., N (operation 334) Use the value of the particle to input data x j over the network (operation 334) to obtain the prediction. p j =Λ θi (x j) Calculate the gradient of the log-likelihood with respect to the network parameters and the Hessian (operation 336). Calculate the drift f and the diffusion matrix Q (operation 338). Update the particle state using numerical integration of the stochastic differential equation (SDE) (Equation 15) (operation 340).

number

[0050] In operation 330, a pair of input and output values ​​is selected from the data set. In operation 332, operation 322 repeats the discretization step of the homotopy parameter λ. In operation 334, a particle state of the N particle states from operation 328 is selected to pass the data through the selected NN. The result of operation 334 may be a prediction. In operation 336, the gradient (derivative) and Hessian are calculated. Drift and diffusion are determined based on the gradient and Hessian in operation 338. In operation 340, the particle state is updated.

[0051] The particle flow is integrated using numerical integration to integrate equation (15). The scalar homotopy parameter λ = {λ k :k∈[1,N λ The interpolation of} can be based on linear, logarithmic, or adaptive scaling. The number of divisions of the scalar homotopy parameter N λ can be balanced between integration accuracy and algorithmic efficiency. There are several different methods for the numerical integration of SDEs, but the Euler-Maruyama algorithm can be used for computational efficiency.

[0052] Prediction 324 may provide an output prediction in operation 342. A marginalized probability distribution of the output prediction given the new input is determined in operation 344.

number

number

number

[0053] Marginalizing (21) over all network realizations θ gives the new input

number

number

number

[0054] Embodiments can evaluate equation (22) using Monte Carlo sampling of the posterior distribution p(θ│D), where the sampling is for all particles {θ i}.

number

[0055] As can be seen, the predictive distribution is the marginalization of the posterior distribution due to the network predictions, which is the sum over the network parameters.

[0056] Hessian Approximation One component of the drift calculation in act 338 is the Hessian of the log-likelihood with respect to the network parameters. The full Hessian is computationally intractable for almost all but the smallest NNs. The Gauss-Newton Hessian approximation provides an effective means of estimating the full Hessian of the system. The matrix elements of the Hessian are:

number

[0057] In the formula, p m is the mth component of the network prediction. Note that as written in equation (11), the subscripts of the predictions, which correspond to the indices of the data, have been omitted for simplicity.

[0058] In the case of regression, the output of an L-layer neural network is a prediction of the dependent variable p. Using the likelihood described in equation (7), the corresponding Hessian term in equation (24) is

number

[0059] where Σ is the covariance matrix that scales the error between the truth and the prediction in the likelihood.

[0060] For classification, the output of an L-layer neural network is a set of class probabilities {pj}. If the predicted class probability of the true class is zero, then the output probability p m The gradient of the log-likelihood (equation (9)) with respect to is unity. To avoid numerical instability, we take the derivative with respect to the output of the L-th layer before the softmax activation. (Outside 3) TIFF2024541007000038.tif7127 can be obtained.

[0061]

number

number

[0062] Diagonal approximation To reduce computational cost, a Gauss-Newton-Hessian approximation can be performed for each network weight. This performance of a Gauss-Newton-Hessian approximation for each network weight is referred to herein as a "diagonal approximation." In a diagonal approximation, all off-diagonal elements of the Hessian are set to zero. The diagonal matrix elements of the Hessian are:

number

[0063] From equation (27), it is clear that this approximation removes all correlations between network parameters in the Hessian matrix. The reduced Hessian matrix, i.e., the Hessian matrix with all correlations between network parameters removed, can be vectorized according to this approximation, significantly reducing the memory required to store the full Hessian matrix. For consistency, we use the prior covariance [Γ ii ] (e.g., the prior variance) we can remove all prior correlations.

[0064] Under these approximations, the vector components of the drift velocity simplify to:

number

[0065] To demonstrate how well the training particle stream works, results are provided for a binary classification problem. The "training" data includes classification results for a subset of digits {0,1} from the Modified National Institute of Standards and Technology (MNIST) database. The MNIST database was created by Yann LeCun, Corinna Cortes, and Christopher J.C. Burges using images from two separate NIST databases. The MNIST database can be accessed at http: / / yann.lecun.com / exdb / mnist / . Because this is a binary problem, the basic convolutional network architecture consists of two convolutional layers (with only four filters each) followed by a dense output layer. This network has a total of 286 network parameters (i.e., sum of the total weights and biases). The test data consists of the traditional MNIST test set (limited to the two digits "0" and "1"). This results in a total of 2,115 test images.

[0066] By approximating the particle flow equations using a diagonal approximation, we are able to run simulations involving 200 particles to train the network. Each of these particles is initialized with an initial covariance matrix. Table 2 provides an overview of the network and particle flow settings used to achieve these results. [Table 1]

[0067] It may be instructive to look at some test predictions using these initialized particles. Note that the network predictions are probabilities of each class, not specific classes. For example, the output prediction after passing an image of "1" through the network might be p(0.2,0.8), which might mean that the network predicts that the image is class "0" with probability 0.2, and class "1" with probability 0.8.

[0068] As an example, Figure 4 shows a histogram of each prediction made by the BNN before training. Figure 4 shows that for each of the four inputs, the initial predictions are roughly 50 / 50 (as expected).

[0069] The training process can be structured so that the classes alternate, such that the training particle flow technique uses images of "1", followed by images of "0", etc. Alternating in this way can help reduce bias of particles towards any one class. Note that this alternating procedure can be performed in the more general case of multiple classes.

[0070] As an example, Figure 5 shows histograms of the respective predictions made by the BNN after 120 training updates using the training particle stream. Note that in these results there are only 60 samples per class. For comparison, a standard NN uses the entire MNIST training dataset, which consists of about 6,000 training images per class.

[0071] Based on Figure 5, we can see that the network has low variance (which can be interpreted as high confidence) for images (A) and (B). Images A and B "look" like "standard" 1s and 0s. It is also clear that images (C) and (D) in Figure 5 look less "standard" and have a higher variance in their predictions.

[0072] To compare with standard classification metrics, we can calculate the accuracy of a network trained using the training particle flow by using the mean of the predicted distribution. Doing this with the network and update procedure described above achieved an accuracy of 99.9% (2113 / 2115), which is comparable to standard training techniques. However, it is important to emphasize that here we obtain a measure of uncertainty or confidence based on the variance of the predictions.

[0073] To further highlight the power of training particle streams, we can look at the results when a network trained using training particle streams is presented with images that are neither 0 nor 1. The Fashion-MNIST dataset provides such images. Fashion-MNIST is described in Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. Han Xiao, Kashif Rasul, Roland Vollgraf. arXiv:1708.07747. The Fashion-MNIST dataset consists of images of clothing and accessories. The results obtained using the Fashion-MNIST dataset as a test dataset are shown in Figure 6. It is clear that the variance of the prediction distribution for each of the Fashion-MNIST test cases is much higher compared to the MNIST test cases.

[0074] In this setting, one can set a threshold on the variance, and above this given threshold the network is essentially saying "I don't know what the correct class prediction is." In contrast, standard NN predictions have zero prediction variance and must rely on other methods to estimate the uncertainty. To make this point clearer, a histogram of the variance of predictions using the MNIST and Fashion-MNIST datasets as inputs may be informative.

[0075] As an example, Figure 7 shows a line graph diagram of the variance between MNIST input to a BNN trained using the training particle stream and Fashion-MNIST input to a BNN trained using the training particle stream. Although the BNN was only shown a total of 120 images during training (60 images of 0 and 60 images of 1), there is a clear separation in prediction variance between the MNIST and Fashion-MNIST datasets.

[0076] AI is a field concerned with the development of decision-making systems to perform cognitive tasks that previously required living actors such as humans. NNs are computational structures loosely modeled on biological neurons. In general, NNs encode information (e.g., data or decisions) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the basis of many AI applications such as speech recognition.

[0077] Many NNs are represented as a matrix of weights that correspond to the connections that are modeled. NNs frequently operate by accepting data into a series of input neurons that have many outgoing connections to other neurons. With each pass between neurons, the corresponding weights modify the input and are passed to an activation function. The result of the activation function is then sent to another neuron further down the NN graph. The process of weighting and processing by the activation function continues until it reaches an output neuron, and the result of the ANN processing constitutes the pattern and value of the output neuron.

[0078] Most NNs depend on accurate weights to work properly. However, NN designers usually do not know which weights will work for a particular application. NN designers usually choose a large number of layers of neurons, or specific connections between layers that include circular connections. A training process can be used to determine the appropriate weights by selecting the initial weights. In some examples, the initial weights may be chosen randomly. Training data is input into the NN, and the results are compared to an objective function that gives an indication of error. The indication of error is a measure of how wrong the NN's results are compared to the expected results. This error is then used to modify the weights. Over many iterations, the weights collectively converge and the operational data is encoded into the NN. This process is sometimes called optimizing the objective function (e.g., a cost function or loss function), whereby a cost or loss is minimized.

[0079] Gradient descent is a common technique for optimizing a certain objective (or loss) function. The gradient of a scalar field (e.g., a vector of partial derivatives) gives the direction of the steepest increase of this objective function. Thus, adjusting the parameters in the opposite direction by a small amount will generally decrease the objective function. After a sufficient number of iterations, the parameters tend to approach a minimum. In some implementations, the learning rate (e.g., step size) is fixed for all iterations. However, small step sizes tend to take longer to converge, while large step sizes may result in oscillations around the minimum or other undesirable behavior. A variable step size is usually introduced to achieve faster convergence without the drawbacks of large step sizes.

[0080] After a forward pass of input data through a neural network, backpropagation provides a parsimonious approach to evaluating the gradient of the objective function with respect to the network parameters. The final output of the network is constructed by combining the operations of each layer. This requires a chain rule to calculate the gradient of the objective function. Backpropagation starts from the final layer and works backwards to the input layer, exploiting a recursive relationship between the derivative of the objective with respect to the layer output and the corresponding quantity from the previous layer. This recursive relationship eliminates the redundancy of evaluating the entire chain rule of the derivative of the objective with respect to each parameter. Any well-known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0081] FIG. 8 is a block diagram of an example environment including a system for NN training, according to an embodiment. The system can assist in training cybersecurity solutions according to one or more embodiments. The system includes an artificial NN (ANN) 805 that is trained using processing nodes 810. The processing nodes 810 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other processing circuitry. In an example, multiple processing nodes can be used to train different layers of the ANN 805, or even different nodes 807 within a layer. Thus, a set of processing nodes 810 is configured to perform the training of the ANN 805.

[0082] The set of processing nodes 810 is configured to receive a training set 815 for the ANN 805. The ANN 805 comprises a set of nodes 807 (shown as rows of nodes 807) arranged in layers and a set of inter-node weights 808 (e.g., parameters) between the nodes of the set of nodes. In an example, the training set 815 is a subset of the complete training set, where the subset may allow processing nodes with limited storage resources to participate in training the ANN 805.

[0083] The training data may include multiple numerical values ​​that represent domains such as words, symbols, other parts of speech, etc. Each training or input 817 value to be classified as the ANN 805 is trained is provided to a corresponding node 807 in the first or input layer of the ANN 805. The values ​​propagate through the layers and are dependent on the objective function.

[0084] As noted above, the set of processing nodes is configured to train the neural network to create a trained neural network. For example, once trained, data input to the ANN generates a valid classification 820 (e.g., input data 817 is assigned to a category). The training performed by the set of processing nodes 807 is iterative. In an example, each iteration of training of the neural network is performed independently between layers of the ANN 805. Thus, two different layers may be processed in parallel by different members of the set of processing nodes. In an example, different layers of the ANN 805 are trained on different hardware. Members of different members of the set of processing nodes may be located in different packages, housings, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently between nodes in the set of nodes. An example of this is additional parallelization where individual nodes 407 (e.g., neurons) are trained independently. In an example, the nodes are trained on different hardware.

[0085] FIG. 9 illustrates, by way of example, a block diagram of one embodiment of a machine in the exemplary form of a computer system 900 upon which instructions may be executed to cause the machine to perform any one or more of the methodologies described herein. In a networked arrangement, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a network switch, or a network bridge, or any machine capable of executing instructions (sequentially or otherwise) that specify actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methodologies described herein.

[0086] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 904, and a static memory 906, which communicate with each other via a bus 908. The computer system 900 may further include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 900 also includes an alphanumeric input device 912 (e.g., a keyboard), a user interface (UI) navigation device 914 (e.g., a mouse), a mass storage device 916, a signal generator 918 (e.g., a speaker), a network interface device 920, and a radio 930, such as Bluetooth, WWAN, WLAN, and NFC, to enable application of security controls to such protocols.

[0087] The mass storage device 916 includes a machine-readable medium 922 on which is stored one or more sets of instructions and data structures (e.g., software) 924 that embody or are utilized by any one or more of the methods or functions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904 and / or within the processor 902 during execution thereof by the computer system 900, with the main memory 904 and the processor 902 also constituting machine-readable media.

[0088] While the machine-readable medium 922 is shown in the exemplary embodiment as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more instructions or data structures. The term "machine-readable medium" should also be construed to include any tangible medium that can store, encode, or carry instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present invention or that can store, encode, or carry data structures utilized by or related to such instructions. The term "machine-readable medium" should accordingly be construed to include, but is not limited to, solid-state memory, and optical and magnetic media. Specific examples of machine-readable media include, by way of example, non-volatile memory, including semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0089] The instructions 924 may further be transmitted or received over a communications network 926 using a transmission medium. The instructions 924 may be transmitted using a network interface device 920 and any one of several well-known transfer protocols (e.g., HTTP). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet, a cellular network, a plain old telephone service (POTS) network, and wireless data networks (e.g., WiFi networks and WiMax networks). The term "transmission medium" shall be interpreted as including any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communications signals or other intangible media for facilitating communication of such software.

[0090] Additional Notes and Examples Example 1 includes a method for training and operating a Bayesian Neural Network (BNN), the method including initializing particles that each individually represent pointwise values ​​of NN parameters of a respective NN that collectively represent a distribution of parameters of the BNN, optimizing the particles using a training particle stream to result in optimized distributions of the parameters, determining a predictive distribution using the optimized distributions of the parameters and predictions from each of the NNs, and resulting in marginal distributions that represent the predictive distributions.

[0091] In example 2, example 1 can further include, the training particle stream includes a particle stream technique used in a Daum-Huang particle filter that replaces internal states and measurements with weights and truth values, respectively.

[0092] In Example 3, at least one of Examples 1-2 may further include, each particle in the set of particles representing all NN network parameters of an instantiation of the NN that collectively represent the distribution of parameters of the BNN.

[0093] In Example 4, at least one of Examples 1-3 may further include, wherein the training particle stream includes iteratively evolving values ​​of the network parameters along a log homotopy.

[0094] In Example 5, at least one of Examples 1-4 can further include, wherein the training particle stream includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.

[0095] In Example 6, example 5 can further include, wherein the training particle stream includes using a diagonal approximation to the Gauss-Newton approximation.

[0096] In Example 7, Example 6 can further include, wherein the diagonal approximation includes performing a Gauss-Newton approximation for each weight.

[0097] Example 8 includes a non-transitory machine-readable medium containing instructions that, when executed by a machine, cause the machine to perform operations for training and operating a Bayesian Neural Network (BNN), the operations including initializing particles, each individually representing pointwise values ​​of a NN parameter of a respective NN that collectively represent a distribution of parameters of the BNN; optimizing the particles using a training particle stream to result in an optimized distribution of the parameters; determining a predictive distribution using the optimized distribution of the parameters and predictions from each of the NNs; and resulting in marginal distributions representing the predictive distributions.

[0098] In Example 9, Example 8 can further include, the training particle stream includes a particle stream technique used in a Daum-Huang particle filter, and the internal states and measurements are replaced with weights and truth values, respectively.

[0099] In Example 9, Example 8 can further include, wherein the training particle flow includes a particle flow technique used in a Daum-Huang particle filter, and the internal state and the measurement values ​​are replaced with weights and truth values, respectively.

[0100] In Example 10, at least one of Examples 8-9 can further include, each particle in the set of particles representing all NN network parameters of an instantiation of the NN that collectively represent the distribution of parameters of the BNN.

[0101] In Example 11, at least one of Examples 8-10 can further include, wherein the training particle stream includes iteratively evolving network parameter values ​​along a log homotopy.

[0102] In Example 12, at least one of Examples 8-11 can further include, wherein the training particle flow includes use of a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.

[0103] In Example 13, Example 12 can further include, wherein the training particle stream includes use of a diagonal approximation to the Gauss-Newton approximation.

[0104] In Example 14, Example 13 can further include, wherein the diagonal approximation includes performing a Gauss-Newton approximation for each weight.

[0105] Example 15 includes a device including a memory device including instructions stored thereon; and a processing circuit coupled to the memory device, the processing circuit being configured to execute instructions that, when executed by the processing circuit, cause the processing circuit to perform operations for training and operating a Bayesian neural network (BNN), the operations including initializing particles, each individually representing pointwise values ​​of a NN parameter of a respective NN that collectively represent a distribution of parameters of the BNN; optimizing the particles using a training particle stream to result in an optimized distribution of the parameters; determining a predictive distribution using the optimized distributions of the parameters and predictions from each of the NNs; and resulting in marginal distributions representing the predictive distributions.

[0106] In Example 16, example 15 can further include, wherein the training particle stream includes a particle stream technique used in a Daum-Huang particle filter that replaces internal states and measurements with weights and truth values, respectively.

[0107] In Example 17, at least one of Examples 15-16 may further include, where each particle in the set of particles represents all NN network parameters of an instantiation of the NN that collectively represent the distribution of parameters of the BNN.

[0108] In Example 18, at least one of Examples 15-17 can further include, wherein the training particle stream includes iteratively evolving values ​​of the network parameters along a log homotopy.

[0109] In Example 19, at least one of Examples 15-18 can further include, wherein the training particle stream includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.

[0110] In Example 20, Example 19 may further include, wherein the training particle stream includes using a diagonal approximation to the Gauss-Newton approximation, the diagonal approximation including performing a Gauss-Newton approximation for each weight.

[0111] Although the embodiments have been described with reference to certain exemplary embodiments, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present invention. Accordingly, the specification and drawings should be regarded in an illustrative and not a limiting sense. The accompanying drawings, which form a part of this specification, show by way of illustration, and not by way of limitation, specific embodiments in which the present subject matter may be practiced. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Since other embodiments may be utilized and derived therefrom, structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Accordingly, the detailed description is not to be construed in a limiting sense, and the scope of the various embodiments is defined solely by the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A method for training and operating a Bayesian Neural Network (BNN), comprising: initializing particles each individually representing a pointwise value of a respective NN parameter of a NN which collectively represent a distribution of parameters of said BNN; optimizing said particles using a training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and providing a marginal distribution representing the prediction distribution.

2. The method of claim 1 , wherein the training particle stream comprises a particle stream technique used in a Daum-Huang particle filter that replaces internal states and measurements with weights and truth values, respectively.

3. The method of claim 1 , wherein each particle in the set of particles represents all NN network parameters of the NN instantiation that collectively represent the distribution of parameters of the BNN.

4. The method of claim 1 , wherein the training particle stream comprises iteratively evolving values ​​of the network parameters along a log homotopy.

5. The method of claim 1 , wherein the training particle stream includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.

6. The method of claim 5 , wherein the training particle stream includes using a diagonal approximation to the Gauss-Newton approximation.

7. The method of claim 6 , wherein the diagonal approximation comprises performing a Gauss-Newton approximation for each weight.

8. 1. A non-transitory machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations for training and operating a Bayesian Neural Network (BNN), the operations comprising: initializing particles each individually representing a pointwise value of a respective NN parameter of a NN which collectively represent a distribution of parameters of said BNN; optimizing said particles using a training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and yielding a marginal distribution that represents the prediction distribution.

9. 10. The non-transitory machine-readable medium of claim 8, wherein the training particle stream includes a particle stream technique used in a Daum-Huang particle filter, in which internal states and measurements are replaced with weights and truth values, respectively.

10. 9. The non-transitory machine-readable medium of claim 8, wherein each particle in the set of particles represents all NN network parameters of the NN instantiation that collectively represent the distribution of parameters of the BNN.

11. 10. The non-transitory machine-readable medium of claim 8, wherein the training particle stream comprises iteratively evolving network parameter values ​​along a log homotopy.

12. 10. The non-transitory machine-readable medium of claim 8, wherein the training particle flow includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.

13. 13. The non-transitory machine-readable medium of claim 12, wherein a training particle stream includes use of a diagonal approximation to the Gauss-Newton approximation.

14. 14. The non-transitory machine-readable medium of claim 13, wherein the diagonal approximation comprises performing a Gauss-Newton approximation for each weight.

15. A device, comprising: a memory device containing stored instructions; a processing circuit coupled to the memory device and configured to execute instructions that, when executed by the processing circuit, cause the processing circuit to perform operations for training and operating a Bayesian Neural Network (BNN), the operations including: initializing particles each individually representing a pointwise value of a respective NN parameter of a NN which collectively represent a distribution of parameters of said BNN; optimizing said particles using a training particle stream to produce an optimized distribution of said parameters; determining a prediction distribution using the optimized distributions of the parameters and predictions from each of the neural networks; and yielding a marginal distribution representing the prediction distribution.

16. 16. The device of claim 15, wherein the training particle stream comprises a particle stream technique used in a Daum-Huang particle filter that replaces internal states and measurements with weights and truth values, respectively.

17. 16. The device of claim 15, wherein each particle in the set of particles represents all NN network parameters of the NN instantiation that collectively represent the distribution of parameters of the BNN.

18. The device of claim 15 , wherein the training particle stream comprises iteratively evolving values ​​of the network parameters along a log homotopy.

19. The device of claim 15 , wherein the training particle stream includes using a Gauss-Newton approximation to a Hessian matrix in determining drift and diffusion of the particles.

20. 20. The device of claim 19, wherein the training particle stream includes using a diagonal approximation to the Gauss-Newton approximation, the diagonal approximation including performing a Gauss-Newton approximation for each weight.

Citation Information

Patent Citations

  • System and method for generating adversarial examples

    US20210124999A1