Bayesian flow networks

Abstract

A Bayesian flow network generative model receives a batch of data samples from a data set (205). A total accuracy level is randomly sampled for each data sample of the batch of data samples. A Bayesian flow distribution (210) is constructed corresponding to the total accuracy levels. The Bayesian flow distribution is sampled to produce training input distribution parameters. The training input distribution parameters and the first accuracy level (215) are provided as inputs to a neural network (220). Training output distribution parameters (225) of a training output distribution are received from the neural network, and a discrete or continuous-time loss is determined from the training output distribution parameters.

Description

[0001] Bayesian Flow Networks August 8, 2024 1 CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application serial number 63 / 518,944, filed 08 / 11 / 2023, entitled “Bayesian Flow Networks,” which is incorporated by reference herein in its entirety. 2 FIELD OF THE INVENTION The present invention relates to generative models, and more particularly, is related to generative models applying Bayesian statistics with neural networks. 3 BACKGROUND OF THE INVENTION Learning generative models is a fundamental problem in machine learning. Generative models are so named because they attempt to represent a probabilistic process that is likely to have generated the data. By doing so, a model learned from a set of data points can be used to compute the probability of a data point occurring, or the relative probability of two data points. In addition, the model can be used to generate new samples or data points, that are plausible under the learned probabilistic process. One can also straightforwardly extend any generative modeling framework to conditional generative modeling – where the model is conditioned on additional inputs. For example, a generative model of images can be conditioned on text inputs that can describe the contents of the desired images to be generated. Similarly, a text generative model can be conditioned on additional inputs intended to precede the generated text, thus providing context for the model’s generation. As an example, the context may encode a question, in which case the model will attempt to generate a likely answer. Building a generative model for a given dataset involves two tasks: how to represent the generative model (the model class), and how to train the generative model (the modeling framework). Modern generative models of high-dimensional data (such as image, text, audio, video, molecules etc) all use neural networks as the model class. However, the modeling frameworks are often different depending on the data due to various tradeoffs inherent in the choices. Two modeling frameworks are the most successful and common: autoregressive models and diffusion models. Autoregressive models are the framework of choice for categorical or discrete data, in which data variables always take a value that this one out of several choices, such as words in a vocabulary for text. Diffusion models are commonly used for continuous data, where data variables are modeled as being real-valued, such as pixel values in images. A general way to view such distribution models may be thought of as an exchange of messages transmitted by a sender, Alice, who has access to some data, to a receiver, Alice’s friend Bob, who wishes to receive the data in as few bits as possible. At each step Alice sends (transmits) a message to Bob that reveals something about the data. Bob attempts to guess what the message is: the better his guess the fewer bits are needed to transmit it. After receiving the message, Bob uses the information he has just gained to improve his guess for the next message. The loss function is the cumulative number of bits for all the transmitted messages. For example, in an autoregressive language model the messages are text divided into word-pieces. The distribution encoding Bob’s prediction for thefirst message is of necessity uninformed: a zero-gram prior based on the relative frequencies of different word-pieces. The transmission cost is the negative log-probability under this prior. Bob then uses thefirst word-piece to predict the second; on average, the second prediction will be slightly more informed than thefirst, and the expected transmission cost becomes slightly lower. The process repeats with the predictions improving at each step. The sum of the transmission costs is the negative log-probability of the complete text sequence, which is the loss function minimized by maximum likelihood training. The sum of the transmission costs is also the minimum number of bits that would be required for Alice to transmit the pieces to Bob using arithmetic coding. There is therefore a direct correspondence betweenfitting an autoregressive model with maximum likelihood and training it for data compression. Autoregressive networks are currently state-of-the-art for language modeling, and in general perform well on discrete data where a natural ordering exists. However, autoregressive networks have proven less effective in domains such as image generation, where the data is continuous and no natural order exists among variables (e.g. there is no reason to generate one pixel before another). Autoregressive networks also have the drawback that generating samples requires as many network updates as there are variables in the data. Diffusion models are an alternative framework that have proved particularly effective for image generation. For image generation, the transmission procedure is a little more complex than language modeling, as each received message is a noisy version of the message before, where the noise is designed so that in expectation the messages approach the data The transmission cost at each step is the Kullback-Leibler (KL) divergence between the distribution from which Alice draws the message and Bob’s prediction of that distribution (which is a reparameterisation of his prediction of the data). The sum of the KL divergences is the evidence lower bound minimized by diffusion training; it is also the expected number of bits needed to transmit the data using an efficient bits-back coding scheme. Once again there is an exact equivalence between the loss function used to train the model and the model’s ability to compress data. Diffusion may be advantageous over autoregression for image generation in the way diffusion progresses from coarse tofine image details as the level of noise decreases — a more natural way to construct an image than one dot at a time. However diffusion has yet to match autoregression for discrete data. One challenge is that when the data is discrete the noise in the diffusion process is also discrete, and therefore discontinuous. To return to the transmission metaphor, if the data is a piece of text, then Bob begins the process with a totally garbled text, every symbol of which is either randomly altered or left unchanged by each of Alice’s messages. Bob’s task would be easier if Alice were able to smoothly transform his beliefs about the data, as she does for continuous data. Therefore, there is a need in the industry to address these deficiencies. 4 SUMMARY OF THE INVENTION Embodiments of the present invention provide a framework for a Bayesianflow network. Briefly described, the present invention is directed to a Bayesianflow network generative model framework. In one aspect, a Bayesianflow network generative model framework receives a batch of data samples from a data set (205). A total accuracy level is randomly sampled for each data sample of the batch of data samples. A Bayesianflow distribution (210) is constructed corresponding to the total accuracy levels. The Bayesianflow distribution is sampled to produce training input distribution parameters. The training input distribution parameters and thefirst accuracy level (215) are provided as inputs to a neural network (220). Training output distribution parameters (225) of a training output distribution are received from the neural network, and a discrete or continuous-time loss is determined from the training output distribution parameters. In another aspect of the present invention, a sampling function receives parameters for afirst input distribution (215) and selects afirst accuracy level. Thefirst input distribution parameters and thefirst accuracy level are provided to a neural network (220). Output distribution parameters (225) of an output distribution are received from the neural network. An output prediction (325) is sampled from the output distribution (320). A sender sample (335) is selected from a sender distribution (330). The input distribution parameters (345) is updated by a Bayesian update function (340), and a second accuracy level (355) is selected for a subsequent sample function pass. Other systems, methods and features of the present invention will be or become apparent to one having ordinary skill in the art upon examining the following drawings and detailed description. It is intended that all such additional systems, methods, and features be included in this description, be within the scope of the present invention and protected by the accompanying claims. 5 BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present invention. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. FIG. 1 represents one step of the modeling process of a Bayesian Flow Network. FIG. 2 is aflowchart of a training method under an exemplaryfirst embodiment. FIG. 3 is aflowchart of a generation method under an exemplaryfirst embodiment. FIG. 4 is a plot of exemplary Bayesian updates for continuous data. FIG. 5 is a plot of an exemplary Bayesian update distribution for continuous data. FIG. 6 is a plot illustrating an example of input variance for Bayesian Flow Networks and diffusion models. FIG. 7 is a plot of an example of sender, output and receiver distributions for continuous data. FIG. 8 is a plot of an exemplary output distribution for discretized data. FIG. 9A is a plot of an exemplary sender distribution for discretized data. FIG. 9B is a plot of an exemplary output distribution for discretized data. FIG. 9C is a plot of an exemplary receiver distribution for discretized data. FIG. 10 is a plot illustrating an example of an accuracy schedule vs. expected entropy for discrete data. FIG. 11A shows an example of MNIST real data. FIG. 11B shows an example of MNIST generated data. FIG. 12A shows an example of MNIST input distribution. FIG. 12B shows an example of MNIST output distribution. FIG. 13A is afirst plot of MNIST losses against time. FIG. 13B is a second plot of MNIST losses against time. FIG. 14A is an example of CIFAR-10 real data with 256 bins. FIG. 14B is an example of CIFAR-10 generated data with 256 bins. FIG. 14C is an example of CIFAR-10 real data with 16 bins. FIG. 14D is an example of CIFAR-10 generated data with 16 bins. FIG. 15A is an example of an CIFAR-10 input distribution. FIG. 15B is an example of an CIFAR-10 output distribution. FIG. 16A is a plot of CIFAR-10 time loss. FIG. 16B is a plot of CIFAR-10 cumulative time loss. FIG. 17A shows an example of text8 real data. FIG. 17B shows an example of text8 generated data. FIG. 18 is a schematic diagram illustrating an example of a system for executing functionality of the present invention. 6 DETAILED DESCRIPTION The following definitions are useful for interpreting terms applied to features of the embodiments disclosed herein, and are meant only to define elements within the disclosure. As used within this disclosure, an “autoregressive (AR) model” refers to a representation of a type of random process. An AR model may describe certain time-varying processes in nature, economics, behavior, among others. The autoregressive model specifies that the output variable depends linearly on its own previous values and on a stochastic (imperfectly predictable) term. The AR model is generally in the form of a stochastic difference equation. As used within this disclosure, “probabilistic modeling” refers to a statistical approach incorpo- rating the effect of random occurrences or actions to forecast the possibility of future results. As used within this disclosure, a “generative model” refers to a probabilistic model of the joint probability distribution P(X,Y) on given observable variable X and a target variable Y. Generative models generate new data that looks like existing (real) data. A generative model is a type of machine learning model that learns underlying patterns or distributions of data in order to generate new, similar data. There are many examples of practical applications for generative models, including art / music creation, drug discovery (predicting molecular structures for new potential drugs), image recognition, content creation, and video game design, amongst others. As used within this disclosure, a “distribution” generally refers to a probability distribution. A probability distribution is a function that describes the likelihood of different outcomes for a random variable. Several types of distributions are described below. A generative model includes a distribution of the data itself, and indicates the likelihood of a give sample. The exemplary embodiments herein are directed to modeling data consisting of multiple variables (”data variables”). For example, images of size NxN may be represented as being composed of 3xNxN continuous data variables whose values are the pixel intensities for red, green and blue channels at each location. As another example, text strings of length L can be represented as having L discrete data variables each of which may take K possible values, where K is the size of a vocabulary. As used within this disclosure, “predictive modeling” refers to a statistical technique to predict future behavior. Predictive modeling solutions are a form of data-mining technology that analyze historical and current data and generate a model to predict future outcomes. As used within this disclosure, a “generative model” refers to a class of statistical models that can generate new data instances. Generative models may be used in unsupervised machine learning as a means to perform tasks. For example, generative models can be used to perform probability and likelihood estimation, model data points, describe phenomena in the data, or distinguish between classes based on these probabilities. For example in the context of multimedia, generative models can be used to generate data such as images and audio. As used within this disclosure, “generative artificial intelligence (AI)” refers to artificial intelli- gence capable of generating text, images, or other media in response to prompts. Generative AI models learn the patterns and structure of their input training data by applying neural network machine learning techniques, and then generate new data that has similar characteristics. As used within this disclosure, a “loss function” refers to a function that compares a target value and a predicted output value. The loss function measures how well a neural network models training data. When training, an objective is to minimize the loss between the predicted and target outputs. As used within this disclosure, “Kullback–Leibler (KL) divergence” refers to a method for measuring a difference between two probability distributions. As denoted, DKL (P —— Q) indicates a type of statistical distance, namely a measure of how one probability distribution P is different from a second, reference probability distribution Q. As used within this disclosure, “Bayesian statistics” refers to an approach to data analysis and parameter estimation based on Bayes’ theorem. Under Bayesian statistics, all observed and unobserved parameters in a statistical model are given a joint probability distribution, termed the prior (observed parameters) and data (unobserved parameters) distributions. As used within this disclosure, “Bayesian inference rules” refers to Bayesian a method of statistical inference in which Bayes’ theorem is used to update the probability for a hypothesis as more evidence or information becomes available. In the philosophy of decision theory, Bayesian inference is closely related to subjective probability, often called a “Bayesian probability.” As used in this disclosure, a “prior,” or “prior probability” is used in the context of a Bayesian statistical inference, and refers to the probability of an event occurring before new data is collected. Here, a prior represents the best rational assessment of the probability of a particular outcome based on current knowledge before an experiment is performed. As used within this disclosure, a “simple prior” generally refers to a uniform noise distribution. As used within this disclosure a “noise distribution” generally refers to a Gaussian noise distribution. Generally, Gaussian noise is a type of noise that is generated by adding random values that are normally distributed with a mean of zero and a standard deviation to the input data. The normal distribution, also known as the Gaussian distribution, is a continuous probability distribution that is defined by its probability density function (PDF). Gaussian noise can be added to input data or weights by adding generated random values with a normal distribution. As used within this disclosure, an “autoencoder” refers to a type of artificial neural network used to learn efficient codings of unlabeled data (unsupervised learning). In general, an autoencoder learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation. For example, an autoencoder may be an unsupervised artificial neural network that learns how to efficiently compress and encode data and then learns how to reconstruct the data back from the reduced encoded representation to a representation that is as close to the original input as possible As used within this disclosure, a “deep autoencoder” refers to an autoencoder having two symmetrical deep-belief networks with a plurality of shallow layers. Afirst network of the two symmetrical deep-belief networks represents the encoding half of the network and the secondfirst network of the two symmetrical deep-belief networks network makes up the decoding half. As used within this disclosure, a “Variational Autoencoder (VAE)” refers to an autoencoder whose encodings distribution is regularized during a training to ensure that its latent space has properties allowing for generation of new data. Here, the term “variational” indicates the close relation between the regularization and the variational inference method in statistics. As used with this disclosure, a “diffusion model” refers to a class of latent variable models in machine learning. Diffusion models are also known as diffusion probabilistic models or score-based generative models. A diffusion model may include, for example, Markov chains trained using variational inference. As used within this disclosure, “data transmission” or just “transmission” refers to providing data as input to a neural network, for example, compressed image data (pixels) in the form of bits or nats. Transmission may typically occur over time as a sequence of transmission steps. As used within this disclosure, a “nat” refers to the natural unit of information. A nat is a unit of information or information entropy, based on natural logarithms and powers of e, rather than the powers of 2 and base 2 logarithms. One nat is the information content of an event when the probability of that event occurring is 1 / e. As used in this disclosure, data may be described as being one of three types of data: continuous, discretized, and discrete. Continuous data refers to data that can be measured. This data has values that are notfixed and have an infinite number of possible values. These measurements can also be broken down into smaller individual parts. While in general data processed by a computer is not actually continuous, for the purposes of this disclosure the data is referred to as being continuous if a corresponding loss function is configured to treat the descretized data as continuous data. Discrete data (also referred to as discrete values) is data that only takes certain values, for example whole numbers or integers, this is data that can be counted and has afinite number of values. Discretized (descretized) data may be thought of as bins or groupings of ranges (segments) of data. For example, real numbers may be discretized into bins corresponding to whole numbers n, where each bin collects data falling between (n - ½) and (n + ½). For this disclosure, the loss functions described in the embodiments may be specifically directed to discretized data, typically over the value range between -1 and +1. As used within this disclosure, the “input space” refers to all potential sets of values for input to a supervised learning model. In general, the input space is classified as having continuous data, discrete data, and / or discretized data. As used within this disclosure, “transmission” refers to a process wherein an electronic device that transmits a (sender) data sample to a receiver device in a compressed form. The efficiency of the transmission is measured by the amount of compressed information, measured in bits or nats, that needs to be transmitted so that the receiver can reconstruct the data sample precisely. The transmission process serves as practical motivation for learning a generative model and defining its objective function, since it is well known that a better generative model makes the transmission of a set of data samples more efficient on average such that fewer bits need to be sent. Therefore, even though the embodiments may serve many purposes such as data compression, measurement of likelihood and generation of new data samples, transmission embodiments are useful to illustrate BFN through the lens of data compression for efficient transmission. As used within this disclosure, a “transmission step” refers to a single step of a transmission process. In this disclosure, each transmission step transmits a single noisy data sample at a particular noise level obtained by adding noise to a ”clean” data sample. Adding noise reduces information, so transmitting data with higher amount of added noise requires fewer bits. As used within this disclosure, “accuracy” refers to a scalar quantity ranging from zero to infinity having a one-to-one inverse relationship with noise. Zero accuracy implies infinite noise, while increasing accuracy implies decreasing noise, and in the limit infinite accuracy implies zero noise. Herein, the embodiments are described in terms of accuracy levels instead of noise levels for ease of mathematical treatment. As used within this disclosure, an “accuracy schedule” refers to a mathematical function whose input is the transmission step number, and the output is the cumulative accuracy level at that step, that is, the sum of accuracy levels up to that step. The transmission step numbers may be represented as non-negative integers or as real numbers by mapping the non-negative integers to the interval between zero and one. As used within this disclosure, a “sender distribution” refers to a distribution configured to send noisy samples to a receiver distribution. At each transmission step, the sender corrupts the ”clean” data sample to obtain a sender distribution at the accuracy level defined by the accuracy schedule. For example, for continuous data, the sender adds Gaussian noise with a precision given by the accuracy schedule. Notably, the added noise is applied independently to each ”variable” of the data. As used within this disclosure, a “sender sample” refers to a sample from the Sender distribution to be compressed and transmitted at each transmission step. Sender Space: refers to all potential sets of values for sender samples. Depending on the data type, the sender space may be same as the input space. This is the case for continuous data. For discrete data, this disclosure defines a type of noise such that the sender space is continuous instead. As used within this disclosure, an “input distribution” refers to a distribution for each data variable for which parameters are used as inputs to the neural network. As used within this disclosure, an “output distribution” refers to a distribution for each data variable for which parameters are predicted by the neural network. As used within this disclosure, a “Bayesian update distribution” refers to a distribution over the parameters of the input distribution. For example, the Bayesian update distribution may be computed by the mathematical procedure described in the ”Bayesian Updates” section. As used within this disclosure, a “Bayesianflow distribution” refers to a distribution over the parameters of the input distribution. For example, the Bayesianflow distribution may be computed by the mathematical procedure described in the Bayesian Flow Distribution section. A distribution may be described via a set of distribution parameters. For example, the parameters for a normal distribution may be the distribution class (e.g., Gaussian), mean, and variance. Distribution parameters may be provided to a neural network in lieu of directly providing distribution data. As used within this disclosure, a ”categorical distribution” refers to a discrete probability distribution that describes the possible results of a random variable that can take on one of K possible categories, with the probability of each category separately specified. The distribution parameters specifying the probabilities of each possible outcome are in the range 0 to 1, and all must sum to 1. A categorical distribution may also be called a generalized Bernoulli distribution, or a multinoulli distribution. 6.1 OVERVIEW Bayesian Flow Networks (BFN) are a new class of generative model where the parameters of a set of independent distributions are modified with Bayesian inference in view of noisy data samples, then passed as input to a neural network. The neural network outputs a second, interdependent distribution. Starting from a simple prior and iteratively updating the two distributions yields a generative procedure similar to the reverse process of diffusion models; however a BFN is conceptually simpler in that no forward process is required. Discrete and continuous-time loss functions are derived for continuous, discretized, and discrete data, along with sample generation procedures. Notably, the network inputs for discrete data lie on the probability simplex, and are therefore natively differentiable, facilitating gradient-based sample guidance and few-step generation in discrete domains such as language modeling. The loss function directly optimizes data compression and places no restrictions on the network architecture. Exemplary embodiments of the present Bayesian Flow Networks (BFN) invention are directed to a single modeling framework that naturally extends to continuous and discrete domains. Under the embodiments, a unified framework enables a single stack of techniques across data modalities and disciplines, enabling advances from one domain to be easily carried forward to others, making it easier to learn generative models of multiple data modalities jointly. Besides the gains from unification of modalities, a generative modeling framework retains the attractive properties of existing methods while avoiding their pitfalls. For example, in autoregressive models, performing sampling to generate data requires as many calls to the models as there are variables in the data. Diffusion models do not have such a restriction, which makes them attractive. The exemplary embodiments address a need for frameworks that avoid this restriction (like diffusion models) while being effective at modeling discrete variables (like autoregressive models). The BFN model introduced in these embodiments, can be summarized by the following trans- mission scheme. FIG. 1 presents an overview of a system 100 under the present embodiments. FIG. 1 represents one step of the modeling process of a Bayesian Flow Network. The data in this example is a sequence of ternary symbols 120, of which thefirst two variables (variable 1,‘B and variable 2,‘A) are shown. At each step a network 110 emits the parameters of an output distribution 130 based on the parameters of a previous input distribution. A sender distribution 140 and a receiver distribution 150 (both of which are continuous, even when the data is discrete) are created by adding random noise 160 to the data and the output distribution respectively. A sample from the sender distribution 140 is then used to update the parameters of the input distribution, following the rules of Bayesian inference. Continuing with the example introduced in the Background section, conceptually, this may be though of the message sent by Alice to Bob, and its contribution to the loss function is the KL divergence from the receiver to the sender distribution. Bob then uses the sample to update his input distribution, following the rules of Bayesian inference. Usefully, the Bayesian updates are available in closed-form as long as the input distribution models all the variables in the data independently. Once the update is complete, Bob again feeds the parameters of the input distribution to the network which returns the parameters of the output distribution. The process repeats for n steps, at which point Bob can predict the data accurately enough that Alice can send it to him without any noise. Note the key difference between the input and output distributions: the input distribution receives information about each variable in the data independently (via the Bayesian updates), and is therefore unable to exploit contextual information, such as neighboring pixels in an image or related words in a text; the output distribution, on the other hand, is produced by a neural network that jointly processes all the parameters in the input distribution, giving it access to all available context. Intuitively, the combination of the input and output distributions represents a division of labour between Bayesian inference and deep learning that plays to both of their strengths: the former provides a mathematically optimal andfinely controllable way to collect and summarize information about individual variables, while the latter excels at integrating information over many interrelated variables. The above transmission process defines an n-step loss function that can be generalized to continuous time by sending n to ∞. In continuous time, the Bayesian updates become a Bayesian flow of information from the data to the network. As well as removing the need to predefine the number of steps during training, the continuous-time loss function is easier to compute than the discrete-time loss. A BFN trained with continuous-time loss may be run for any number of discrete steps during inference and sampling, with performance improving as the number of steps increases. The Bayesian Flow Networks described herein have many practical applications. For example, in thefield of graphical design, a logo designer is interested in exploring the space of new logos for design ideas. The designer decides to use generative models to model a logo design dataset and generate samples from the generative model. The designer then decides to treat logo images as continuous valued data, and normalizes logo images so each pixel value is treated as a continuous value over a range of [0, 1]. The designer chooses the continuous data formalism of BFNs, which defines the forms of noise process, input, output, receiver distributions, among others, as well as the Bayesian updates and Bayesian Flow distributions corresponding the noise process. The designer uses a trainingflow chart of FIG. 2 for training the model, and a generationflowchart of FIG. 3 for generating samples. The designer may use the generated logos for new design project, or to provide ideas the designer may manually incorporate into new logo designs. FIG. 2 is aflowchart 200 of a forward pass for a training method under an exemplaryfirst embodiment. It should be noted that any process descriptions or blocks inflowcharts should be understood as representing modules, segments, portions of code, or steps that include one or more instructions for implementing specific logical functions in the process, and alternative implementations are included within the scope of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention. Training samples are sampled from a dataset, as shown by block 205. Here, a total accuracy level is randomly sampled for each data sample of the received data set, and a Bayesianflow distribution is constructed corresponding to these total accuracy levels. A sample is randomly drawn from the Bayesianflow distribution, as shown by block 210. This is in contrast with diffusion models, where there is no Bayesianflow distribution, and noise is sampled during training and added to the data. It should be noted that diffusion models refer to noise, while the present embodiments refer to accuracy levels, which are inversely related to noise levels. Input distribution parameters and an accuracy level (block 215) are provided as input to a neural network, as shown by block 220. For example, the accuracy level for thefirst forward pass is the lowest accuracy level of a pre-determined accuracy schedule. Note, the type of neural network is selected appropriately for the corresponding input data. For example, a convolutional neural network (CNN) may be selected for imaging data, or a transformer network may be selected for text. The input distribution parameters are obtained by sampling the Bayesianflow distribution. This is in contrast with diffusion models, where the input is merely the data plus noise, as mentioned above. The neural network produces output distribution parameters, as shown by block 225. A discrete or continuous time loss (block 235) is computed, as shown by block 230, according to whether the dataset represents discrete or continuous time data. Once the forward pass of the training is complete, a corresponding backward pass is executed. For the backward pass, automatic differentiation (AD) techniques may be used to compute the gradient of the loss 235 with respect to the network parameters, for example, using widely used open source software such as TensorFlow or PyTorch. The network parameters are updated using the computed gradient and applying a stochastic gradient descent algorithm, such as ADAM. Thereafter the forward / backward paths may be repeated until training is complete. Note that FIG.2 shows how a “loss” value is computed on a batch of data sampled from a dataset. Any competent practitioner can train a model once they know how to compute the loss, since widely used software packages like PyTorch or Tensorflow allow one to compute the gradients of the loss with respect to the neural network parameters, and update the parameters in a direction that reduces the loss in expectation. The details for implementing the training of FIG. 2 depend upon the type of data being used. The training algorithms for continuous data are described by Algorithm 1 (discrete-time loss), and Algorithm 2 (continuous-time loss). The training algorithms for discretized data are described by Algorithm 4 (discrete-time loss), and Algorithm 5 (continuous-time loss). The training algorithms for discrete data are described by Algorithm 7 (discrete-time loss), and Algorithm 8 (continuous-time loss). FIG. 3 is aflowchart 300 of a sample generation method under an exemplaryfirst embodiment. Here, a sample is generated from an input distribution per iteration. The input distribution is updated with each pass, informing subsequent generated samples. It should be noted that blocks with the same reference numerals as FIG. 2 correspond to the same or similar elements. Input distribution parameters and accuracy levels (block 215) are provided as input to the neural network, as shown by block 220. In the initial forward pass, the accuracy level is the lowest accuracy level of the same accuracy schedule used for the training of FIG. 2. It should be noted that BFNs work with input distribution parameters, in contrast to diffusion networks, that work with noise-added inputs. The neural network 220 produces output distribution parameters, as shown by block 225. A sample output prediction (block 325) is produced from the output distribution, as shown by block 320. A sender distribution is (randomly) sampled at a current accuracy level, as shown by block 330, yielding a sender sample 325. Here, the sender sample 335 may be thought of as being analogous to the noise-added input of a diffusion model, but unlike diffusion models, for Bayesianflow networks the sender samples are predicted, not inputs to the network. The predicted sender samples are fed into a Bayesian update function (block 340) that applies the rules of Bayesian inference to update the input distribution parameters (block 345) for the next pass. It should be noted that diffusion models do not include a Bayesian update function (block 340) producing updated input distribution parameters (block 345). The accuracy level is increased and the method iterates, as shown by block 355. Here, the accuracy level is the next higher accuracy in the accuracy level schedule (e.g., found by incrementing an accuracy level index corresponding to each of the accuracy levels in the accuracy schedule). In each subsequent pass sample generation pass, the input distribution parameters and accuracy level (block 215) are those updated in the previous pass. The implementation details for the sample generation of FIG. 3 depend upon the type of data being used, namely, whether the data is continuous, discretized, or discrete. The type of data also informs the selection of input distribution. These implementation details are described in detail below, and summarized in tables displaying of pseudo code algorithms. The training algorithm for continuous data is described by Algorithm 3, the training algorithm for discretized data is described by Algorithm 6. The training algorithms for discrete data is described by Algorithm 9. The following provides further details for the framework of BFN framework, followed by a general derivation of the discrete and continuous time loss functions provided (Section 7), and specializations of the framework to continuous, discretized, and discrete data (Sections 8–10), along with pseudocode for training, evaluating and sampling from the network. 7 Bayesian Flow Networks The following describes a framework for embodiments for Bayesian Flow Networks, laying out the structure of the various functions and distributions incorporated by the model, along with the discrete and continuous-time loss functions used for training. It should be noted while the framework is expressed using mathematical expressions typically used by practitioners in thisfield to effectively and efficiently communicate the technical features of the embodiments, the exemplary embodiments are not directed to mathematical concepts. Specific instantiations of the general framework for continuous, discretized, and discrete data are given in Sections 8–10. 7.1 Input and Sender Distributions Given D-dimensional databe the parameters of a factorized input distribution pI(· | θ), with For example, θ(d)may consist of the probabilities of a categorical distribution. Let pS(· | x;α) be a similarly factorized sender distribution withand where α ∈ R+is an accuracy parameter defined such that when α = 0, the sender samples are entirely uninformative about x and as α increases the samples become progressively more informative. 7.2 Output Distribution pO(· | θ, t) During the data transmission process, the input parameters θ are passed along with the processtime t as input to a neural network . The network then emits an output vector Ψ(θ, t) =) which is used to parameterize an output distribution factorized in the same way as the input and sender distributions: As discussed in the Background section, an important difference between the input and output distri- s is that while each pI(x(()butiond)| θ(d)) depends only on information gathered via about x(d), each pO(x(d)| Ψ(d)(θ, t)) depends (via the network) on all of θ and hence all of x. The output distribution, unlike the input distribution, may therefore exploit context information, such as surrounding pixels in an image or related words in a text. 7.3 Receiver Distribution pR(· | θ; t, α) Given sender distribution pS(· | x;α) and output distribution pO(· | θ, t) the receiver distribution over YDis defined as Intuitively this may be understood as a receiver that knows the form of the sender distribution pS(· | x;α) but does not know x, and therefore integrates over all x′ ∈ XD, and hence all possible sender distributions, weighted by the probability given to x′ by the output distribution pO(x | θ, t). The receiver distribution therefore combines two sources of uncertainty: the “known unknown” of the sender distribution entropy (which is a function of α), and the “unknown unknown” of the output distribution entropy. 7.4 Bayesian Updates Given parameters θ and sender sample y drawn with accuracy α the Bayesian update function h is derived by applying the rules of Bayesian inference to compute the updated parameters θ′: The Bayesian update distribution pU(· | θ,x;α) is then defined by marginalizing out y: () where δ (· − a) is the multivariate Dirac delta distribution centered on the vector a. Sections 8.4 and 10.7 show that both forms of pU(· | θ,x;α) considered here have the following property: the accuracies are additive in the sense that if α = αa+ αbthen It follows from this property that given prior input parameters θ0, the probability of observing parameters θnafter drawing a sequence of n sender samples y1, ... ,ynwith accuracies α1, ... , αnis ( ) 7.5 Accuracy Schedule β(t) By performing an infinite number of transmission steps, the Bayesian update process can be generalized to continuous time. Let t ∈ [0, 1] be the process time and let α(t) > 0 be the accuracy rate at time t. Now define the accuracy schedule β(t) as It follows from the above definitions that β(t) is a monotonically increasing function of t, that β(0) = 0, and that Specific forms of β(t) for continuous and discrete data are provided in Sections 8.5 and 10.8. Both may be derived, for example, using simple heuristics. 7.6 Bayesian Flow Distribution pF(· | x; t) Given prior parameters θ0, Bayesian update distribution pU(· | θ,x;α) and accuracy schedule β(t), the Bayesianflow distribution pF(· | x; t) is the marginal distribution over input parameters at time t, defined by 7.7 Loss Function L(x) Given prior parameters θ0and accuracy schedule consider a sequence of n sender samples y1, ... ,ynsampled at times t1, ... , tnwhere ti= i / n. The sender distribution at step i is pS(· | x;αi) where the receiver distribution at step i is pR(· | θi−1; ti−1, αi), and the input parameter sequence θ1, ... ,θnis recursively calculated from Define the n-step discrete-time loss Ln(x) as the expected number of nats required tofirst transmit y1, ... ,yn, and the reconstruction loss Lr(x) as the expected number of nats required to then transmit x. Since — using a bits-back coding scheme — it requires DKL(pS‖ pR) nats to transmit a sample from pSto a receiver with pR, where and since the number of nats needed to transmit x using an arithmetic coding scheme based on p(x) is − ln p(x), and the marginal probability of θnis given by pF(· | x, 1), Note that Lr(x) is not directly optimized in this paper; however it is indirectly trained by optimizing Ln(x) since both are minimized by matching the output distribution to the data. Furthermore, as long as β(1) is high enough, the input distribution at t = 1 will be very close to x, making it trivial for the network tofit pO(x | θ; 1). The loss function L(x) is defined as the total number of nats required to transmit the data, which is the sum of the n-step and reconstruction losses: L(x) = Ln(x) + Lr(x) (16) Alternatively L(x) can be derived as the loss function of a variational autoencoder (VAE).Consider the sequence y1, ... ,ynas a latent code with posterior probability given by and autoregressive prior probability given by Then, noting that the decoder probability p(x | y1, ... ,yn) = pO(x | θn; 1), the complete trans- mission process defines a VAE with loss function given by the negative variational lower bound (VLB) 7.8 Discrete-Time Loss Ln(x) Eq. 13 can be rewritten as where U{1, n} is the uniform distribution over the integers from 1 to n. Furthermore, it follows from Eqs. 8 and 10 that and hence which allows us approximate Ln(x) via Monte-Carlo sampling without computing the n-step sum. 7.9 Continuous-Time Loss L∞(x) Eq. 24 can be used to train the network directly. However this presupposes that n isfixed during training. Furthermore, for discrete and discretized data the KL terms do not have analytic solutions, leading to noisy gradient estimates. Inspired by Variational Diffusion Models [? ] a continuous-time loss function L∞(x) may be derived by taking the limit of Ln(x) as n → ∞. This turns out to be mathematically simpler than the discrete-time loss, as well as removing both the noisy gradients for the discrete and discretized KL terms and the need tofix n during training. Let Then, from the definition of Ln(x) in Eq. 24, where U(a, b) is the continuous uniform distribution over the interval [a, b]. As described below, for all the sender, receiver distribution pairs in the exemplary embodiments, where g : X → Y is a function from data space to sender space, P(d)(θ, t) is a distribution over Y withfinite expectation and variance, ∗ denotes the convolution of two probability distributions and C is a scalar constant. The following proposition is now required: Proposition 7.1. For a continuous univariate probability distribution P withfinite expectation () ( ) 2 ( ) Proof. Let ∈ be some variance in the interval and consider the sequence of random variables

[0002] It follows from the continuity of β(t) and Eq. 26 that Therefore, Proposition 7.1 can be applied to Eq. 29 to yield where Therefore, Substituting from Eq. 26, and hence 7.10 Sample Generation Given prior parameters θ0, accuracies α1, ... , αnand corresponding times ti= i / n, the n-step sampling procedure recursively generates θ1, ... ,θnby sampling x′ from pO(· | θi−1, ti−1), y from pS(· | x′, αi) (meaning that y ~ pR(· | θi−1; ti−1, αi) — see Eq. 4), then setting θi= h(θi−1,y). Given θnthe network is run one more time and thefinal sample is drawn from pO(· | θn, 1). 8 Continuous Data For continuous data X = R and hence x ∈ RD. In the experiments, x is normalised to lie in [−1, 1]Dto ensure that the network inputs remain in a reasonable range; however this is not essential for the mathematical framework. 8.1 Input Distribution pI(· | θ) The input distribution for continuous data is a diagonal normal: where I is the D ×D identity matrix. The prior parameters are defined here as where 0 is the length D vectors of zeros. Hence the input prior is a standard multivariate normal: The usual Bayesian approach would be tofit the prior mean and variance to the training data. However, a standard prior may work better in practice, as well as simplifying the equations. It is important to remember that the distributions pI(x | θ0) are never used directly to make predictions, but rather to inform the network’s predictions. All that matters is that the parameters fed into the network accurately and accessibly encode the information received so far about x. The network can easily learn the empirical prior of the training set and use that to correct its predictions. 8.2 Bayesian Update Function h(θi−1,y, α) Given a univariate Gaussian prior( ) over some unknown data x the Bayesian posteriorafter observing a noisy sample y from a normal distribution( ) with known precision α is() where Since both pI(x | θ) and pS(y | x;α) distributions are normal with diagonal covariance, Eqs. 46 and 47 can be applied to obtain the following Bayesian update function for parameters θi−1= {μi−1, ρi−1} ( ) and sender sample y drawn from with FIG. 4 is a plot illustrating Bayesian updates for continuous data. For univariate data x = 0.7, the initial input distribution parameters θ0= {μ0= 0, ρ0= 1} are updated to θ1= {μ1, ρ1}, θ2= {μ2, ρ2}, θ3= {μ3, ρ3} by iterating Eqs. 49 and 50 with sender samples y1, y2, y3drawn with accuracies 2, 4, 6 respectively. FIG. 4 shows how the input mean (μ1, μ2, μ3) stochastically approaches the data, while the input precision smoothly increases. 8.3 Bayesian Update Distribution pU(· | θ,x;α) Eq. 50 computes μigiven a single sample y from the sender distribution. To marginalise over ()as defined in Eq. 6, the following standard identity for normal distributions can be applied: Substituting ( ) and therefore (since μiis the only random part of θi) ( ) FIG. 5 is a plot of an exemplary Bayesian update distribution for continuous data. For x = 0.7, the plot shows the distribution p(μ | θ0, x;α) over input mean μ from Eq. 52 given initial parameters and 11 α values spaced log-linearly between Note how the distribution is tightly concentrated around μ0for very low alpha, then smoothly progresses to a tight concentration around x for high alpha 8.4 Additive Accuracies The sender accuracies are may be checked to be additive in the sense required by Eq. 7 byfirst observing that is drawn from then Define and apply Identity 51 with to see that Now observe that if θi= {μi, ρi} is drawn from p(· | θi−1,x;αb) then ( ) and hence where ( ) Another standard identity for Gaussian variables can now be applied: () ( ) ( ) to see that and hence as required. 8.5 Accuracy Schedule β(t) One may derive β(t) for continuous data by requiring that the expected entropy of the input distribution linearly decreases with t. Intuitively, this means that informationflows into the input distribution at a constant rate. Define Then if H(t) linearly decreases with t, Define σ1to be the standard deviation of the input distribution at t = 1. σ1may be chosen empirically to minimise the loss; in general it should be small enough to ensure that the reconstruction loss is low, but not so small as to create unnecessary transmission costs. Recalling that the precision ρ at time t is 1 + β(t), it follows that

[0003] Therefore 8.6 Bayesian Flow Distribution pF (· | x; t) Recall from Eq. 10 that Therefore, setting θi−1= θ0= {0, 1} and α = β(t) in Eq. 53, and recalling that ρ = 1 + β(t), where FIG. 6 is a plot illustrating an example of input variance for Bayesian Flow Networks and diffusion models. For σ1= 0.001 and γ(t) defined as in Eqn. 80, the blue line shows the variance γ(t)(1− γ(t)) of the distribution over the input mean μ as a function of t (see Eq. 77). Note that the variance is 0 at t = 0 (since the input prior μ0is deterministic) and becomes small again as t approaches 1 and μ becomes increasingly concentrated around the data. The green and red lines show the equivalent network input variance for two different noise schedules from the literature (linear and cosine) during the reverse process of a diffusion model (note that t is reversed relative to diffusion convention). The input variance is much lower for Bayesian Flow Networks. 8.7 Output Distribution pO(· | θ; t) Following standard practice for diffusion models, the output distribution is defined by reparameter- ising a prediction of the Gaussian noise vector ∈ ~ N (0, I) used to generate the mean μ passed as input to the network. Recall from Eq. 77 that and hence The network outputs an estimate of ∈ and this is transformed into an estimate of x by Given the output distribution is Note that γ(0) = 0, making the transformation from to pO(x | θ; t) undefined at t = 0. Therefore pO(x | θ; t) = 0 is set for t under some small threshold tmin. Also, is clipped to lie within the allowed range [xmin, xmax] for x. In experiments tmin= 1e−6 and [xmin, xmax] . 8.8 Sender Distribution pS(· | x;α) The sender space Y = X = R for continuous data, and the sender distribution is normal with precision α: 8.9 Receiver Distribution pR(· | θ; t, α) Substituting Eqs. 85 and 86 into Eq. 4, FIG. 7 is a plot of an example of sender, output and receiver distributions for continuous data. Here, the sender and receiver distributions have identical variance and the output distribution is a Dirac delta distribution centered on the network prediction 8.10 Reconstruction Loss Lr(x) Truly continuous data requires infinite precision to reconstruct, which makes the reconstruction loss problematic. However it would be reasonable to assume that either the data isfinely discretized (as all information is on a digital computer), or that it contains some noise. The reconstruction loss for discretized data is presented in Section 9.3. Alternatively, assuming the presence of normally distributed measurement noise on x, withfixed isotropic variance σ2, then a noisy version of thereconstruction loss can be defined as the expected KL divergence between N( )and the output distribution at t = 1: The noise does not directly affect training, as the reconstruction loss is not optimized. However the value of σ places a natural upper limit on the value that should be chosen for σ1: there is no point transmitting the data to greater precision than it was originally measured. Empirically, when σ1< σ / 2 the reconstruction loss is very small. 8.11 Discrete-Time Loss Ln(x) From Eqs. 86 and 88, and from Eqs. 11 and 72, Therefore, substituting into Eq. 24, where ti−1= (i− 1) / n. 8.12 Continuous-time Loss L∞(x) Eq. 29 claimed that for some embedding function g : X → Y , constant C and distribution pθover YDwithfinite mean and variance. If g is the identity function, C = 1 and then P (θ, t) hasfinite mean and variance and so the claim is true and the continuous-time loss from Eq 41 applies, with and α(t) as defined in Eq 74, yielding 8.13 Pseudocode Pseudocode for evaluating the n-step loss Ln(x) and continuous-time loss L∞(x) for continuous data is presented in Algorithms 1 and 2, while the sample generation procedure is presented in Algorithm 3.

[0004] 9 discretized Data This section considers continuous data that has been discretized into K bins. For example, 8-bit images are discretized into 256 bins, 16-bit audio is discretized in 216= 65, 536 bins. This data is represented by tiling [−1, 1] into K intervals, each of length 2 / K. Let kl, kcand krdenote respectively the left, centre and right of interval k, and let {1,K} denote the set of integers from 1 to K. Then for k ∈ {1,K}, Let be the vector of the indices of the bins occupied by( and let kl(x), kc(x) and kr(x) be the corresponding vectors of left edges,centers and right edges of the bins. If the data has not already been discretized, x = kc(x) is set. For example if the red channel in an 8-bit RGB image has index 110, it will be represented by the number Note that each x(d)therefore lies in the range and not [−1, 1]. The input distribution pI(x | θ), prior parameters θ0, sender distribution pS(y | x;α), Bayesian update function h(θi−1,y, α), Bayesian update distribution pU(θi| θi−1,x;α), Bayesianflow distribution pF(θ | x; t) and accuracy schedule β(t) are all identical to the continuous case described in Section 8. It may surprise the reader that the output distribution is discretized while the input, sender and receiver distributions are not. This was chosen partly for mathematical convenience (Bayesian updates are considerably more complex for discretized distributions) and partly because it may be more efficient for the network to interpret continuous means than discrete probabilities as input. In a similar vein to the argument for standard priors in Sec. 8.1, as noted previously the input distribution only serves to inform the network and not directly to model the data; all that matters is that the input parameters contain enough information to allow the network to make accurate predictions. Section 8.11 noted that the level of measurement noise assumed for continuous data should inform the choice of standard deviation σ1for the input distribution at t = 1 (which in turn defines the accuracy schedule β(t)). For discretized data a similar role is played by the width of the discretization bins, as these place a natural limit on how precisely the data needs to be transmitted. For example, for 8-bit data with 256 bins and hence a bin width of 1 / 128, setting σ1= 1e−3 corresponds to afinal input distribution with standard deviation roughly one eighth of the width of the bin, which should be precise enough for the network to identify the correct bin with very high probability. One caveat with discretization is that calculating the loss has O(K) computational cost, which may be prohibitive for veryfinely discretized data. In any case, the benefits of discretization tend to decrease as the number of bins increases, as demonstrated by experiments below. FIG. 8 is a plot of an exemplary output distribution for discretized data. For univariate data xdiscretized into K = 16 bins, the green line shows the continuous distribution that isdiscretized to yield the output distribution pO(x | θ, t), as described in Section 9.1. Bin boundaries are marked with vertical grey lines. The heights of the green bars represent the probabilities assigned to the respective bins by pO(x | θ, t). For ease of visualisation these heights are rescaled relative to the probability density, as indicated on the right axis. Note the clipping at ±1: the area under the dotted green line to the left of −1 is added to the probability of thefirst bin, the area under the dotted green line to the right of 1 is added to the probability of the last bin. 9.1 Output Distribution pO(· | θ, t) discretized continuous distributions offer a natural and expressive way to model discretized data with neural networks [? ]. As in Section 8.7, the network outputs Ψ(θ, t) are not used to predict x directly, but rather to model the Gaussian noise vector e used to generate the mean sample μ passed as input to the network. First Ψ(θ, t) is split into two length D vectors, μ^and lnσ^. Then these are transformed to μx and σxusing For each d ∈ {1, D}, define the following univariate Gaussian cdf and clip at [−1, 1] to obtain Then, for k ∈ {1,K}, and hence 9.2 Receiver Distribution pR(· | θ; t, α) Substituting Eq. 110 and Eq. 86 into Eq. 4 gives FIGS. 9A-9C are plots of exemplary sender, output and receiver distributions for discretized data. For data x discretized into 8 bins, the three plots depict the sender distribution (red line), the discretized output distribution (bars; heights reflect the probabilities assigned to each bin, rescaled) and receiver distribution (line) for progressively increasing values of α, and for progressively more accurate predictions of x (both of which typically happen as t increases). Also shown are the continuous distribution N (x | μx, σx2) (dotted line) which is discretized to create the output distribution and the continuous receiver distribution from Section 8 (dashed line). Bin boundaries are marked with vertical lines. Note the KL divergences printed in the top right: taking discretization into account leads to a lower KL due to the density “bumps” at the bin centers where x could be. The advantage of discretization becomes more pronounced as the prediction gets closer to x and more of the probability mass is concentrated in the correct bin. 9.3 Reconstruction Loss Lr(x) The reconstruction loss for discretized data is 9.4 Discrete-time Loss Ln(x) From Eqs. 86 and 113, which cannot be calculated in closed form, but can be estimated with Monte-Carlo sampling. Substituting into Eq. 24, 9.5 Continuous-time Loss L∞(x) Justifying the claim made in Eq. 29 follows almost the same reasoning here as in Section 8.12, with C = 1 and g the identity function. The only difference is that which clearly hasfinite variance and mean. Since the claim holds and the continuous time loss from Eq 41 can be applied with and α(t) as defined in Eq 74, yielding Note that kˆ(θ, t) is a function of the complete discretized distribution pO(x | θ, t), hence L∞(x) depends on both μxand σx, and not only on μx, as for continuous data. This also means that calculating L∞(x) has O(K) computational cost for discretized data. 9.6 Pseudocode Pseudocode for evaluating the discrete-time loss Ln(x) and continuous-time loss L∞(x) for discretized data is presented in Algorithms 4 and 5, while sample generation is presented in Algorithm 6.

[0005]

[0006] 10 Discrete Data The following is directed to discrete data in which no meaningful order or distance exists between the classes, unlike the discretized continuous data covered in the previous section. Some obvious examples are text characters, classification labels or any binary data. In this context the data ()is represented as a D dimensional vector of class indices: where {1,K} is the set of integers from 1 to K. 10.1 Input Distribution pI(· | θ) For discrete data, the input distribution is a factorized categorical over the class indices. Let () ( )is the probability assigned to class k for variable d. Then The input prior is uniform with where is the length KD vector whose entries are all A uniform prior was chosen rather than an empirical priorfit to the training data as a standard normal prior was similarly chosen for continuous data: it is mathematically simpler, and the disparity between the true prior and the simple prior can easily be corrected by the network. 10.2 Output Distribution pO(· | θ; t) ( ) Given data x, network inputs θ, t and corresponding network outputs ∈ RKD, the output distribution for discrete data is as follows: Note that for binary data only the probability that k = 1 is fed into the network, on the grounds that the probability of k = 2 can easily be inferred from The output distribution for binary data is determined by applying the logistic sigmoid function elementwise to the length D output vector to get the probability for k = 1: where then inferring the probabilities for k = 2 from In principle one class could also be removed from the inputs and outputs when K > 2 and inferred from the others. However this would require the network to internalise a slightly more sophisticated inference procedure that could potentially slow down learning. A deep-learning convention was therfore followed and a redundant input and output unit was included for K > 2. All probabilities are rescaled to the range [−1, 1] by multiplying by two then subtracting one before feeding them into the network. 10.3 Sender Distribution pS (· | x;α)Given ω ∈ [0, 1], and a vector of D class indices( ) let where is the Kronecker delta function. Clearly so the vector ( ) defines a valid distribution over K classes. To simplify notation hereafter superscripts are dropped and x(d)is referred to as x, p(k(d)| x(d);ω) as p(k | x;ω) and so on, except where necessary to remove ambiguity. Consider a vector of integer counts c = (c1, ... , cK) ∈ {1,m}K, corresponding to the number of times each of the K classes is observed among m independent draws from a(x, ω). Then the probability of observing c is given by the following multinomial distribution: Now consider the fraction ck / m of observations of class k in c. Clearly meaning that for anyfinite ω it would be possible to deduce from c what the value of x is if m is sufficiently large. However as ω shrinks, p(k | x;ω) becomes closer to uniform, meaning that a larger m is required to unambigously identify x from c. By defining the accuracy and sending m → ∞ (and hence ω → 0 for anyfinite α), p(c | x, ω) can therefore be used to define a continuous-valued sender distribution that smoothly varies from totally uninformative at α = 0 to totally informative as α → ∞, like the sender distribution for continuous data. It can be proved from the central limit theorem that for any set of discrete probabilities p = {p1, ... , pK}, where 0 < pk< 1 ∀k, that if c ~ Multi(m, p) then in the limit m → ∞ the following result holds [? ]: where I is the K ×K identity matrix. Therefore Now define And the length K sender sample y = (y1, ... , yK) as Note that y, unlike x, is continuous (Y = RK,X = {1,K}), and that( )measures the number of times each class is observed, minus the average number of observations per class. Intuitively, y provides information about the relative concentration of the classes among the counts, with (since ln ξ > 0) positive values for classes observed more frequently than the mean and negative values for those observed less frequently than the mean. As mω2grows the concentration increases around the true class, and hence y become more informative about x. Rearranging Eq. 141, which may be used for the following change of variables: ∣∣ leveraging the fact that ξ ≥ 1 and hence Recall that α = mω2and hence which can be substituted into the above to yield Substituting from Eq. 131, ( ) and hence Applying the identity it can be seen that and hence Furthermore, it follows directly from Eq. 131 that Now define Plugging Eq. 150 and 151 into Eq. 148, Restoring the superscript, where 1 is a vector of ones, I is the identity matrix and ej∈ RKis the projection from the class index j to the length K one-hot vector defined by (ej)k= δjk, and therefore where 10.4 Receiver Distribution pR(· | θ; t, α) Substituting Eq. 127 and Eq. 157 into Eq. 4 gives the following receiver distribution for dimension d: 10.5 Bayesian Update Function h(θi−1,y, α) Recall from Section 10.1 that is the probability assigned to Dropping the superscript and returning to the count distribution p(c | x, ω) defined in Eq. 133, the posterior probability that x = k after observing c is Substituting Eq. 135 into Eq. 160 and cancelling terms in the enumerator and denominator, Now define Substituting the definition of ykfrom Eq. 141 into the definition of h(θ, y) from Eq. 165, and hence, from Eq. 164, Therefore in the limit m → ∞ with mω2= α, the stochastic parameter update from θi−1to θiinduced by drawing c from multi(m, a(x, ω)) can be sampled byfirst drawing y from pS(· | x, α) then setting θi= h(θi−1, y). Hence the Bayesian update function is where the redundant parameter α has been included for consistency with the update function for continuous data. 10.6 Bayesian Update Distribution pU(· | θi−1,x;α) Substituting Eqs. 157 and 171 into Eq. 6, ( ) 10.7 Additive Accuracies It follows from the definition of the update distribution that if yais drawn from pS(· | x;αa) then From Eqn. 156 and hence, from Identity 61 Therefore, if y is drawn from pS(· | x;αa+ αb) and θi= h(y, θi−2) then θiis drawn from EpU(θi−1|θi−2,x;αa)pU(θi| θi−1, x;αb) and as required. 10.8 Accuracy Schedule β(t) As with continuous data, the guiding heuristic for β(t) was to decrease the expected entropy of the input distribution linearly with t. In the continuous case, where the entropy is a deterministic function of σ2, applying the heuristic was straightforward; in the discrete case an explicit computation of EpF(θ|x;t)H [pI(x | θ)] would be needed. An analytic expression for this term, but is a reasonable approximation, with β(1) determined empirically for each experiment. Therefore FIG. 10 is a plot illustrating an example of an accuracy schedule vs. expected entropy for discrete data. The surface plot shows the expectation over the parameter distribution p(θ | x;β) of the entropy of the categorical input distribution p(x | θ) for K = 2 to 30 and β = 0.01 to 4. The red and cyan lines highlight the entropy curves for 2 and 27 classes, the two values that occur in the experiments. The red and cyan stars show the corresponding values chosen for β(1). 10.9 Bayesian Flow Distribution pF(· | x; t) Substituting Eq. 172 into Eq. 10, Since the prior is uniform with this reduces to which can be sampled by drawing y from N (β(t) (Kex− 1) , β(t)KI) then setting θ = softmax(y). The sender distribution for discrete data can therefore be interpreted as a source of softmax logits for the Bayesianflow distribution; the higher the sender accuracy α is, the larger in expectation the logits corresponding to x will be in y, hence the closer θ will be to exand the more information the network will gain about x. 10.10 Reconstruction Loss Lr(x) The reconstruction loss for discrete data is 10.11 Discrete-time Loss Ln(x) From Eqs. 156 and 158, Therefore, substituting into Eq. 24, where, from Eq. 182, 10.12 Continuous-time Loss L∞(x) Let and apply Identity 51 to see that if ( ) then and similarly if then The Kullback-Leibler divergence is invariant under affine transformations of variables, hence Now set which hasfinite variance and the followingfinite expectation where The conditions in Eq. 29 are therefore satisfied and Eqs. 203 and 183 can be substituted into Eq. 41 to yield where 10.13 Pseudocode Pseudocode for evaluating the discrete-time loss Ln(x) and continuous-time loss L∞(x) for discrete data is presented in Algorithms 7 and 8, while sample generation is presented in Algorithm 9. Table 1: Comparison of dynamically binarized MNIST and CIFAR-10 results with other methods. The best published results for both datasets (*) use data augmentation for regularization. Results for models marked with (†) are exact values; all other results are upper bounds. 11 Experiments Bayesian Flow Networks (BFNs) were evaluated on the following generative benchmarks: CIFAR-10 (32×32 8-bit color images), dynamically binarized MNIST (28×28 binarized images of handwritten digits) and text8 (length 256 character sequences with a size 27 alphabet). The continuous (Sec. 8) and discretized (Sec. 9) versions of the system were compared on CIFAR-10, while the discrete version (Sec. 10) was applied to the other datasets. In all cases, the network was trained using the continuous-time loss L∞(x), with the discrete-time loss Ln(x) evaluated for testing only, with various values of n. Standard network architectures and training algorithms were used throughout to allow for direct comparison with existing methods. Because the focus of this paper is on probabilistic modeling rather than image generation, FID scores were not calculated. However, examples of generated data are provided for all experiments. Table 2: Dynamically binarized MNIST results. NPI is nats per image averaged over 2,000 passes through the test set with Ln(x) or L∞(x) sampled once per test image per pass. The reconstruction loss Lr(x) (included in NPI) was 0.46. 784 is the total number of pixels per image, hence the number of steps required to generate an image with an autoregressive model. 11.1 Dynamically Binarized MNIST Data. The binarized MNIST benchmark data was originally created from the MNIST dataset of handwritten images by treating the grayscale pixel intensities as Bernoulli probabilities and sampling a particular binarization which is heldfixed during training. In recent years, a variant of the same benchmark has become more popular, with a new binarization sampled from the probabilities for every training batch. The two are not comparable, as the latter, refered to here as dynamically binarized MNIST, effectively has a larger training set and hence gives better test set performance. The experiments and the results referenced here use dynamically binarized MNIST. FIG. 11A shows an example of MNIST real data. FIG. 11B shows an example of MNIST generated data. Setup. The network architecture was based on a U-Net introduced for diffusion models. Starting from the hyperparameters used for the CIFAR-10 dataset, the following modifications were made: the number of resblocks was reduced from three to two and the layer widths were reduced from [C, 2C, 2C, 2C] to [C, 2C, 2C] with C = 128. Finally, the input and output of the standard network were concatenated and projected back to the output size. 600 randomly selected training images (1% of the training set) were used as a validation set. The optimizer was AdamW with learning rate 0.0001, weight decay 0.01 and (β1, β2) = (0.9, 0.98). Dropout was used with probability 0.5, the training batch size was 512, and β(1) was set to 3 (see Sec. 10.8). The network was trained for 150000 weight updates until early stopping. An exponential moving average of model parameters with a decay rate of 0.9999 was used for evaluation and sample generation. The total number of learnable parameters was approximately 25M. FIG. 12A shows an example of a MNIST input distribution. FIG. 12B shows an example of MNIST output distribution. For two test set images FIGS. 12A-B show the white pixel probability at 20 steps evenly spaced between t = 0 and t = 1 / 3. Note how the input probabilities are initially uniform whereas the output distribution initially predicts a superposition of multiple digits, closely matching the per-pixel marginal prior over the training set: this supports the conclusion that the network learns to correct for the uniform prior in the input distribution. Also note that the output distribution is much less noisy than the input distribution, and that it changes more dramatically as new information is received (e.g., the network appears to switch from predicting a 6 to a 2 to a 7 for thefirst image). This highlights the network’s use of context to resolve ambiguity and noise in the input distribution. Results. As can be seen from Table 1, BFN is close to state-of-the-art for this task with no data augmentation. Table 2 shows the expected inverse relationship between loss and number of steps. Direct optimisation of the n-step loss may lead to reduced loss for low values of n. One issue is that the reconstruction loss was relatively high at 0.46 nats per image. A seemingly apparent way to decrease this would be to increase β(1), but in practice doing so led to slower learning and worse performance. Along with the loss curves in Figure ??, this suggests that the accuracy schedule may be suboptimal for binary data. FIG. 13A shows the mean over the test set of the cts. time loss L∞(x) used for training for transmission time t between 0 and 1. FIG. 13B shows the average cumulative value of L∞(x) up to t, along with the reconstruction loss Lr(x) evaluated at t and the sum of these two losses, which would be the total loss if the transmission process halted at t. 11.2 CIFAR-10 Data. Two sets of generative modeling experiments were conducted on the CIFAR-10 database, one at the standard bit-depth of 8, corresponding to 256 discretized bins per colour channel, and one at a reduced bit-depth of 4, corresponding to 16 bins per channel. In both cases the bins evenly partitioned the interval [−1, 1] and the data was pre-processed by assigning each channel Table 3: CIFAR-10 results. All losses are bits per dimension (BPD) averaged over 100 passes through the test set with Ln(x) or L∞(x) sampled once per test image per pass. The reconstruction losses Lr(x) (included in BPD) and the number of training updates for each network are shown below. intensity to the nearest bin centre, as described in Section 9. The purpose of comparing 16 and 256 bin discretization was twofold: (1) to test the hypothesis that the advantage of training with the discretized loss from Section 9 rather than the continuous loss from Section 8 would be greater when the number of bins was lower, and (2) to test whether modeling the data at lower precision would lead to improved perceptual quality. No data augmentation, such as horizontalflips or random crops, was used on the training set. FIGS. 14A-14B show examples of CIFAR-10 real and generated data (256 bins). FIGS. 14C-14D show examples of CIFAR-10 real and generated data (16 bins). Setup. The network architecture was essentially the same as that used for Variational Diffu- sion Models (VDMs), including the Fourier feature inputs. The only modification was an extra input-output connection similar to the network for MNIST. In total there were approximately 31M learnable parameters. The following hyperparameters were used for all CIFAR-10 experiments: a validation set of 500 randomly selected training images (1% of the training set), the AdamW optmizer with weight decay 0.01, learning rate 0.0002 and (β1, β2) = (0.9, 0.99), dropout with probability 0.1, training batch size of 128, tmin= 1e−6, [xmin, xmax] = [−1, 1], and an exponential moving average of model parameters with a decay rate of 0.9999 for evaluation and sample generation. For the 256 bin experiments σ1= 0.001, while for the 16 bin experiments For the networks trained with continuous loss, the reconstruction loss was measured using the discretized version of Lr(x) from Section 9.3 rather than the continuous version from Section 8.10, using a discretized Gaussian with mean equal to and std. deviation chosen empirically to be σ1for 256 bins and 0.7σ1for 16 bins. This ensured the results were comparable between continuous and discretized training, and consistent with the literature. For two test set images FIGS. 15A-B show the means of the input and output distributions at steps evenly spaced between t = 0 and t = 0.25. Results. Table 1 shows that the best performing BFN gives 2.66 BPD for the 256 bin data, which is close to the state-of-the-art at 2.64 BPD. The most obvious performance benchmark (given the shared network architecture and similarity in loss function) is the VDM result at 2.65 BPD. However this took 10M weight updates to achieve, and due to time constraints BFNs were only trained for 5M updates. Validation performance was still improving after 5M updates, and it is unclear how much performance might improve with 10M updates. Table 3 shows that discretized loss gave better performance than continuous loss for 16 bins, as well as much faster training time (250K updates vs. 1M). This supports the hypothesis that training with discretized loss is most beneficial when the number of bins is relatively low. Furthermore, for both 16 and 256 bins, discretized training gave much better results when the number of steps n was low (e.g., 10 or 25). However continuous loss gave better performance than discretized loss on 256 bins (2.66 BPC vs 2.68); more investigation would be needed to understand why. FIGS. 17A-17C show that discretized training with 16 bins gives better sample quality than training with 256 bins. This is presumably because the loss function of the former is restricted to thefirst four bits of the data in which — as can be seen by comparing the test data at 16 and 256 bins — most of the perceptually relevant information is contained. FIGS. 16A-16B are plots of CIFAR-10 losses against time. The plot was made using the network trained with discretized loss on 256 bins. Note the high loss at the very start of the process, which were observed with discrete data. Table 4: Comparison of text8 results with other methods. The best published model on this dataset (*) was trained on sequences of length 512. Rest of the above models were trained on sequences of length 256. Results for models marked with (†) are exact values; all other results are upper bounds. Table 5: text8 results. BPC is bits per character averaged over 1M randomly cropped sequences from the test set with Ln(x) or L∞(x) sampled once per crop. The reconstruction loss Lr(x) (included in BPC) was 0.006. 11.3 text8 Data. The text8 dataset was derived from a subset of the enwik9 Wikipedia dataset by removing punctuation and restricting the text to lowercase Latin letters and spaces, giving an alphabet of size 27. FIG. 20A shows text8 test data. FIG. 20B shows text8 generated data. For clarity, the space character is represented with an underscore infigures. FIG. 17A shows an example of text8 real data. FIG. 17B shows an example of text8 generated data. Setup. The network architecture was a Transformer similar to the small model (dmodel= 768) using the GELU activation function with the depth increased to 24 layers. The input and output of the Transformer were concatenated and then projected back to the output size to produce thefinal output. The standard training / validation / test split of 90M / 5M / 5M consecutive characters was used, and the network was trained with a batch size of 3328 sequences of length 256, randomly cropped from the training set, for 1.2M weight updates using the AdamW optimizer. The learning rate was set to 10−4, weight decay to 0.1 and (β1, β2) to (0.9, 0.98). An exponential moving average of model parameters with a decay rate of 0.9999 was used for evaluation and sample generation. Dropout was not used, but overfitting was observed towards the end of training indicating that regularization may further improve results. was 0.75. The total number of learnable parameters was approximately 170M. Note that the batch size and number of layers were larger than prior results from diffusion models. Thefirst choice increases model capacity while the second tends to make overfitting more likely. These choices were made to maximize the utilization of available resources while achieving results in reasonable time. Results. Table 4 shows that BFN yielded a 1.41 BPC on the text8 test set, which is better than discrete diffusion models, and close to the best order-agnostic model, MAC at 1.40 BPC. It is noted, however, that both a standard autoregressive baseline and a discreteflow model perform substantially better at 1.23 BPC. Table 5 shows that performance is reasonably robust to decreased n, with only 100 steps required to reach 1.43 BPC. This result may be improved by training with the discrete-time loss. 12 System The present system for executing the functionality described in detail above may be a computer, an example of which is shown in the schematic diagram of FIG. 18. The system 500 contains a processor 502, a storage device 504, a memory 506 having software 508 stored therein that defines the above mentioned functionality, input, and output (I / O) devices 510 (or peripherals), and a local bus, or local interface 512 allowing for communication within the system 500. The local interface 512 can be, for example but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface 512 may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the local interface 512 may include address, control, and / or data connections to enable appropriate communications among the aforementioned components. The processor 502 is a hardware device for executing software, particularly that stored in the memory 506. The processor 502 can be any custom made or commercially available single core or multi-core processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the present system 500, a semiconductor based microprocessor (in the form of a microchip or chip set), a microprocessor, or generally any device for executing software instructions. While FIG. 18 shows the processor as a single unit, alternatively the processor may include two or more processing units distributed across two or more locations, for example, communicating via a communication network in addition to or in place of the local interface 512. The memory 506 can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), volatile memory elements (e.g., a hard drive, a solid state drive (SSD), aflash drive, an optical drive, tape) and nonvolatile memory elements (e.g., ROM, CDROM, etc.). Moreover, the memory 506 may incorporate electronic, magnetic, optical, holographic, and / or other types of storage media. Note that the memory 506 can have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor 502. The software 508 defines functionality performed by the system 500, in accordance with the present invention. The software 508 in the memory 506 may include one or more separate programs, each of which contains an ordered listing of executable instructions for implementing logical functions of the system 500, as described below. The memory 506 may contain an operating system (O / S) 520. The operating system essentially controls the execution of programs within the system 500 and provides scheduling, input-output control,file and data management, memory management, and communication control and related services. The I / O devices 510 may include input devices, for example but not limited to, a keyboard, mouse / trackpad, haptic sensor, touchscreen, scanner, microphone, barcode reader, QR code reader, etc. Furthermore, the I / O devices 510 may also include output devices, for example but not limited to, a printer, display (2D, 3D, virtual reality headset), transducer, etc. Finally, the I / O devices 510 may further include devices that communicate bidirectionally via both inputs and outputs or a combined interface such as a full duplex serial bus (for example, a universal serial bus (USB)), for instance but not limited to, an interface for accessing another device, system, or network), a wireless transceiver, a copper, optical or wireless telephonic interface, a bridge, a router, or other device. The outputs may include an interface to control a manufacturing device, such as a 3D printer, a computerized numerical control (CNC) machine, and / or a milling machine, among others. When the system 500 is in operation, the processor 502 is configured to execute the software 508 stored within the memory 506, to communicate data to and from the memory 506, and to generally control operations of the system 500 pursuant to the software 508, as explained above. While the exemplary embodiments relate to loss functions for generative modeling with neural networks, the exemplary embodiments take a different approach from previous generative learning techniques such as diffusion. For example, while diffusion can be applied to both continuous and discrete data, the embodiments take a different approach based upon Bayesian probabilities. The embodiments provide many of the advantages of diffusion models with some differences, importantly, the ability to achieve results similar to those produced by discrete diffusion but instead with continuous inputs. For example, previous techniques (large language models like ChatGTP) are autoregressive, generating one word piece at a time, which is problematically slow, requiring the network to be re-run with every new generated step. Diffusion models can progressively de-noise several pieces of data at once, so diffusion can generate multiple characters per step, rather than one character per step. However, diffusion models treat discretized data as continuous data for purposes of the loss function. In contrast, the exemplary embodiments may be directly adapted to receive discretized data. In particular, the loss function may be adapted to discretized data. The advantages of the discretization over continuous data becomes more apparent as the granularity of the discretized data bins becomes more coarse. For a loss function according to the embodiments configured for discretized data, as the level of granularity becomesfiner, the expense of running the loss function becomes more expensive, to the point where it may be more efficient to use a loss function configured for continuous data. At the other extreme, in instances where the number of discretization bins is too small, the results may not differ appreciatively from employing a loss function configured for discrete data. The experiment describe above compares the relative performance of continuous and discretized BFN at 16 bins and 256 bins on CIFAR-10. At 16 bins the discretized model performs better, at 256 bins the continuous model performs better. Mathematically, as the number of bins approaches infinity, the discretized loss appears to approach the continuous loss. Intuitively, as the number of discretization becomes smaller, it becomes less appropriate for the continuous loss to treat the data as continuous, as here the continuous loss does not account for the distance of a given data point from the center of its respective bin. This difference matters more when there are fewer / larger bins because the gap between the mid-points of neighboring bins becomes larger. While it is possible to have the continuous model learn the locations of bin centers, even here the continuous model will not be entirely accurate and would involve an unnecessary modeling effort on information known a priori. There are difficulties applying diffusion methods for discrete data. While diffusion applies a distribution directly over the provided data (so noisy data becomes clean data), the embodiments instead act upon parameters of a distribution over the data, which may be thought of a distribution over distributions. It is important to distinguish the terminology “Bayesian Flow Model” of the present exemplary embodiments with aflow based model” as applied in other distribution models. Specifically, the previousflow based models have not applied Bayesian statistics, as per the Bayesianflow models of the present exemplary embodiments. Herein, “Bayesianflow model” is used to distinguish from previousflow based models. In a Bayesianflow, Bayesian updates are applied to noisy samples data by applying Bayes’ rule to the input distribution. When the number of updates approaches a limit where the system behaves as if the number of updated becomes infinite, behaving as a continuous distribution, the distribution becomes a Bayesianflow. Therefore, the exemplary embodiments disclose a framework that is readily adaptable across discrete, discretized, and continuous data, contrasting from previous frameworks that must be specific to one of discrete data, discretized data, or continuous data. Of existing methods, Bayesian Flow Networks are most closely related to diffusion models. However the two differ in some crucial aspects. First, BFNs embody a function from one distribution to another, rather than from data to a distribution, like diffusion models and most other probabilistic networks. One advantage of this approach is that because the parameters of a categorical distribution are real-valued probabilities, the inputs to the network are continuous even when the data is discrete. This contrasts with discrete diffusion, which natively uses discrete samples as input. While continuous variants of discrete diffusion have been proposed previously, typically these approaches rely either on mapping to and from a continuous embedding space, or on restricting continuous diffusion to the probability simplex. While the exemplary embodiments do not directly compare against the above methods, it should be noted that continuity is an inherent property of the Bayesian Flow framework (the network inputs automatically lie on the probability simplex by virtue of being the parameters of a categorical distribution), rather than a constraint added to an existing system. As well as reducing the number of free parameters and design choices (e.g. the continuous embedding space, the mapping functions), this ensures that BFNs directly optimize the negative log-likelihood of discrete data, unlike continuous diffusion methods for discrete data, which typically require either simplified loss functions or auxiliary loss terms to make learning stable. For continuous data, BFNs are most closely related to variational diffusion models, with a similar continuous-time loss function. Significantly, the network inputs are considerably less noisy in BFNs than in variational diffusion and other continuous diffusion models because the generative process of BFNs begins with the parameters of afixed prior, whereas that of diffusion models begins with pure noise. Here the reduction in noise may lead to faster learning on large datasets where the model underfits. Another key difference from diffusion models is that there is no need to define and invert a forward process for BFNs, which arguably makes it easier to adapt BFNs to different distributions and data types. The embodiments showcase thisflexibility by adapting BFNs to continuous, discretized, and discrete data, with minimal changes to the training procedure. This contrasts with e.g. discretized diffusion, which requires carefully defined transition matrices. It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A method (200) for implementing a Bayesian flow network generative model, comprising steps of: receiving a batch of data samples from a data set (205); randomly sampling a total accuracy level for each data sample of the batch of data samples; constructing a Bayesian flow distribution (210) corresponding to the total accuracy levels; sampling from the Bayesian flow distribution to produce training input distribution parameters; providing the training input distribution parameters and a first accuracy level (215) as inputs to a neural network (220); receiving training output distribution parameters (225) of a training output distribution from the neural network; and determining a discrete or continuous-time loss from the training output distribution parameters.

2. The method of claim 1 , further comprising a step of selecting a type of the neural network corresponding to the data set.

3. The method of claim 1, further comprising a backward pass comprising steps of:computing a gradient of the loss (235) with respect to network parameters of the neural network (220); and updating the network parameters using the computed gradient and applying a stochastic gradient descent algorithm.

4. The method of claim 3, wherein computing the gradient of the loss further comprises a step of applying automatic differentiation techniques.

5. The method of claim 1, further comprising steps of: providing first input distribution parameters for a first input distribution (215) and the first accuracy level to the neural network (220); receiving output distribution parameters (225) of an output distribution from the neural network; sampling an output prediction (325) from the output distribution (320); selecting a sender sample (335) from a sender distribution (330); updating the input distribution parameters (345) by a Bayesian update function (340); and selecting a second accuracy level (355).

6. The method of claim 5, wherein the first accuracy level and the second accuracy level are selected according to an accuracy schedule.

7. The method of claim 5, wherein selecting a sample output prediction (325) from the output distribution further comprises a random selection.

8. The method of claim 5, wherein the neural network is selected according to a data type of the sender distribution.

9. The method of claim 8, wherein the data type is selected from the group of continuous data, discretized data, and discrete data.

10. The method of claim 5, wherein the input distribution comprises a factorized distribution.

11. The method of claim 10, wherein the parameters of the factorized input distribution and parameters of the sender distribution comprise probabilities of a categorical distribution.

12. The method of claim 5, wherein the neural network emits an output vector used to parameterize the output distribution factorized the same way as the input and sender distributions.

13. The method of claim 5, further comprising a step of providing updated distribution parameters as inputs to the neural network (220).

14. The method of claim 1, wherein the Bayesian flow distribution is a marginal distribution over input parameters at a time t given the prior distribution, the accuracy schedule, and a Bayesian update distribution.

15. The method of 5, wherein the Bayesian update function is configured to apply rules of Bayesian inference to the input distribution parameters and the sender sample.

Images