Automated variational inference using probabilistic models with irregular beliefs

Abstract

A system and method for automatic construction of probabilistic deep neural network (DNN) architectures are provided. The framework of the present invention automatically explores the most relevant probabilistic modes of the basis of a dataset for variational Bayesian inference. The present invention provides a method for using heterogeneous, irregular, and mismatched beliefs in probabilistic sampling for intermediate representations in DNNs, with the ability of automatic tuning mechanisms of posterior, prior, and likelihood models to enable accurate generative and uncertainty models for machine learning tasks. The system further enables an adjustable discrepancy measure to regularize the intermediate representations by variants of divergence metrics, including Renyi's alpha, beta, and gamma divergence. The present invention enables diverse mixed combinations of probabilistic models for misspecified and unspecified probabilistic relationships in an automatic manner. Thus, the representation capabilities of variational autoencoders, variational information bottlenecks, denoising diffusion probability models, and other probabilistic DNNs are improved.

Description

[Technical field]

[0001] The present invention relates to an automated learning system for artificial neural networks, and more particularly to an automated Bayesian inference system based on probabilistic neural networks with irregular beliefs. [Background technology]

[0002] Significant advances in deep learning techniques based on deep neural networks (DNNs) have solved various problems in data processing, including media signal processing for video, audio, and images; physical data processing for radio waves, electrical pulses, and light beams; and physiological data processing for heart rate, body temperature, and blood pressure. However, DNN architectures are often created manually with insight from experts who know the unique data models and structures. Methods for optimizing the architecture of DNNs require time / resource consuming trial-and-error techniques. An automatic machine learning (AutoML) framework has been used to automatically explore different DNN architectures to solve this problem. Automation of hyperparameter and architecture search in the context of AutoML can facilitate DNN design suitable for specific data processing. AutoML includes architecture search, learning rule design, and augmentation search. Most AutoML methods use either evolutionary optimization, hypergradient, or reinforcement learning frameworks to tune hyperparameters or build network architectures from a pre-chosen selection of building blocks. A recent AutoML-Zero considers an augmentation to eliminate expert knowledge and insight for a completely automated design from a clean slate.

[0003] Learning data representations that capture task-relevant features but are invariant to nuisance variations remains a key challenge in machine learning. A probabilistic DNN called variational autoencoder (VAE) introduces a variational Bayesian inference method and incorporates an autocorrelation architecture, where generative and inferential models are jointly trained by using a pair of decoder and encoder architectures. This method has been extended with a conditional VAE that introduces conditioning variables that can be used to represent nuisance, and a regularized VAE that considers separating nuisance variables from the latent representation. The concept of adversarial learning has been considered in generative adversarial networks (GANs) and adopted for countless applications. Concurrently discovered, adversarially learned inference (ALI) and bidirectional GANs (BiGANs) proposed adversarial approaches toward training autoencoders. Adversarial training is also combined with VAEs to regularize and separate latent representations so that nuisance-robot learning is realized.

[0004] For Bayesian inference, VAEs are composed along with parametric encoders and decoders to learn the latent variables underlying the data within a variational inference (VI) framework. Many variants of such probabilistic DNNs exist. For example, β-VAE uses emphasized Kullback-Leiber divergence (KLD) to regularize the latent distribution more strongly than typical evidence lower bound (ELBO)-based losses. Continuous Bernoulli and beta distributions are studied as alternative likelihood beliefs for data reconstruction models. Laplace and Cauchy distributions are considered as alternative prior beliefs for sparse latent representations. The normal posterior beliefs are adjusted by inverse autoregressive flows (IAFs), importance weighted autoencoders (IWAEs), and Gibbs sampling. IWAE is further extended to variational Renyi (VR), which is based on Renyi's α-divergence. Generalized VI (GVI) then discusses arbitrary losses, divergences, and posterior selection. And it shows greater robustness with different discrepancy measures, instead of KLD, when data statistics are not specified.

[0005] While generalized VAEs offer great freedom, it then makes their selection difficult to engineer in addition to other architecture hyperparameters. There is no framework for designing variational Bayesian inference and probabilistic DNNs when an intrinsic model is not specified to capture data statistics and probabilistic uncertainty. For example, probabilistic DNNs, including VAEs, variational information bottlenecks (VIBs), and denoising diffusion probability models (DDPMs), typically use homogeneous statistics for latent representations based on normal distributions. Normal distributions are computationally convenient because simple sampling, reparameterization tricks, and closed-form expressions of divergence are possible for gradient computations. However, when the underlying statistics of real-world datasets are not known a priori, normal distributions are generally no longer optimal.

[0006] However, AutoML has the drawback that it requires a lot of search time to find the best hyperparameters due to search space explosion. In addition, without any good inference, the search space of most link connectivity becomes meaningless. In order to develop a system for automatic construction of artificial neural networks with correctness, a method called AutoBayes was proposed. The AutoBayes method represents the inherent graphical relationships between random variables for the generative model, and then explores different Bayesian graphs to build the most plausible inference graph to connect the encoder, decoder, classifier, regressor, adversary, and estimator. The use of the so-called Bayes-ball algorithm allows the automatic construction of the most compact inference graph for a given Bayesian graph, and some factors are identified as variable independent of the domain factors to be censored by the adversarial block. Adversarial censorship, which separates nuisance factors from the feature space, was verified to be effective for domain generalization in pre-shot transfer learning and domain adaptation in post-shot transfer learning. However, adversarial training requires careful selection of hyperparameters, as too strong censorship will impair primary task performance as the main objective function is relatively underweighted. Moreover, adversarial censorship is not the only regularization technique for promoting independence from nuisance variables in the feature space.

[0007] Therefore, there is a need to efficiently identify the best probabilistic model for a probabilistic DNN that depends on the particular problem and dataset. Summary of the Invention

[0008] The present invention is based on the recognition that probabilistic deep neural networks (DNNs) use a specific probability model, and that although there are many different probability models to impose, including normal distribution, Laplace distribution, Cauchy distribution, logistic distribution, Gumbel distribution, student-t distribution, uniform distribution, exponential distribution, and hyperexponential distribution, the true probability model underlying real-world datasets is generally unknown. In addition, there are unlimited possibilities for expressing latent variables and data probabilistics, as well as relevance measures such as divergence metrics in generalized variational inference (GVI). For example, there is no formal framework for determining the discrepancy measure to be selected from infinite possibilities, including Kullback-Leiber divergence (KLD), Renyi's alpha divergence, beta divergence, gamma divergence, Jensen-Shannon divergence, Jeffery divergence, and Fisher divergence, to regularize probabilistic nodes in DNNs. Furthermore, data uncertainty for identifying likelihood beliefs is also generally not specified. For example, the normal distribution is typically used to minimize the weighted mean squared error (MSE) for regression problems, and the Bernoulli distribution is often used to minimize the binary cross entropy as the reconstruction loss for approximately binary pixel images.

[0009] The present invention allows probabilistic DNNs to efficiently identify data statistics and uncertainty models through the use of an AutoML framework for better combination of prior beliefs, posterior beliefs, and likelihood beliefs and discrepancy measures. This provides benefits over the automatic construction of DNN architectures such as DNN layer size, node size, activation type, and link connectivity. Specifically, the present invention provides a method for exploring irregular combinations of probabilistic models on top of traditional architecture hyperparameters. Some embodiments allow probabilistic DNNs to impose mismatched pairs of posterior beliefs and prior beliefs to increase the freedom of modeling capabilities under unspecified model uncertainty. Yet another embodiment provides a method for generalizing probabilistic DNNs towards heterogeneous pairs of posterior beliefs and prior beliefs in different hidden nodes and hidden layers. In this way, more robust and complex latent representations can be achieved. In addition, the present invention provides a method for automatically exploring irregular, mismatched, heterogeneous assignments of discrepancy measures of individual probabilistic nodes. For example, the first latent node uses a logistic distribution as the posterior belief and a normal distribution as the prior belief using the KLD discrepancy measure, and the second latent node uses a Cauchy distribution as the posterior belief and a uniform distribution as the prior belief using Renyi's alpha divergence of order 0.4. Such mixed heterogeneous probabilistic models can improve generality across different datasets subject to unknown uncertainty. Some embodiments use a categorical search space across different beliefs using variants of reparameterization tricks such as the Gumbel softmax trick for latent representations in VAE, VIB, DDPM and variants thereof in the context of network architecture search. The prior beliefs, posterior beliefs, and likelihood beliefs include, but are not limited to, normal distribution, Cauchy distribution, logistic distribution, Laplace distribution, uniform distribution, triangular distribution, Gumbel distribution, exponential distribution, generalized Gaussian distribution, beta distribution, gamma distribution, Poisson distribution, Bernoulli distribution, and the like.In addition, different censorship methods, such as adversarial disentanglement, can facilitate latent representations in probabilistic DNN models to be independent of nuisance parameters, resulting in feature extraction that is robust to nuisance for some embodiments.

[0010] The present invention provides a method for tuning these hyperparameters and irregular beliefs under the AutoML framework based on hypergradient methods such as Bayesian optimization, implicit gradient, reinforcement learning, and heuristic optimization. Yet another embodiment uses mismatched posterior-prior pairing and heterogeneous distribution in probabilistic DNN. For example, the posterior beliefs of the VAE encoder use a logistic distribution, and the prior beliefs of the VAE decoder use a normal distribution of probabilistic latent variables. The present invention is based on the recognition that smaller divergence is not always better due to model mismatch, and suggests that heterogeneous posterior-prior pairing has the potential to provide a better Bayesian model for the entire probabilistic DNN, especially when the uncertainty model is not specified in advance.

[0011] Another embodiment uses multiple intermediate representations for variational sampling to improve model accuracy. In this case, the number of combinations to be selected for each latent layer grows rapidly, and therefore automatic search becomes important. In addition, the present invention provides a method for enabling irregular, non-uniform posterior-prior pairing in every latent variable for some embodiments. One embodiment realizes an ensemble method to explore a stack of protocols over cross-validation during AutoML. The present invention provides a method for automatically generating auxiliary models that directly control the parameters of the base inference model by analyzing consistent evolutionary behavior in the main DNN model. In some embodiments, the type of divergence is also automatically selected in a non-uniform and non-uniform manner. For example, Renyi's alpha divergence, beta divergence, gamma divergence, Wasserstein distance, and orders thereof are jointly explored to be robust to numerically intractable data statistics encountered in the real world.

[0012] The present disclosure relates to a system and method for automatic construction of probabilistic DNNs through the exploration of different uncertainty models and hyperparameters. Specifically, the system of the present invention introduces an automatic variational Bayesian inference framework that explores different posterior / prior / likelihood / disparity sets for a variational inference model linking classifier, encoder, decoder, and estimator blocks to optimize a nuisance invariant machine learning pipeline. In one embodiment, the framework is applied to a set of physiological datasets, where we have access to the subject and class labels during training, and provide an analysis of its capabilities for variational modeling and subject transfer learning with / without adversarial training. The framework can be effectively utilized for various dataset forms, such as media and electrical signals as well as biological signals, in semi-supervised multi-class classification, multidimensional regression, and data reconstruction tasks.

[0013] Some embodiments of the present disclosure are based on the recognition that a new concept called AutoBayes, which explores various different Bayesian graph models to facilitate the search for the best inference strategy, is suitable for an inference system robust to nuisance. Using the Bayesball algorithm, the method and system of the present invention can automatically construct plausible link connections between classifiers, encoders, decoders, nuisance estimators, and adversarial DNN blocks, as well as different belief combinations for probabilistic nodes. From the no-free-lunch theorem, using one specific model without exploring its variants may potentially suffer from poor inference results. In addition, the best model in one dataset does not always perform best on different data, which prompts the use of AutoML frameworks for adaptive model generation given a target dataset. One embodiment extends the AutoBayes framework to integrate probabilistic DNNs for the underlying unspecified / misspecified uncertainty of the dataset by exploring different irregular sets of beliefs for posterior, prior, likelihood, and discrepancy.

[0014] Another embodiment uses variational sampling for semi-supervised settings for cases where the dataset contains missing target labels. Yet another embodiment uses ensemble stacking to combine estimates of multiple different Bayesian models to improve performance. Another embodiment uses probabilistic graph neural networks to exploit the geometric information of the data, and the pruning strategy is aided by belief propagation over the Bayesian graph to verify relevance. Wasserstein distance can also be used instead of divergence. In some embodiments, the system and method can be combined with existing test-time online adaptation techniques from zero-shot and few-shot learning frameworks to achieve even better, robust performance against nuisance. The system can provide the advantage of learning nuisance invariant representations by exploring various regularization modules and uncertainty models. Yet another embodiment uses quantum and molecular devices to sample probabilistic nodes by exploiting the randomness of the devices. Some embodiments use reversible network architectures to model the encoder and decoder simultaneously without having two disjoint models. Some embodiments use probabilistic implicit layers, such as neural ordinary differential equations, convex optimization, deep equilibrium, diffusion models, and quantum mechanics, embedded in the DNN architecture. Thus, the present invention provides a method for designing a Bayesian machine learning model, where an unknown uncertain basis dataset is automatically modeled with a certain robustness by exploring various combinations of prior beliefs, posterior beliefs, and likelihood beliefs in probabilistic representation learning such as VAE, DDPM, and VIB under incomplete and misspecified knowledge of data statistics.

[0015] Further, according to some embodiments of the present invention, a system for data analysis using a probabilistic DNN block is provided. In this case, the system for signal analysis includes an interface, a probabilistic deep neural network (DNN) block, a memory bank, and a processor. The interface is configured to receive and transmit a signal, such as a dataset, which is a multidimensional signal associated with a task label. The probabilistic DNN block is used to identify the task label from the multidimensional signal through probabilistic nodes, which are individually specified by irregular beliefs and discrepancy measures for posterior distributions, prior distributions, and likelihood distributions. The memory bank is used to store the dataset, irregular beliefs, other hyperparameters, and trainable parameters to determine the probabilistic DNN block. The processor, in association with the interface and the memory bank, performs probabilistic inference to analyze the dataset by using the probabilistic DNN block. The probabilistic inference is realized by importance-weighted accumulation after variational sampling at the probabilistic nodes according to the irregular beliefs. Probabilistic DNN blocks include variants of VAE, VIB, and DDPM, which consist of a combination of transformation layers, interconnections, nonlinear activations, and regularization layers. Variational sampling uses a random number generator based on a reparameterization trick that follows variational parameters including position, scale, shape, and temperature specified by random beliefs. Discrepancy measures include combinations of Renyi's alpha divergence, beta divergence, gamma divergence, Fisher divergence, and Jeffrey divergence.

[0016] The irregular beliefs include mismatched or heterogeneous combinations of discrete univariate distributions, normal related distributions, exponential related distributions, extreme value distributions, bounded distributions, heavy tailed distributions, quantile based distributions, systematic distributions, multivariate continuous distributions, and multivariate discrete distributions. The probabilistic DNN block uses heterogeneous assignment of different beliefs for at least two disjoint sets of probabilistic nodes in the same layer. Furthermore, the probabilistic DNN block uses mismatched pairs of posterior beliefs and prior beliefs for at least one set of probabilistic nodes. At least two different types of uncertainty models for posterior beliefs, prior beliefs, and likelihood beliefs are imposed simultaneously at the disjoint probabilistic nodes. Furthermore, different discrepancy measures such as KLD and Renyi's alpha divergence are imposed simultaneously at the probabilistic DNN block to regularize the probabilistic nodes. The irregular beliefs can also be adjusted on the fly to use different probability distributions for variational sampling at the probabilistic nodes when analyzing a set of newly available datasets.

[0017] In some embodiments, the system uses the steps of exploring different values ​​for irregular beliefs and hyperparameters by using a supergradient method; constructing a probabilistic DNN by modifying the connectivity between multidimensional signals, task labels, and probabilistic nodes; calculating a loss function by forward propagating the dataset across the probabilistic DNN blocks according to probabilistic inference; modifying the loss function by regularizing the probabilistic nodes according to a discrepancy measure; backpropagating the gradient of the loss function with respect to the trainable parameters; and updating the trainable parameters using a gradient method.

[0018] By allowing irregular beliefs in the probabilistic DNN block, more complex model descriptions for unknown statistics of real-world datasets can be realized. Thus, by exploring irregular combinations of posterior, prior, and likelihood distributions in the probabilistic DNN block, the system of the present invention provides better variational inference results when the real-world dataset is not intractable for any closed-form statistical model.

[0019] Further, some embodiments of the present invention provide a computer-implemented method for variational inference using a probabilistic DNN. Specifically, the method of variational inference includes: feeding a data signal to a probabilistic DNN; propagating the signal according to layers in the probabilistic DNN by adopting variational sampling according to irregular beliefs at the probabilistic nodes; and accumulating the output of the probabilistic DNN as a probabilistic inference result. In addition, a method for automatic design of a variational inference model includes calculating a loss function based on a variational bound to regularize the probabilistic nodes according to a discrepancy measure; backpropagating the gradient of the loss function with respect to the trainable parameters; updating the trainable parameters according to a gradient method; and searching for different values ​​for the irregular beliefs according to a supergradient method. The gradient method uses a combination of stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, root mean square propagation, and its variants. Supergradient methods use combinations of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, Bayesian optimization, and variations thereof.

[0020] The accompanying drawings, which are included to provide a further understanding of the invention, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. [Brief description of the drawings]

[0021] [Figure 1A] 1 illustrates an exemplary inference model for predicting Y given data X, according to some embodiments. [Figure 1B]1 illustrates an exemplary inference model for predicting Y given data X under latent variables Z and nuisance factors S, in accordance with some embodiments. [Figure 1C] 1 illustrates an exemplary inference model for predicting Y given data X under latent variables Z and nuisance factors S, in accordance with some embodiments. [Diagram 2] 1 illustrates an exemplary schematic diagram of an automated VAE, according to some embodiments. [Diagram 3] 1 illustrates an example variation of an irregular belief, according to some embodiments. [Figure 4A] 1 illustrates example beliefs for posterior, prior, and likelihood distributions used for probabilistic DNNs, according to some embodiments. [Figure 4B] 1 illustrates an exemplary family of standard location scale distributions used for variational sampling based on the reparameterization trick, in accordance with some embodiments. [Figure 5A] 1 illustrates an example landscape of KLD discrepancies for various pairs of posterior and prior beliefs at two independent probabilistic nodes, according to some embodiments. [Figure 5B] 1 illustrates an example pair of posterior and prior beliefs quantified by a KLD discrepancy measure, according to some embodiments. [Figure 5C] 1 illustrates example KLD properties for various pairs of posterior and prior beliefs as a function of position value, according to some embodiments. [Figure 6] 1 illustrates an exemplary disparity measure based on Renyi's alpha-divergence that covers variants of generalized variational inference including importance-weighted autoencoder (IWAE) and standard VAE, in accordance with some embodiments. [Figure 7] FIG. 1 illustrates an example likelihood belief P of a probabilistic DNN for determining a generalized negative log-likelihood loss, in accordance with some embodiments. [Figure 8A] 1 illustrates exemplary properties of various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and discrepancy measures for a generalized VAE model, according to some embodiments. [Figure 8B] 1 illustrates exemplary ELBO properties for various pairs of posterior and prior beliefs as a function of divergence order for a generalized VAE model, in accordance with some embodiments. [Figure 8C] 13A-13C show example image snapshots generated by a VAE decoder generalized with variational sampling according to prior beliefs after being trained on various pairs of posterior and prior beliefs, in accordance with some embodiments. [Figure 8D] 13A-13C show example image snapshots generated by a VAE decoder generalized with variational sampling according to prior beliefs after being trained on various pairs of posterior and prior beliefs, in accordance with some embodiments. [Figure 9A] 1 illustrates an exemplary probabilistic DNN that sequentially uses multiple different beliefs and disparity measures in multiple intermediate layers, in accordance with some embodiments. [Figure 9B] FIG. 1 illustrates an exemplary probabilistic DNN that uses different beliefs and disagreement measures at different probabilistic nodes, in parallel within one hidden layer, and sequentially across multiple hidden layers, according to some embodiments. [Figure 10] 1 illustrates an exemplary schematic diagram of a system comprising a processor, a memory, and an interface according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Various embodiments of the present invention will now be described with reference to the drawings. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. It should also be noted that the drawings are intended only to facilitate the description of certain embodiments of the present invention. They are not intended as an exhaustive description of the present invention or as limitations on the scope of the present invention. Moreover, aspects described in connection with a particular embodiment of the present invention are not necessarily limited to that embodiment, and may be implemented in any other embodiment of the present invention.

[0023] The present invention provides a system and method for analyzing signals in a dataset based on a generalized variational inference (GVI) framework, using an underlying probabilistic model of the dataset as an artificial intelligence (AI) pipeline. FIG. 1A illustrates an exemplary inference model for predicting a task label Y given data X, according to some embodiments. The task label is either a categorical identification number or a non-categorical continuous value. In the case of categorical inference, the AI ​​model performs a classification task, and in the case of non-categorical inference, the AI ​​model performs a regression task. In some embodiments, the task label is a mixture of multiple categorical and continuous labels as a multi-task inference. The task label is either a scalar value or a vector of multiple values ​​in the case of supervised learning. In the case of unsupervised learning, the task label is not explicitly provided or, in some embodiments, is equivalent to the input data for reconstruction. The data in the dataset is in tensor format with at least one axis to represent multiple signals and sensor measurements, including, but not limited to, media data such as images, photographs, movies, text, characters, voices, music, audio, and speeches; physical data such as radio waves, optical signals, electrical pulses, temperature, pressure, acceleration, speed, vibration, and force; and physiological data such as heart rate, blood pressure, mass, hydration, electroencephalogram, electromyogram, electrocardiogram, mechanomyogram, electrooculogram, galvanic skin response, magnetoencephalogram, and electrocorticography.

[0024] For example, an AI model predicts emotions from a person's electroencephalogram measurements, where the data is a three-axis tensor representing a spatiotemporal spectrogram from a multi-channel sensor over a period of measurement. All available data signals with X and Y pairs are bundled together as a whole batch of data sets for training the AI ​​model, which are called training data or training data sets. In some embodiments, the task labels Y are missing for a portion of the training data set in semi-supervised learning, and the missing labels are predicted by the AI ​​model in the self-supervised case.

[0025] The AI ​​model can be realized by a deep neural network (DNN) model, whose architecture and behavior are specified by a set of hyperparameters. The set of hyperparameters includes, but is not limited to, training hyperparameters that determine scheduling policies such as learning rate, weight decay, gradient momentum, divergence order, batch size, number of epochs, early stopping, dropout rate, and regularization weight; architecture hyperparameters such as connectivity, type of layer operation, type of activation function, size of trainable parameters, selection of probabilistic nodes, number of layer depths, and set of layer widths to determine the DNN architecture; and deployment hyperparameters such as initialization and quantization policies for trainable parameters.

[0026] DNN models are typically based on multi-layer perceptrons that use a combination of cells including, but not limited to, transformation layers such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, implicit layers, attention layers, pooling layers, padding layers, initiation layers, and transformer layers; interconnections such as multi-loop connections, skip-add, skip-multiplication, skip-concatenation; non-linear activations such as rectified linear transformation, sigmoid transformation, shrinkage transformation, softmax transformation, gated linear, soft-plus, and soft-sine; and regularization layers such as dropout, swapping out, zone out, block out, drop connect, shaking, shuffling, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

[0027] For example, the residual network architecture uses skip addition from one hidden layer to another, which allows for stable learning of deeper layers. A DNN model has several trainable parameters, such as affine transformation weights and biases.

[0028] The DNN model is trained on a training dataset to minimize or maximize an objective function (also called a loss function or utility function) by gradient methods including, but not limited to, stochastic gradient descent, Adam gradient, root mean square propagation, ada gradient, ada delta, ada max, ada bound, Nesterov accelerated gradient, elastic backpropagation, and weighted Adam. In some embodiments, the training dataset is split into multiple sub-batches for local gradient updates. In some embodiments, a portion of the training dataset is held out for a validation dataset to evaluate the performance of the DNN model. The validation dataset from the training dataset is rotated for cross-validation in some embodiments. Methods for splitting the training data into sub-batches for cross-validation include, but are not limited to, random sampling; weighted random sampling; one session holdout; one subject holdout; one region holdout. Typically, the data distribution of each sub-batch is not identical due to domain shift.

[0029] Gradient-based optimization algorithms have several hyperparameters, such as learning rate and weight decay. The learning rate is an important parameter to choose and can be automatically adjusted by several scheduling policies, such as step function, exponential function, trigonometric function, and adaptive decay to plateau. Other optimization methods, such as evolutionary strategies, genetic algorithms, differential evolution, simulated annealing, particle swarm, Bayesian optimization, and Nelder-Mead, can also be used to optimize the trainable parameters. The objective function is a combination of various functions, including but not limited to L1 loss; Lp norm; mean squared error; cross entropy; connectionist time series classification loss; negative log-likelihood; Kullback-Leiber divergence (KLD); cross-covariance; structural similarity; cosine similarity; clustering loss; margin ranking loss; hinge loss; Huber loss; negative sampling; Waserstein distance; and triplet loss.

[0030] AI models without guidance for hidden nodes often suffer from local minimum trapping due to over-parameterized DNN architectures to solve task problems. To stabilize training convergence, several regularization techniques are used. For example, L1 / L2 norm is used to regularize affine transformation weights. Batch normalization and dropout techniques are also widely used to prevent overfitting. Other regularization techniques include, but are not limited to, drop connect; drop block; drop path; shake drop; spatial drop; zone out; probabilistic depth; probabilistic width; spectral normalization; and shake-shake. However, these well-known regularization techniques do not exploit the underlying data statistics. Most datasets have certain probabilistic relationships between X and Y, and potential nuisance factors S that interfere with task prediction performance. For example, physiological datasets such as EEG signals are highly dependent on the subjects' mental states and measurement conditions, which are treated as nuisance factors S. Nuisance variations include, but are not limited to, a set of subject identifications, session numbers, biological states, environmental states, sensor states, sensor locations, sensor orientations, sampling rates, time, and sensitivity. In yet another example, electromagnetic datasets such as Wi-Fi signals are susceptible to room environments, surrounding users, interference, and hardware imperfections. The present disclosure provides a method to efficiently regularize DNN blocks by taking into account nuisance factors such that the AI ​​model is insensitive to domain shifts caused by changes in those nuisance factors.

[0031] The present disclosure is based on a probabilistic DNN that imposes a probabilistic model with hidden layers and latent variables, whose nodes are called probabilistic nodes. Specifically, the probabilistic hidden layers are randomly sampled from a particular distribution family, such as normal distribution, and their variational parameters, such as location factor (or mean) and scale factor (or standard deviation), are defined by the hidden layers of the probabilistic DNN. In order to make the variational parameters differentiable, a reparameterization trick is used, in which the random variables sampled from the standard distribution are shifted and scaled so that they are distributed with the distribution of the desired latent variables. To regularize the probabilistic nodes, some embodiments modify the loss function based on a discrepancy measure, including but not limited to KLD; Renyi's alpha divergence; beta divergence; gamma divergence; Jeffery divergence; Fisher divergence; Jensen-Shannon divergence; Wassestern distance.

[0032] In the GVI framework, the target distribution is specified as the prior belief, and the sample distribution is specified as the posterior belief. In addition, the task label and data prediction are also probabilistically defined under a specific uncertainty model to determine the likelihood belief. The likelihood belief includes, but is not limited to, normal distribution; Bernoulli distribution; unspecified normal distribution; continuous Bernoulli distribution; Laplace distribution; Cauchy distribution; beta distribution; unspecified Lapalce distribution; and gamma distribution. Given a specific likelihood belief, the corresponding negative log-likelihood (NLL) is used to minimize the objective function of the probabilistic DNN. For example, a variational autoencoder (VAE) uses an encoder model and a decoder model to infer the latent variables and generative model, respectively. In a standard VAE, a reconstruction loss based on the mean squared error (MSE) is used under the assumption of the likelihood belief based on an unspecified normal distribution, whose variance is not defined. In addition, the latent variables are typically sampled from a normal distribution for both the posterior and prior beliefs. Some embodiments of the present invention allow for irregular heterogeneous pairing of posterior and prior beliefs to provide additional degrees of freedom for unknown data uncertainty modeling. Yet another embodiment uses irregular heterogeneous discrepancy measures at individual probabilistic nodes in the DNN model to be robust to misspecified uncertainty. Besides VAE, probabilistic DNN models in some embodiments include, but are not limited to, variational information bottleneck (VIB), denoising diffusion probability model (DDPM), and variational Bayesian neural network.

[0033] The present invention uses random beliefs about the posterior, prior, and likelihood distributions, as well as their discrepancy measures, to determine statistics for probabilistic nodes in a probabilistic DNN block.Irregular beliefs include discrete univariate distributions such as the Bernoulli distribution, binomial distribution, geometric distribution, Pascal distribution, hypergeometric distribution, Waring Yule distribution, Poisson distribution, Skellam distribution, discrete uniform distribution, Zipf distribution, log series distribution, Benford distribution, Polya distribution, Tanner distribution, and Poisson Consul distribution; normally related distributions such as the normal distribution, lognormal distribution, Johnson distribution, half normal distribution, skew normal distribution, Voigt distribution, Tsallis distribution, hyperbolic distribution, variance gamma distribution, Student-t distribution, chi distribution, Rayleigh distribution, Maxwell distribution, chi-squared distribution, F-ratio distribution, Fisher distribution, Hotelling distribution, and Tracy distribution; exponential distribution, Laplace distribution, Lindley distribution, logistic distribution, sech distribution, exponential power distribution, shifted Gompertz distribution, Erlang distribution, hypoexponential distribution, Coxian distribution, gamma distribution, Nakagami distribution, Moyal exponentially related distributions such as the Meixner distribution; extreme value distributions such as the max stable, min stable, Gumbel, Frechet, and Weibull distributions; bounded distributions such as the uniform, arcsine, Bates, Kumaraswamy, power, triangular, uniform sum, Mises, Wigner, and Pastur distributions; Pareto, second kind beta, Dagum, Davis, Singh, heavy-tailed distributions such as the Maddala, Cauchy, Landau, Levy, Benini, Gibrat, and Suzuki distributions; quantile-based distributions such as the Tukey and Wakeby distributions; systematic distributions such as the Pearson distribution; multivariate continuous distributions such as the multivariate normal, multinomial, multivariate t-distribution, Dirichlet, and Copula distributions; and mismatched or heterogeneous combinations of various distributions, including, but not limited to, multivariate discrete distributions such as the multivariate hypergeometric, negative multinomial, and multivariate Poisson distributions.

[0034] For example, the probabilistic DNN block uses heterogeneous assignment of different beliefs for at least two disjoint sets of probabilistic nodes in one layer; for example, the first 70% of the probabilistic nodes in the second layer follow a logistic distribution, and the remaining 30% of the probabilistic nodes in the same layer follow a Cauchy distribution. In another example, the probabilistic DNN block uses mismatched pairs of posterior beliefs and prior beliefs for at least one set of probabilistic nodes; for example, the probabilistic nodes in the third layer use a uniform distribution for the posterior beliefs, and the same probabilistic nodes use a Laplace distribution for the prior beliefs. In addition, in some embodiments, the selection of irregular beliefs is adaptively modified on the fly to use different probability distributions for variational sampling in the probabilistic nodes in the same probabilistic DNN block when analyzing a newly available set of data sets, even though the probabilistic DNN block was previously trained with different irregular beliefs. For example, the probabilistic nodes are trained to follow a normal distribution for the training data set, but the same probabilistic nodes use a logistic distribution for the testing data set. This allows further adaptation to reduce model mismatch in unseen testing data sets.

[0035] The probabilistic DNN block provides probabilistic inference to analyze a dataset by using importance-weighted accumulation after variational sampling at probabilistic nodes according to irregular beliefs. Variational sampling uses a random number generator based on a reparameterization trick according to variational parameters such as position, scale, shape, and temperature specified by the irregular beliefs. To optimize the probabilistic DNN block, some embodiments perform the steps of: exploring different values ​​for the irregular beliefs and hyperparameters by using a supergradient method; constructing the probabilistic DNN block by modifying the connectivity between multidimensional signals, task labels, and probabilistic nodes according to the hyperparameters; calculating a loss function by forward propagating the dataset through the probabilistic DNN block according to probabilistic inference; modifying the loss function by regularizing the probabilistic nodes according to the discrepancy measure; backpropagating the gradient of the loss function with respect to the trainable parameters; and updating the trainable parameters using a gradient method.

[0036] The steps mainly include two methods for inference and optimization. The inference method for the probabilistic DNN model uses: feeding a data signal to a DNN with a certain number of layers and probabilistic nodes; propagating the signal according to the layers while the probabilistic nodes use variational sampling according to irregular beliefs; and accumulating the output of the DNN as a probabilistic inference result. In addition to the inference method, the optimization method further uses: calculating a loss function based on the variational bound to regularize the probabilistic nodes according to the discrepancy measure; backpropagating the gradient of the loss function with respect to the trainable parameters; updating the trainable parameters using a gradient method; and searching for different values ​​for the irregular beliefs and hyperparameters using a supergradient method. The supergradient method enables the efficient construction of a probabilistic DNN model with higher performance than a traditional variational inference model that relies on regular beliefs with homogeneous matched assignments for the probabilistic nodes. Auxiliary Regularization Module

[0037] 1B and 1C show an exemplary inference model for predicting Y given data X under latent variables Z and nuisance factors S according to some embodiments. The DNN model uses a main pipeline, which is decomposed into an encoder unit 101 and a classifier unit 102 (or a regressor unit for regression tasks), in which the encoder unit 101 extracts a feature vector as latent variables Z from data X, and the classifier unit 102 predicts a task label Y from the latent variables Z. For example, the latent variables Z are vectors of hidden nodes in the intermediate layer of the DNN model. To stabilize the training convergence of the DNN model, some embodiments use a regularization technique on the latent variables Z. Besides the main pipeline of the encoder 101 and the classifier 102, FIG. 1B uses an additional auxiliary regularization module to regularize the latent variables Z. Specifically, the decoder DNN block 103 reconstructs the original data X with the optional conditional information of the nuisance variation S, appended with the latent variables Z. This conditional decoder 103 can facilitate disentanglement of nuisance domain information S from latent variables Z. For example, for a physiological dataset, the nuisance domain variations S include subject identification (ID), measurement session ID, noise level, subject height / weight / age information, etc. By separating these nuisance factors S from the latent variables Z, the present invention can, in some embodiments, realize a subject-invariant universal human-machine interface without lengthy calibration sessions. An auxiliary DNN block called decoder 103 is trained to minimize another loss term, such as the mean squared error and Gaussian negative log-likelihood loss, to reconstruct X from Z.

[0038] In some embodiments, the latent variables are further denoted by Z1, Z2, ..., Z L Each of them is decomposed into multiple latent factors S1, S2, ..., S NIn addition, some nuisance factors are partially known or unknown depending on the dataset. For known labels of nuisance factors, the DNN block can be trained in a supervised manner, while for unlabeled nuisance factors, a semi-supervised method is required. In the semi-supervised case, in some embodiments, pseudo-labeling based on variational sampling across all potential labels of nuisance factors is used, which is based on reparameterization tricks such as the Gumbel softmax trick, which uses a variational temperature parameter to control the sharpness of the nearly one-hot samples for the classification nuisance factors. For continuous nuisance factors, variational sampling is based on a reparameterization trick that uses variational parameters including position, scale, and shape to determine a probability distribution. For example, some of the data in the dataset lacks subject age information, while the remaining data has age information to be used for supervised regularization.

[0039] The DNN block in FIG. 1B has another auxiliary regularization module 104 attached to the latent variables Z to estimate the nuisance variation S. This regularization DNN block 104 is used to further promote the disentanglement of nuisance factors to make it more robust, which is often called an adversarial network because the main pipeline DNN block is trained to maximize a loss function while the regularization DNN block is trained to minimize a loss function to estimate S from Z to censor the nuisance information. The adversarial block is trained in an alternating fashion with associated hyperparameters including adversarial coefficients, adversarial learning rate, interval of adversarial alternation, and architecture specifications.

[0040] The graphical model in FIG. 1B is known as an Adversarial Conditional Variational Autoencoder (A-CVAE) for Unsupervised Feature Extraction in conjunction with the downstream task classifier 102. The graphical model of A-CVAE has various graph nodes and graph edges to represent the connectivity covering the random variable factors X, Y, Z, and S. With the regularization block, the A-CVAE model has higher robustness against nuisance factors. Therefore, the auxiliary regularization module is used as a so-called pre-shot transfer learning or domain generalization technique to make the AI ​​model robust against unseen nuisance factors.

[0041] We provide a method for efficiently assigning multiple distinct beliefs to individual probabilistic nodes Z1, Z2, ..., Zn. If there are 10 distinct posterior-prior pairs for 100 probabilistic nodes, then we can assign up to 10 distinct beliefs to specify a probabilistic DNN. 100 There are potential combinations. Some embodiments use an automated machine learning (AutoML) framework based on a supergradient method to facilitate the discovery of better combinations of such heterogeneous pairings of posterior / prior beliefs. In addition, some embodiments allow different discrepancy measures for individual probabilistic nodes. For example, Renyi's alpha divergence of order 0.1 is used for Z1, Z2 uses an alpha order of 0.2, and KLD is used for Z3. The present invention allows such irregular probabilistic models through the use of AutoML search, and the search space grows rapidly. AutoML's supergradient method is based on a combination of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, and Bayesian optimization to efficiently search for better combinations of irregular beliefs for posterior, prior, and likelihood distributions, as well as discrepancy measures, simultaneously for individual probabilistic nodes. Architecture Exploration

[0042] There are many possible ways to connect the encoder 101, the classifier 102, the decoder 103, and the adversarial network block 104. For example, FIG. 1C shows one exemplary DNN block in which there is another auxiliary module 105 to estimate the nuisance factor S from the data X. The possible number of feasible DNN connections explodes rapidly with the size of the AI ​​model, so there is a need to efficiently build a reasonable AI model. In addition, randomly connected DNN blocks tend to be useless and unreasonable. In some embodiments, the graph connectivity is explored by using an automatic Bayesian graph search method called AutoBayes. At the heart of AutoBayes is the consideration of a graphical Bayesian model that captures the probabilistic relationships between the random variables representing the data features X, the task labels Y, the nuisance variation labels S, and the latent representations Z. The main goal is to infer the task labels Y from the measured data features X, which is hindered by the presence of nuisance variations (e.g., inter-subject / inter-session variations) that are partially labeled by S. A potential representation Z (and optionally Z1, Z2, ..., Z L ) is also optionally introduced into the AI ​​model to help capture the underlying relationships between S, X, and Y.

[0043]

number

[0044] The above graphical models do not impose any assumptions of potentially inherent independence in the dataset and are therefore the most comprehensive. However, some edges in these graphs can be pruned depending on the underlying independence in the dataset. For example, if the data has a Markov chain of YX independent of S and Z, it automatically results in the AI ​​model shown in Figure 1A. This means that the most complex inference model with high degrees of freedom does not always perform best across any dataset. This encourages us to consider an extended AutoML framework that automatically searches for the best pair of inference factor graph models and corresponding Bayesian graph models that match the dataset, in addition to other hyperparameters.

[0045] AutoBayes starts by exploring any potential Bayesian graph by cutting links in the full-chain graph and imposing possible independence. Then, we employ the Bayesball algorithm on each hypothetical Bayesian graph to examine conditional independence across different inference strategies. The Bayesball algorithm justifies the reasonable pruning of links in the full-chain inference graph, and also justifies potential adversarial censorship when Z is independent from S. This process automatically builds the connectivity of inference, generative, and adversarial blocks with good inference, for example, to build the A-CVAE classifier in Figure 1B, but not any extended model in Figure 1C. An example probabilistic DNN model

[0046] FIG. 2 shows an exemplary schematic diagram of an automatic VAE according to some embodiments. Generalized variational inference (GVI) offers a large degree of freedom, but in turn makes it difficult to design an appropriate choice of probabilistic models in addition to other architecture hyperparameters. To solve this problem, the present invention uses automatic construction of probabilistic DNN models. FIG. 2 shows such an example of an automatic VAE, also called AutoVAE, which facilitates finding appropriate choices of posterior, prior, likelihood, and mismatch measures by using an automatic machine learning (AutoML) framework based on hypergradient methods. Although normal distributions are often used for potential sampling in the probabilistic nodes of standard VAEs, it is not always optimal when the underlying uncertainty in the dataset is complex or unknown. The present invention provides a method to improve VAE models by allowing irregular beliefs for probabilistic nodes. By exploring diverse settings of irregular beliefs, the variational Renyi (VR) bound can be improved by mismatch pairing, for example, when a logistic distribution is used as the posterior belief in the VAE encoder, even though the VAE decoder assumes a normal distribution as the prior belief.

[0047] The VAE uses an encoder block 201 and a decoder block 203 to recover the data signal from the latent variables 202. The last node of the VAE encoder is a probabilistic node, which provides variational parameters to follow a certain probability distribution given the data signal. Thus, the VAE encoder represents a posterior belief that is specified by the variational parameters such as position, scale, and shape. For example, the first half of the output node of the VAE encoder is the position parameter, and the second half of the output node of the VAE encoder is the scale parameter. The latent variables 202 are then sampled by a random number generator and the random numbers sampled from the standard distribution are scaled and transformed by the variational parameters generated by the VAE encoder according to the posterior belief by a reparameterization trick 205.

[0048] The VAE decoder takes sampled latent variables 202 and generates an estimated data signal. The final node of the VAE decoder is a probabilistic node, which provides variational parameters to follow a particular probability distribution given the latent samples. Thus, the VAE decoder expresses likelihood beliefs 208 that are specified by variational parameters such as location, scale, and shape. For example, the output node of the VAE decoder is the location parameter of a normal distribution, but the scale parameter is an unspecified constant, i.e., the generalized NLL loss is reduced to the MSE loss.

[0049] The VAE encoder and decoder are jointly trained by gradient methods to maximize the evidence lower bound (ELBO), which is a function of the generalized NLL loss and the discrepancy 207 between the posterior and prior beliefs 206. The discrepancy term acts as a regularizer of the probabilistic nodes that control the probability distribution. Once the VAE encoder and decoder are trained, the VAE decoder can be used as a generative model to generate synthetic signals close to the original dataset by feeding random numbers sampled from the prior beliefs. Some embodiments of the present invention use adaptive adaptation of the prior beliefs to sample latent variables from different probability distributions between the training and testing datasets.

[0050] AutoVAE uses the AutoML framework 204 to explore various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and mismatch measures by using a hypergradient method so that the best ELBO is achieved. The AutoVAE system allows various irregular pairings for probabilistic nodes, such as i) matched pairings using the same probability distribution for posterior beliefs and prior beliefs; ii) mismatched pairings using different probability distributions for posterior beliefs and prior beliefs; and iii) heterogeneous pairings using different probability distributions for posterior beliefs and prior beliefs non-uniformly at individual probabilistic nodes. FIG. 3 illustrates exemplary variations of irregular beliefs, according to some embodiments. The first variation 301 of FIG. 3 is when the posterior beliefs and prior beliefs are matched to be Laplace distributions. The second variation 302 of FIG. 3 is when the posterior beliefs using Laplace distributions are mismatched with the prior beliefs using normal distributions. The third variant 303 of Figure 3 is when the probabilistic nodes are heterogeneous, specifically using posterior-prior pairs where the first half of the latent variable is different from the second half of the latent variable. The use of such irregular beliefs can significantly improve the expressive power of the probabilistic DNN model, thus offering the benefit of being able to efficiently model more complex statistics. In addition, the standard KLD regularization can be significantly extended by exploring discrepancy measures across Renyi's alpha-divergence, beta-divergence, gamma-divergence, etc.

[0051] FIG. 4A shows exemplary beliefs of posterior, prior, and likelihood distributions used in probabilistic DNNs according to some embodiments. Corresponding notations of various random distributions and their probability density function (PDF) representations are listed for convention. Some distributions are specified with a single variational parameter, e.g., scale for exponential distributions. Some distributions are specified with two variational parameters, e.g., location and scale for Laplace distributions. In some embodiments, variational parameters of more than two shapes are also used for other distributions, including but not limited to skew normal, exponential power, hyperbolic, gamma, Johnson, Pearson, student-tt, and Pareto distributions. Some distributions belong to the location scale distribution family (LSF), which is suitable for variational sampling because non-standard distributions can be easily generated by standard distributions through the use of reparameterization tricks. For discrete probability distributions such as the Gumbel distribution, a modified reparameterization trick called the softmax trick is used with a variational parameter called temperature to adjust the sharpness towards one-hot encoding samples.

[0052] As an example, x ∈ R N Let be the N-dimensional data input to the VAE encoder. The encoder generates variational parameters such as the location, scale, and shape of the posterior belief hypotheses to encode an L-dimensional latent variable z∈R L The latent variables are then fed into a VAE decoder to generate the variational parameters of the likelihood belief hypotheses to obtain the reconstructed data x′∈R N The encoder and decoder are constructed using parameterized DNNs, and are φ :x → z and p ψ :z→x', where φ and ψ are the trainable parameters of the DNN, such as weights and biases. The DNN attempts to minimize the reconstruction loss, which would typically be the negative log-likelihood (NLL) under the likelihood-belief hypothesis.

[0053] For a given choice of parameters φ and ψ, the VAE encoder and decoder models, respectively, have the posterior q φ The conditional distribution known as (z|x) and the likelihood p ψ This means the conditional distribution known as (x|z). If π(z) is the probability distribution of the latent variable z under the prior beliefs, then the VAE is

number

number

number

[0054] The VAE encoder and decoder are jointly trained such that the ELBO is maximized under a variational Bayesian framework.

[0055] In ELBO, there are four important factors: the likelihood belief P=p ψ (x|z); posterior belief Q=q φ (z|x); prior belief Π = π(z); and discrepancy measure D = D KL(.||.) is specified. Standard VAEs often use a normal distribution (or a Bernoulli distribution for near-binary image reconstruction) for the likelihood beliefs P=N(λ,γ), and a specific KLD D KL We regularize the latent variables by assuming a normal posterior Q=N(mu,sigma) and a normal prior Π=N(0,1) using (Q||Π). Based on GVI, the present invention allows for arbitrary selection of generalized NLL loss, discrepancy measure, prior belief, and posterior belief. Standard VAE is asymptotically optimal for infinite dimensions, but optimality is no longer justified when likelihood / posterior / prior beliefs are misspecified or unspecified in contrast to the real data distribution.

[0056] Although the exponential family prior beliefs will result in the same exponential family posterior beliefs under certain conditions, the present invention provides an extended GVI by considering irregular beliefs such as mismatched pairing of posterior-prior beliefs, e.g., logistic posterior and Laplace prior. Furthermore, the generalized VAE allows for another irregular belief using heterogeneous pairing, e.g., 30% of latent nodes use Laplace-normal pairing and 70% of nodes use uniform-Cauchy pairing. Some prior art considers non-Gaussian probabilities such as Laplace and Cauchy distributions, but does not consider irregular beliefs with mismatched or heterogeneous pairing. To design such an irregular VAE, the present invention provides a set of convenient pairings with closed-form differentiable representations and straightforward reparameterization of KLD, without requiring high complexities such as information autoregressive flow (IAF) and Gibbs sampling.

[0057] In the prior art, the normal distribution is the most widely used belief for probabilistic DNNs, allowing simple sampling, reparameterization tricks, and closed-form expressions of KLD. In some embodiments of the present disclosure, we consider a variety of alternatives, such as the location-scale distribution family LSF(μ,σ), which preserves the same distribution within transformations with variational parameters for location μ∈R and scale σ∈R+. For example, given a random variable ε drawn as ε~LSF(0,1), its transformed random variable Z=μ+σ·ε follows the same family as Z~LSF(μ,σ), where the notation "~" means "distributed to". This transformation is known as the reparameterization trick for variational sampling, and is one of the key methods that enable probabilistic DNN training to back-propagate gradients of the variational parameters μ and σ while maintaining the target distribution in forward variational sampling.

[0058] As shown in the example list in FIG. 4A, the present disclosure considers a diverse set of beliefs including LSF distributions: normal N; Laplace La; Cauchy C; logistic Lo; uniform U; Gumbel G; and exponential E (which is not an LSF but is a scale distribution family). FIG. 4B shows an example standard position scale distribution family used for variational sampling based on reparameterization tricks according to some embodiments. Several PDFs of its standard form LSF(0,1) are plotted in FIG. 4B. These distributions are useful as a choice of potential posterior beliefs Q to generate a target non-standard distribution of LSF(μ,σ) 404 for a VAE encoder due to a simple reparameterization trick 400 by transformation of a standard random sample LSF(0,1) 403 with two variational parameters: position 401 and scale 402. For example, sampling of the logistic distribution can be done as Z=μ+σε~Lo(μ,σ), with V,W~E(1) for ε=log(V / W)~Lo(0,1). Position vector μ∈R L and the scale vector σ∈R L is generated from the data x by a parametric VAE encoder, and the latent z is a standard random variable ε~LSF(0,1)L are stochastically sampled by transformation from

[0059] For the selection of the prior beliefs Π=π(z) used in the generative model with the VAE decoder, we typically use a standard normal distribution N(0,1), or for a particular selection of the posterior beliefs Q=LSF(μ,σ), a matched standard prior Π=LSF(0,1). The probabilistic nodes are then regularized to achieve a posterior distribution close to the prior distribution by a discrepancy measure such as KLD. FIG. 5A shows an example landscape of KLD discrepancies for various pairs of posterior and prior beliefs at two independent probabilistic nodes according to some embodiments. For example, the KLD landscape for the pair of normal posterior and normal prior beliefs 501 is circularly isotropically symmetric across any axis so as to penalize samples with larger magnitudes to behave like a Gaussian distribution. For the pair of Laplace posterior and Laplace prior beliefs 502, the KLD landscape becomes non-isotropic and takes on a rounded square shape, which promotes sparse regularization. For another pair of Cauchy posterior and prior beliefs 503, the KLD landscape is shaped like a rounded cross, which generally encourages a sparser regularization.

[0060] Although we can choose any distribution for the prior beliefs, regardless of the posterior beliefs, it is desirable to have a closed-form simple expression of the KLD to measure the discrepancy between the posterior Q and the prior Π. In some embodiments, we consider such pairs of posterior and prior beliefs. FIG. 5B shows example pairs of posterior and prior beliefs quantified by a KLD discrepancy measure, according to some embodiments. In FIG. 5B, closed-form expressions of the KLD discrepancy measure for some pairs of posterior and prior beliefs are listed for convention. In some embodiments, the KLD is also obtained by numerical integration via variational sampling and accumulation for other irregular beliefs.

[0061] Yet another embodiment uses adaptive prior modification on the fly after the VAE is trained. For example, the distribution of latent variables of the optimized VAE encoder may end up having a different distribution than the target prior distribution. To solve the mismatch problem, the prior belief is modified when synthetically generating new data, depending on some statistics such as the mean, variance, and skewness of the latent variables that match the newly available dataset.

[0062] Matched posterior-prior pairings generally have smaller KLDs, but mismatched pairs are not necessarily worse than matched posterior-prior pairings. FIG. 5C shows exemplary KLD properties for various pairs of posterior and prior beliefs as a function of position value, according to some embodiments. For example, mismatched pairs of logistic-Laplace beliefs have slightly smaller KLDs than matched pairs of Laplace-Laplace beliefs at moderate position values. More importantly, such mismatched cases have sufficiently small KLD values ​​for a wide range of positions μ. Since the KLD values ​​cannot be exactly zero even for matched pairs unless μ=0 and σ=1, it is not necessarily important to minimize the KLD terms. This implies that there is a higher probability that exploring different irregular pairings may ultimately provide a better ELBO than the traditional regular pairings, since there is more freedom in adjusting the KLD landscape and ELBO. The present invention is based on this recognition.

[0063] It should be noted that KLD is the most commonly used discrepancy measure to evaluate the “difference” between the posterior Q and the prior Π for VAEs. However, besides KLD, probabilistic DNNs can use various other discrepancy measures, including Fisher, Jeffrey, and Renyi’s α, β, and γ divergence. In particular, Renyi’s α-divergence is useful because it covers many variants, such as importance-weighted autoencoders (IWAEs) and standard VAEs, as a special case in adjusting the divergence order alpha. Figure 6 shows an example discrepancy measure based on Renyi’s α-divergence, which covers variants of generalized variational inference, including importance-weighted autoencoders (IWAEs) and standard VAEs, according to some embodiments. For example, when α→1, the α-divergence D α (Q∥Π) is KLD D KL (Q∥Π). In another example, the alpha divergence reduces to a variant of the Hellinger distance and the chi-squared divergence. The Renyi divergence of order α≧0 can be expressed as:

number

[0064] More importantly, it has been shown that the ELBO based on the Renyi divergence has a tighter bound than the KLD case (α → 1). Specifically, the variational Renyi (VR) bound is given by k ~p φ It is approximated by an importance-weighted accumulation of (z|x):

number

[0065] The IWAE is a special case of the VR framework with α=0, which converges to the marginal probability as K→∞. Since the α-divergence is closely related to the KLD, the aforementioned pair in FIG. 5B also has a closed-form expression for the α-divergence. Thus, some embodiments use the Renyi α-divergence as a generalized discrepancy measure, where the Renyi order alpha is adjusted to find the best ELBO.

[0066] For VAE, in practice any differentiable loss measure can be used as the reconstruction loss, e.g., mean squared error (MSE), mean absolute error (MAE), and binary cross entropy (BCE) in addition to NLL. Nevertheless, most loss functions are based on a particular likelihood belief P=p ψ Under (x|z), we express the generative model of the data through the VAE decoder, closely related to the generalized NLL. FIG. 7 shows an example of the likelihood belief P of a probabilistic DNN for a generalized negative log-likelihood loss according to some embodiments. Various likelihood beliefs P for a probabilistic DNN are listed in FIG. 7, where the generalized NLL loss is specified for different likelihood beliefs. For example, BCE is often used for nearly binary images such as the MNIST dataset. BCE is equivalent to NLL under a likelihood belief based on the Bernoulli distribution B(λ). Some embodiments use the beta distribution Be as the appropriate likelihood belief, while some embodiments use its modified belief called the continuous Bernoulli distribution CB to improve the VAE for less strictly binary images. MSE corresponds to NLL under the normal distribution likelihood N(x',*) when omitting the unspecified variance. MAE corresponds to NLL under the Laplace distribution likelihood La(x',*) when omitting the unspecified measure.

[0067] A VAE decoder may require multiple variational outputs, such as location, scale, and shape, to generate x' given a likelihood belief P. For example, a normal distribution likelihood P=N(λ,γ) provides the mean as the reconstructed data x'=λ and the standard deviation of that γ as the confidence. Similarly, some embodiments use x'=λ in the sense of maximum likelihood for the Laplace distribution. In another example, a beta distribution as the likelihood belief also has two variational parameters to be generated by a VAE decoder. In general, given the decoder output of the variational parameters (such as λ, γ), the data reconstruction should be done by its mode (i.e., the peak of the PDF). For the Bernoulli likelihood B(λ), the mode is binary, since x'=0 or 1 depending on λ, but some embodiments use the mean as the reconstruction x'=λ instead. In some embodiments, the output of the probabilistic DNN after multiple variational sampling is accumulated using a weighted average to improve the accuracy of the variational inference.

[0068] VAE and its variants for probabilistic DNN models require specifying four probabilistic factors: posterior beliefs Q; prior beliefs Π; likelihood beliefs P; and mismatch measure D. To design these factors, we often require manual effort in exploring the best combination of different likelihood beliefs as listed in FIG. 7, different posterior-prior pairs as listed in FIG. 5B, and different divergence orders α>0 as listed in FIG. 6. The search space will grow rapidly when irregular, heterogeneous, and mismatched pairings are considered at each latent node. The present invention facilitates the design of such heterogeneous probabilistic DNNs by exploring those factors for VAE design in an automated manner as shown in FIG. 2 using an AutoML framework, and we call this concept AutoVAE. Exemplary Model Implementation

[0069] Rare irregular pairings of posterior-prior beliefs, such as logistic-normal or Laplace-normal, can outperform the common choice of normal-normal pairings when searching for the best divergence order α. The irregular heterogeneous pairings explored in AutoVAE can further achieve better performance due to the higher degrees of freedom. For example, as an exemplary model implementation, posterior-prior beliefs are mixed in a heterogeneous manner by AutoVAE as logistic-normal (Lo∥N) pairs, Laplace-normal (La∥N) pairs, and Laplace-Laplace (La∥La) pairs for 50%, 30%, and 20% of the latent variables, respectively, to achieve higher ELBO performance than standard VAE when analyzing the MNIST practical benchmark dataset.

[0070] Some embodiments use a simple VAE architecture based on multi-layer perceptrons (MLPs) for the encoder and decoder blocks, where the MLPs consist of several fully connected linear layers with hundreds of hidden nodes with rectified linear unit (ReLU) activations. For example, a two-layer MLP with 400 hidden nodes is used for both the VAE encoder and the VAE decoder. The number of latent nodes is selected in some embodiments to be about tens of them depending on the dataset. For example, L=20 stochastic nodes are used. Some embodiments use a gradient method based on Adam gradient optimization with a learning rate of 0.01 over 100 epochs with a mini-batch size of 1000. AutoML can also design such hyperparameters on irregular beliefs by using a hyper-gradient method such as Bayesian optimization.

[0071] Once the VAE is trained, the VAE decoder can be used as a generative model to reproduce synthetic data by feeding random samples drawn from the prior beliefs Π for some embodiments. For example, the generated images are evaluated by Fréchet initiation distance (FID) and kernel initiation distance (KID) to evaluate their natural distribution from the original image dataset. To evaluate the initiation score, some embodiments generate 50,000 images by sampling random latent variables z from the prior beliefs Π.

[0072] FIG. 8A shows exemplary characteristics of various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and mismatch measures of a generalized VAE model according to some embodiments. More specifically, FIG. 8A shows the results of mismatch VAE under various combinations of likelihood, posterior, and prior beliefs after training on the benchmark MNIST dataset. For the MSE loss based on likelihood beliefs of unspecified normal distribution P=N(λ,*), the ELBO performance of regular normal-normal pairs (for order of alpha=1 and cumulative number of K=1) can be improved by exploring different pairings such as logistic-normal pairs, Laplace-Laplace pairs, and logistic-Laplace pairs. The MSE and starting scores are also improved by irregular beliefs, for example, of logistic-normal pairs. When exploring different likelihood beliefs, the benefit of mismatch pairing can be achieved even more clearly. Furthermore, the irregular heterogeneous use of posterior-prior pairs at individual latent nodes (denoted as “Auto”) can further improve performance in some cases via automatic exploration of mixed pairs for irregular beliefs.

[0073] Moreover, by exploring the Renyi order α, the VR bounds can be further improved. FIG. 8B shows exemplary ELBO properties for various pairs of posterior and prior beliefs as a function of divergence order for a generalized VAE model according to some embodiments. Specifically, FIG. 8B shows the advantage of exploring different disparity measures by adjusting the divergence order α for some embodiments. FIG. 8B demonstrates that when searching for the best divergence order α, rare pairings of posterior and prior beliefs, such as logistic-normal or Laplace-normal, can outperform the common choice of normal-normal pairings. It has also been confirmed that the irregular belief pairings explored in AutoVAE achieve the best performance for the MNIST dataset. The best posterior-prior mixtures for heterogeneous assignment were the logistic-normal (Lo∥N), Laplace-normal (La∥N), and Laplace-Laplace (La∥La) pairs, respectively, for 50%, 30%, and 20% of the L=20 latent nodes at the Renyi divergence order with α=0.5. This exemplary result clearly shows the benefit of the present invention of using irregular beliefs for probabilistic DNN models by enabling mismatched and heterogeneous pairing with an adjustable discrepancy measure.

[0074] 8C and 8D show example image snapshots generated according to prior beliefs with variational sampling by a generalized VAE decoder after being trained on various pairs of posterior and prior beliefs according to some embodiments. Specifically, FIG. 8C and 8D show synthetic image snapshots of randomly generated images by using a VAE decoder trained on the MNIST dataset. By exploring different combinations of posterior beliefs, prior beliefs, likelihood beliefs, and disparity measures, the generator model can be significantly improved, especially with heterogeneous pairing designed by the AutoML framework. Similar benefits can be achieved for different datasets. The present invention uses mismatched, heterogeneous, and irregular combinations of posterior-prior-likelihood beliefs for generalized VAE design. The mismatched VAE can outperform standard VAE by using an automatic search of irregular beliefs. In addition, the systems and methods of the present invention explore different disparity measures by adjusting the divergence order in some embodiments. The concept of AutoVAE can facilitate exploring the appropriate combination of those probabilistic factors to construct a generalized VAE model for variational inference. The irregular belief pairing technique is also applicable to other probabilistic DNNs other than VAE. An example probabilistic DNN model

[0075] Besides VAE, probabilistic DNN models include, but are not limited to, variational information bottleneck (VIB), denoising diffusion probabilistic model (DDPM), and variational Bayesian neural network in some embodiments. VIB uses a similar but different approach of VAE, where VIB is suitable for supervised learning and VAE is suitable for unsupervised learning. Probabilistic nodes of VIB are identified by irregular beliefs in a similar manner of AutoVAE, and thus some embodiments use automatic search of irregular beliefs in VIB to improve performance. DDPM is another probabilistic DNN model similar to VAE, but with multiple variational sampling in the diffusion step. Similar to AutoVAE, some embodiments use automated search of irregular beliefs to realize mismatch and heterogeneous pairing in probabilistic nodes in each diffusion step. Similarly, besides VAE, VIB, and DDPM, some embodiments use different probabilistic DNN models when engaging probabilistic nodes under certain assumptions of underlying probabilistic beliefs.

[0076] FIG. 9A illustrates an exemplary probabilistic DNN that uses multiple different beliefs and discrepancy measures sequentially in multiple intermediate layers according to some embodiments. This exemplary probabilistic DNN model in FIG. 9A takes in data x901 and generates variational inference for task label y905 through multiple DNN blocks sequentially, with some hidden layers being probabilistic nodes that use variational sampling according to individually specified beliefs. For example, the first probabilistic node z1902 uses a normal distribution for posterior beliefs and a uniform distribution for prior beliefs with KLD discrepancy; the second probabilistic node z2903 uses a Cauchy distribution with Renyi alpha divergence of order 0.5 for both posterior and prior beliefs; the third probabilistic node z3904 uses a logistic distribution for posterior beliefs and a normal distribution for prior beliefs with beta divergence of order 0.2. Such probabilistic DNNs with multiple probabilistic layers in turn provide more complex uncertainty models and heterogeneous beliefs can further improve the power of probabilistic modeling for variational inference.

[0077] FIG. 9B illustrates an exemplary probabilistic DNN that uses different beliefs and discrepancy measures in different probabilistic nodes, in parallel in one hidden layer, and sequentially in multiple hidden layers, according to some embodiments. This exemplary probabilistic DNN model realizes heterogeneous beliefs in one probabilistic layer to provide variational inference y 915 from data x 911 through multiple DNN blocks with parallel probabilistic nodes z1 912 and z2 913 in addition to sequential probabilistic node z3 914. Specifically, parallel probabilistic nodes z1 912 and z2 913 are heterogeneously identified with different beliefs and divergence metrics. Many other possibilities for connecting multiple DNN blocks with multiple probabilistic nodes are explored using the hypergradient method to realize a high-performance generalized variational inference model for some embodiments.

[0078] Each of the probabilistic DNN blocks is configured with hyperparameters that specify a set of layers with trainable variables and interconnected neuron nodes to pass signals from layer to layer sequentially. The trainable variables are numerically optimized using gradient methods such as stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, and root mean square propagation. The gradient methods update the trainable parameters of the DNN block by using the training data such that the output of the DNN block provides smaller loss values ​​such as mean squared error, cross entropy, structural similarity, negative log-likelihood, absolute error, cross covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, and triplet loss. The multiple loss functions are further weighted to combine with some regularization factor according to a training schedule policy.

[0079] In some embodiments, the probabilistic DNN block is reconfigurable according to hyperparameters such that the DNN block is composed of a set of fully connected layers, convolutional layers, graph convolutional layers, recurrent layers, multi-loop connections, skip connections, and initiation layers, with a set of nonlinear activations including rectified linear transforms, hyperbolic tangents, sigmoids, gated linear, softmax, and thresholds. The DNN block is further regularized with a set of dropout, swap out, zone out, block out, drop connect, noise injection, shaking, and batch normalization. In yet another embodiment, the layer parameters are further quantized to reduce the size of the memory as specified by the adjustable hyperparameters. In another embodiment of link concatenation, the system uses multidimensional tensor projections with dimensionally trainable linear filters to convert lower dimensional signals to higher dimensional signals for dimensionally mismatched links.

[0080] Another embodiment integrates AutoML with AutoBayes and AutoTransfer for hyperparameter exploration and learning scheduling of each DNN block. Here, AutoTransfer uses exploration of different auxiliary regularization modules from a diverse set of regularization methods, including but not limited to adversarial censorship, mutual information gradient estimation, and Wyner distance. It should be noted that AutoTransfer and AutoBayes can be easily integrated with AutoML based on supergradient methods to optimize any hyperparameter of an individual DNN block. More specifically, the system modifies hyperparameters by exploring different combinations of discrete and continuous hyperparameter values ​​using reinforcement learning, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, Bayesian optimization, hyperbands, and multi-objective Lamarckian evolution.

[0081] The system of the present invention also provides an additional testing step to suit as a post-training step to improve the trained DNN block by unpacking some trainable variables so that the DNN block can be robust to new data sets with new nuisance variations, such as new subjects. This embodiment can reduce the calibration time requirements for new users of the HMI system. Exemplary System

[0082] FIG. 10 illustrates an exemplary schematic diagram of a system comprised of a processor, a memory, and an interface, according to some embodiments. Specifically, FIG. 10 illustrates a block diagram of an example of a system 1000 for signal analysis with automatic configuration of a probabilistic DNN. The system 1000 includes a device 1100 having a set of interfaces and data links 1105 configured to transmit and receive signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and a storage 1140. The processor 1120, in association with the memory 1130, executes computer-executable programs and algorithms stored in the memory 1130 and the storage 1140. The set of interfaces and data links 1105 includes a human machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 can execute computer-executable programs and algorithms in association with the memory 1130 that uploads computer-executable programs and algorithms from the storage 1140. The computer executable programs and algorithms stored in the storage 1140 use a probabilistic deep neural network (DNN) 1141, trainable parameters 1142, hyperparameters 1143, scheduling policies 1144, forward pass signal / backward pass gradient and other temporary caches 1445, a regularization module 1146, gradient methods for training 1147, hypergradient methods for AutoML 1148, and irregular belief sets for probabilistic nodes 1149. The processor performs probabilistic inference to analyze a set of data sets by using the probabilistic DNN 1141, where the probabilistic inference uses importance weighted accumulation after variational sampling at the probabilistic nodes according to the irregular belief sets 1149.

[0083] The system 1000 receives signals via a set of interfaces and data links 1105 over a network 1190. The signals include a set of datasets 1195 having data for training, validation, and testing. The set of datasets includes a set of multidimensional signals X associated with task labels Y to be identified. In some embodiments, the set of datasets further includes other side information such as nuisance variations S.

[0084] In some cases, each of the probabilistic DNN blocks 1141 is configured to either encode a multidimensional signal X into latent variables Z, decode the latent variables Z to reconstruct the multidimensional signal X, classify a task label Y, estimate a nuisance variation S, regularize a latent space by estimating a nuisance variation S, or select a graphical model. The memory bank further includes intermediate neuron signals and temporary calculations including forward pass signals and backward pass gradients.

[0085] The processor 1120, in conjunction with the interface and memory bank 1130, is configured to submit the signal and the dataset 1195 to the probabilistic DNN block 1141 and search for different irregular beliefs 1149 and different hyperparameters 1143 by using a hypergradient method 1148. The processor 1120 further performs the following: configuring the probabilistic DNN 1141 according to the hyperparameters 1143; calculating a loss function by forward propagating the dataset 1195 through the probabilistic DNN 1141 according to probabilistic inference; modifying the loss function by regularizing the probabilistic nodes according to a discrepancy measure specified in the irregular beliefs 1149; and backpropagating the gradient of the loss function through the probabilistic DNN 1141 with respect to the trainable parameters 1142 to update the trainable parameters 1142 using a gradient method 1147. The hypergradient method for AutoML 1148 also explores different auxiliary regularization modules 1146, different hyperparameters 1143, different scheduling policies 1144, and irregular probabilistic beliefs 1149 to improve robustness to nuisance variations and unspecified uncertainties in the dataset 1195.

[0086] In some embodiments, the system 1000 is applied to design an HMI through the analysis of a user's physiological data. The system 1000 receives signals of physiological data 1195 via a network 1190 and a set of interfaces and data links 1105. In some embodiments, the system 1000 receives electroencephalogram (EEG) and electromyogram (EMG) measurements from a set of sensors 1111 as the user's physiological data and from other interface modules such as a pointing device / medium 1112.

[0087] The above-described embodiments of the present invention may be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit with one or more processors within an integrated circuit component. However, the processor may be implemented using any suitable form of circuitry.

[0088] Also, embodiments of the invention may be embodied as methods, of which examples are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that illustrated, which may include performing simultaneously some acts shown as sequential acts in the exemplary embodiments.

[0089] Moreover, the use of ordinal numbers such as "first," "second," etc. to modify claim elements in the claims does not, by itself, imply a priority, precedence, or ordering of a claim element over other claim elements, or the temporal order in which method operations are performed, but rather is merely used as a label to distinguish a claim element having a particular name from other elements of the same name (absent the use of the ordinal number).

[0090] Although the invention has been described by examples of preferred embodiments, it will be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications which come within the true spirit and scope of the invention.

Claims

1. 1. A system for signal analysis, comprising: and an interface configured to receive and transmit signals, the signals including a set of data sets including a set of multidimensional signals associated with a set of task labels, the system further comprising: A probabilistic deep neural network (DNN) block configured to identify the set of task labels from the set of multidimensional signals via a set of probabilistic nodes, the set of probabilistic nodes being individually specified by a set of irregular beliefs for posterior, prior and likelihood distributions quantified by a set of discrepancy measures, the system further comprising: a memory bank configured to store the set of datasets, the set of irregular beliefs, a set of hyperparameters, and a set of trainable parameters; and a processor, associated with the interface and the memory bank, configured to perform probabilistic inference to analyze the set of datasets by using the probabilistic DNN block, the probabilistic inference using importance weighted accumulation after variational sampling at the set of probabilistic nodes according to the set of irregular beliefs.

2. The probabilistic DNN block includes variations of variational autoencoders, variational information bottlenecks, denoising diffusion probability models, and variational Bayesian neural networks, and the probabilistic DNN block: Transformation layers, such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, implicit layers, attention layers, pooling layers, padding layers, initiation layers, and transformer layers; Interconnections such as multi-loop connections, skip addition, skip multiplication, and skip concatenation; Nonlinear activations such as rectified linear, sigmoid, shrinkage, softmax, gated linear, soft plus, and soft sine; and 2. The system of claim 1 , configured with a combination of regularization layers, such as dropout, swap out, zone out, block out, drop connect, shaking, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

3. The system of claim 1 , wherein the variational sampling uses a random number generator based on a reparameterization trick according to a set of variational parameters including position, scale, shape, and temperature specified by the set of irregular beliefs.

4. The system of claim 1 , wherein the set of discrepancy measures includes Renyi's alpha divergence, beta divergence, gamma divergence, Fisher divergence, Jeffrey divergence, and combinations of variants thereof.

5. The set of hyperparameters is A set of training hyperparameters that determine the scheduling policy, such as learning rate, weight decay, gradient momentum, divergence order, batch size, number of epochs, early stopping, dropout rate, and regularization weights; A set of architecture hyperparameters, such as connectivity for determining the probabilistic DNN blocks, type of layer operation, type of activation function, size of the set of trainable parameters, selection of probabilistic nodes, number of layer depths, and set of layer widths; and a set of evolution hyper-parameters, such as an initialization policy for the set of trainable parameters and a quantization policy for the set of trainable parameters.

6. The set of irregular beliefs may include: Discrete univariate distributions such as Bernoulli, binomial, geometric, Pascal, hypergeometric, Waring Yule, Poisson, Skellam, discrete uniform, Zipf, logarithmic series, Benford, Polya, Tanner, and Poisson Consul distributions; Normal-related distributions such as the normal distribution, log-normal distribution, Johnson distribution, half-normal distribution, skew-normal distribution, Voigt distribution, Tsallis distribution, hyperbolic distribution, variance gamma distribution, Student-t distribution, Chi distribution, Rayleigh distribution, Maxwell distribution, Chi-squared distribution, F-ratio distribution, Fisher distribution, Hotelling distribution, and Tracy distribution; Exponential-related distributions such as the exponential, Laplace, Lindley, logistic, sech, exponential power, shifted Gompertz, Erlang, hypoexponential, Coxian, gamma, Nakagami, Moyal, and Meixner distributions; Extreme value distributions such as the max stable, min stable, Gumbel, Frechet, and Weibull distributions; Bounded distributions such as the uniform, arcsine, Bates, Kumaraswamy, power, triangular, uniform sum, Mises, Wigner, and Pastur distributions; Heavy-tailed distributions such as the Pareto, Beta of the second kind, Dagum, Davis, Singh Maddala, Cauchy, Landau, Levy, Benini, Gibrat, and Suzuki distributions; Quantile-based distributions such as Tukey distribution, Wakeby distribution; Systematic distributions such as the Pearson distribution; Multivariate continuous distributions, such as the multivariate normal distribution, the multinomial distribution, the multivariate t-distribution, the Dirichlet distribution, and the Copula distribution; and 13. The system of claim 1, comprising a mismatched or heterogeneous combination of multivariate discrete distributions, such as a multivariate hypergeometric distribution, a negative multinomial distribution, and a multivariate Poisson distribution.

7. The set of data sets is Media data such as images, photographs, film, text, characters, voices, music, audio and speeches; Physical data, such as radio waves, optical signals, electrical pulses, temperature, pressure, acceleration, speed, vibration, and force; and 13. The system of claim 1, comprising a combination of physiological data such as heart rate, blood pressure, mass, hydration, electroencephalogram, electromyogram, electrocardiogram, mechanomyogram, electrooculogram, galvanic skin response, magnetoencephalogram, and electrocorticography.

8. The processor further comprises: configuring the probabilistic DNN block by modifying connectivity between the set of multidimensional signals, the set of task labels, and the set of probabilistic nodes according to the set of hyper-parameters; Calculating a loss function by forward propagating the set of data sets through the probabilistic DNN block according to the probabilistic reasoning; modifying the loss function by regularizing the set of probabilistic nodes according to the set of discrepancy measures; backpropagating the gradient of the loss function with respect to the set of trainable parameters through the probabilistic DNN block; and updating the set of trainable parameters of the probabilistic DNN block with a gradient method.

9. 9. The system of claim 8, wherein the loss function is a variational Renyi bound, evidence lower bound, mean squared error, cross entropy, structural similarity, cosine similarity, negative log-likelihood, absolute error, cross-covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, triplet loss, and combinations thereof.

10. 9. The system of claim 8, wherein the gradient method uses stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, root mean square propagation, and combinations of variants thereof.

11. 10. The system of claim 8, wherein the processor further performs the step of exploring different values ​​for the set of irregular beliefs and the set of hyper-parameters in the memory bank by using a supergradient method, the supergradient method using combinations of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, Bayesian optimization, and variations thereof.

12. The system of claim 1 , wherein the probabilistic DNN block uses heterogeneous assignment of different beliefs to sets of at least two disjoint probabilistic nodes in a layer.

13. The system of claim 1 , wherein the probabilistic DNN block uses mismatched pairs of posterior and prior beliefs for at least one set of probabilistic nodes.

14. 2. The system of claim 1, wherein the set of irregular beliefs is adaptively modified to use different probability distributions for the variational sampling in the set of probabilistic nodes within the probabilistic DNN block when analyzing a newly available set of datasets.

15. 1. A computer-implemented method for data analysis, comprising: providing a set of data signals to a probabilistic deep neural network (DNN), the probabilistic DNN including a set of layers and a set of probabilistic nodes, the method further comprising: Propagating the set of data signals through the set of layers in the probabilistic DNN, the probabilistic DNN using variational sampling in the set of probabilistic nodes according to a set of irregular beliefs, the method further comprising: A computer-implemented method comprising accumulating outputs of the probabilistic DNN as probabilistic inference results.

16. The probabilistic DNNs include variations of variational autoencoders, variational information bottlenecks, denoising diffusion probability models, and variational Bayesian neural networks, and the probabilistic DNNs include: Transformation layers, such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, implicit layers, attention layers, pooling layers, padding layers, initiation layers, and transformer layers; Interconnections such as multi-loop connections, skip addition, skip multiplication, and skip concatenation; Nonlinear activations such as rectified linear, sigmoid, shrinkage, softmax, gated linear, soft plus, and soft sine; and 16. The method of claim 15, configured with a combination of regularization layers, such as dropout, swap out, zone out, block out, drop connect, shaking, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

17. The variational sampling uses a random number generator based on a reparameterization trick that follows a set of variational parameters including position, scale, shape, and temperature specified by the set of irregular beliefs, the set of irregular beliefs being: Discrete univariate distributions such as Bernoulli, binomial, geometric, Pascal, hypergeometric, Waring Yule, Poisson, Skellam, discrete uniform, Zipf, logarithmic series, Benford, Polya, Tanner, and Poisson Consul distributions; Normal-related distributions such as the normal distribution, log-normal distribution, Johnson distribution, half-normal distribution, skew-normal distribution, Voigt distribution, Tsallis distribution, hyperbolic distribution, variance gamma distribution, Student-t distribution, Chi distribution, Rayleigh distribution, Maxwell distribution, Chi-squared distribution, F-ratio distribution, Fisher distribution, Hotelling distribution, and Tracy distribution; Exponential-related distributions such as the exponential, Laplace, Lindley, logistic, sech, exponential power, shifted Gompertz, Erlang, hypoexponential, Coxian, gamma, Nakagami, Moyal, and Meixner distributions; Extreme value distributions such as the max stable, min stable, Gumbel, Frechet, and Weibull distributions; Bounded distributions such as the uniform, arcsine, Bates, Kumaraswamy, power, triangular, uniform sum, Mises, Wigner, and Pastur distributions; Heavy-tailed distributions such as the Pareto, Beta of the second kind, Dagum, Davis, Singh Maddala, Cauchy, Landau, Levy, Benini, Gibrat, and Suzuki distributions; Quantile-based distributions such as Tukey distribution, Wakeby distribution; Systematic distributions such as the Pearson distribution; Multivariate continuous distributions, such as the multivariate normal distribution, the multinomial distribution, the multivariate t-distribution, the Dirichlet distribution, and the Copula distribution; and 16. The method of claim 15, comprising mismatched or heterogeneous combinations of multivariate discrete distributions, such as multivariate hypergeometric distribution, negative multinomial distribution, and multivariate Poisson distribution.

18. moreover, computing a loss function based on a variational bound to regularize the set of probabilistic nodes according to a set of discrepancy measures; backpropagating the gradient of the loss function with respect to a set of trainable parameters; and updating the set of trainable parameters according to a gradient method using stochastic gradient descent, Adam, Ada gradients, Ada bounds, Nesterov accelerated gradients, root-mean-square propagation, and combinations of variants thereof.

19. 19. The method of claim 18, wherein the set of discrepancy measures includes combinations of Renyi's alpha divergence, beta divergence, gamma divergence, Fisher divergence, Jeffrey divergence, and variants thereof, and the loss functions are combinations of variational Renyi bounds, evidence lower bounds, mean squared error, cross entropy, structural similarity, cosine similarity, negative log-likelihood, absolute error, cross-covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, triplet loss, and variants thereof.

20. 20. The method of claim 18, further comprising searching for different values ​​for the set of irregular beliefs according to a supergradient method, the supergradient method using combinations of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, Bayesian optimization, and variations thereof.

Images