Automated variational inference using probabilistic models with irregular beliefs

Abstract

A system and method for automatic construction of probabilistic deep neural network (DNN) architectures are provided. The framework of the present invention automatically explores the most relevant probabilistic modes of the basis of a dataset for variational Bayesian inference. The present invention provides a method for using heterogeneous, irregular, and mismatched beliefs in probabilistic sampling for intermediate representations in DNNs, with the ability of automatic tuning mechanisms of posterior, prior, and likelihood models to enable accurate generative and uncertainty models for machine learning tasks. The system further enables an adjustable discrepancy measure to regularize the intermediate representations by variants of divergence metrics, including Renyi's alpha, beta, and gamma divergence. The present invention enables diverse mixed combinations of probabilistic models for misspecified and unspecified probabilistic relationships in an automatic manner. Thus, the representation capabilities of variational autoencoders, variational information bottlenecks, denoising diffusion probability models, and other probabilistic DNNs are improved.

Description

[Technical Field]

[0001] The present invention relates to an automated learning system for artificial neural networks, and more particularly to an automated Bayesian inference system based on probabilistic neural networks with irregular beliefs. [Background technology]

[0002] Significant advances in deep learning technology based on deep neural networks (DNNs) have solved a variety of problems in data processing, including media signal processing for video, audio, and images; physical data processing for radio waves, electrical pulses, and light beams; and physiological data processing for heart rate, body temperature, and blood pressure. However, DNN architectures are often manually created with insights from experts who know the specific data model and structure. Optimizing DNN architectures requires time- and resource-consuming trial-and-error techniques. To address this issue, an automated machine learning (AutoML) framework has been used to automatically explore different DNN architectures. Automating hyperparameter and architecture search in the context of AutoML can facilitate DNN design suitable for specific data processing. AutoML includes architecture search, learning rule design, and augmented search. Most AutoML methods use either evolutionary optimization, hypergradient, or reinforcement learning frameworks to tune hyperparameters or build network architectures from a pre-selected selection of building blocks. A recent example, AutoML-Zero, considers augmentation to eliminate expert knowledge and insight for fully automated design from scratch.

[0003] Learning data representations that capture task-relevant features while being invariant to nuisance variations remains a key challenge in machine learning. A probabilistic DNN called a variational autoencoder (VAE) introduces a variational Bayesian inference method and incorporates an autocorrelated inference architecture, where generative and inferential models are jointly trained using a pair of decoder and encoder architectures. This method has been extended with a conditional VAE, which introduces conditioning variables that can be used to represent nuisance, and a regularized VAE, which considers separating nuisance variables from the latent representation. The concept of adversarial learning has been considered in generative adversarial networks (GANs) and has been adopted in countless applications. Concurrently discovered adversarial-trained inference (ALI) and bidirectional GANs (BiGANs) proposed adversarial approaches to training autoencoders. Adversarial training has also been combined with VAEs to regularize and separate latent representations, enabling nuisance-robot learning.

[0004] For Bayesian inference, VAEs are configured with parametric encoders and decoders to learn the latent variables underlying the data within a variational inference (VI) framework. Many variants of such probabilistic DNNs exist. For example, the β-VAE uses enhanced Kullback-Leiber divergence (KLD) to regularize the latent distribution more strongly than typical evidence lower bound (ELBO)-based losses. Continuous Bernoulli and beta distributions are explored as alternative likelihood beliefs for data reconstruction models. Laplace and Cauchy distributions are considered alternative prior beliefs for sparse latent representations. Normal posterior beliefs are adjusted by inverse autoregressive flows (IAFs), importance-weighted autoencoders (IWAEs), and Gibbs sampling. IWAE is further extended to variational Renyi (VR), which is based on Renyi's α-divergence. Generalized VI (GVI) then addresses arbitrary losses, divergences, and posterior selection. And it shows higher robustness with different discrepancy measures instead of KLD when data statistics are not specified.

[0005] While generalized VAEs offer significant degrees of freedom, their selection, in addition to other architectural hyperparameters, makes their selection difficult to engineer. Without a specific model specified to capture data statistics and probabilistic uncertainty, there is no framework for designing variational Bayesian inference and probabilistic DNNs. For example, probabilistic DNNs, including VAEs, variational information bottlenecks (VIBs), and denoising diffusion probability models (DDPMs), typically use homogeneous statistics for latent representations based on normal distributions. Normal distributions are computationally convenient because simple sampling, reparameterization tricks, and closed-form expressions for divergence are possible for gradient computation. However, when the underlying statistics of real-world datasets are not known a priori, normal distributions are generally no longer optimal.

[0006] However, AutoML suffers from the drawback of requiring a long search time to find the best hyperparameters due to search space explosion. Furthermore, without any good inference, the search space for most link connectivity becomes meaningless. To develop a system for automatically constructing accurate artificial neural networks, a method called AutoBayes has been proposed. The AutoBayes method represents the inherent graphical relationships between random variables for a generative model, then explores different Bayesian graphs to construct the most plausible inference graph, connecting the encoder, decoder, classifier, regressor, adversarial, and estimator. Using the so-called Bayes-ball algorithm, the most compact inference graph for a given Bayesian graph can be automatically constructed, and some factors are identified as variable-independent of the domain factors to be censored by the adversarial block. Adversarial censoring, which separates nuisance factors from the feature space, has been shown to be effective for domain generalization in pre-shot transfer learning and domain adaptation in post-shot transfer learning. However, adversarial training requires careful selection of hyperparameters, as too strong censoring will impair primary task performance due to the relatively low weighting of the primary objective function. Furthermore, adversarial censorship is not the only regularization technique for promoting independence from nuisance variables in the feature space.

[0007] Therefore, there is a need to efficiently identify the best probabilistic model for a probabilistic DNN that depends on the specific problem and dataset. Summary of the Invention

[0008] The present invention is based on the recognition that probabilistic deep neural networks (DNNs) use specific probability models, and that while there are many different probability models to impose, including normal, Laplace, Cauchy, logistic, Gumbel, Student-t, uniform, exponential, and hyperexponential distributions, the true probability model underlying real-world datasets is generally unknown. Additionally, there are unlimited possibilities for representing latent variables and data probabilities, as well as relevance measures such as divergence metrics in generalized variational inference (GVI). For example, there is no formal framework for determining the discrepancy measure to be selected from infinite possibilities, including Kullback-Leiber divergence (KLD), Renyi's alpha divergence, beta divergence, gamma divergence, Jensen-Shannon divergence, Jeffery divergence, and Fisher divergence, to regularize probabilistic nodes in DNNs. Furthermore, data uncertainty for identifying likelihood beliefs is also generally unspecified. For example, the normal distribution is typically used to minimize the weighted mean squared error (MSE) for regression problems, and the Bernoulli distribution is often used to minimize the binary cross entropy as the reconstruction loss for nearly binary pixel images.

[0009] The present invention enables probabilistic DNNs to efficiently identify data statistics and uncertainty models through the use of an AutoML framework for better combinations of prior beliefs, posterior beliefs, and likelihood beliefs and discrepancy measures. This provides benefits beyond the automated construction of DNN architectures such as DNN layer size, node size, activation type, and link connectivity. Specifically, the present invention provides a method for exploring irregular combinations of probabilistic models on top of traditional architecture hyperparameters. Some embodiments enable probabilistic DNNs to impose mismatched pairs of posterior beliefs and prior beliefs to increase the degrees of freedom of modeling capabilities under unspecified model uncertainty. Yet another embodiment provides a method for generalizing probabilistic DNNs to heterogeneous pairs of posterior beliefs and prior beliefs in different hidden nodes and hidden layers. In this way, more robust and complex latent representations can be achieved. Additionally, the present invention provides a method for automatically exploring irregular, mismatched, and heterogeneous assignments of discrepancy measures for individual probabilistic nodes. For example, the first latent node uses a logistic distribution as the posterior belief and a normal distribution as the prior belief using the KLD discrepancy measure, while the second latent node uses a Cauchy distribution as the posterior belief and a uniform distribution as the prior belief using Renyi's alpha divergence of order 0.4. Such mixed heterogeneous probabilistic models can improve generality across different datasets subject to unknown uncertainty. Some embodiments use variants of reparameterization tricks, such as the Gumbel softmax trick, for latent representations in VAEs, VIBs, DDPMs, and their variants in the context of network architecture search to span a categorical search space across different beliefs. Prior beliefs, posterior beliefs, and likelihood beliefs include, but are not limited to, normal, Cauchy, logistic, Laplace, uniform, triangular, Gumbel, exponential, generalized Gaussian, beta, gamma, Poisson, Bernoulli, and the like.Additionally, different censorship methods, such as adversarial disentanglement, can encourage latent representations in probabilistic DNN models to be independent of nuisance parameters, resulting in nuisance-robust feature extraction for some embodiments.

[0010] The present invention provides a method for tuning these hyperparameters and irregular beliefs under an AutoML framework based on hypergradient methods such as Bayesian optimization, implicit gradient, reinforcement learning, and heuristic optimization. Yet another embodiment uses mismatched posterior-prior pairing and heterogeneous distributions in probabilistic DNNs. For example, the posterior beliefs of a VAE encoder use a logistic distribution, and the prior beliefs of a VAE decoder use a normal distribution for probabilistic latent variables. The present invention recognizes that smaller divergence is not always better due to model mismatch, and suggests that heterogeneous posterior-prior pairing has the potential to provide a better Bayesian model for the entire probabilistic DNN, especially when the uncertainty model is not specified in advance.

[0011] Another embodiment uses multiple intermediate representations for variational sampling to improve model accuracy. In this case, the number of combinations to select for each latent layer rapidly increases, making automated search important. Additionally, for some embodiments, the present invention provides a method for enabling irregular, non-uniform posterior-prior pairing across all latent variables. One embodiment implements an ensemble method for exploring stacks of protocols over cross-validation during AutoML. The present invention provides a method for automatically generating auxiliary models that directly control the parameters of the base inference model by analyzing consistent evolutionary behavior in the main DNN model. In some embodiments, the type of divergence is also automatically selected in a non-uniform and non-uniform manner. For example, Renyi's alpha divergence, beta divergence, gamma divergence, Wasserstein distance, and their orders are jointly explored to be robust to numerically intractable data statistics encountered in the real world.

[0012] This disclosure relates to a system and method for automatically constructing probabilistic DNNs through the exploration of different uncertainty models and hyperparameters. Specifically, the system introduces an automated variational Bayesian inference framework that explores different posterior / prior / likelihood / discrepancy sets for a variational inference model linking classifier, encoder, decoder, and estimator blocks to optimize a nuisance-invariant machine learning pipeline. In one embodiment, this framework is applied to a series of physiological datasets, where we have access to subject and class labels during training, and provide an analysis of its capabilities for variational modeling and subject transfer learning with and without adversarial training. This framework can be effectively utilized for semi-supervised multiclass classification, multidimensional regression, and data reconstruction tasks across a variety of dataset types, such as media signals, electrical signals, and biological signals.

[0013] Some embodiments of the present disclosure are based on the recognition that a new concept called AutoBayes, which explores various different Bayesian graph models to facilitate the search for the best inference strategy, is suitable for an inference system robust to nuisances. Using the Bayes-Ball algorithm, the method and system of the present invention can automatically construct plausible link connections between classifiers, encoders, decoders, nuisance estimators, and adversarial DNN blocks, as well as different belief combinations for probabilistic nodes. Due to the no-free-lunch theorem, using one specific model without exploring its variations can potentially suffer from poor inference results. Additionally, the best model for one dataset does not always perform best on different data, which encourages the use of AutoML frameworks for adaptive model generation given a target dataset. One embodiment extends the AutoBayes framework to integrate probabilistic DNNs with the underlying unspecified / misspecified uncertainty of a dataset by exploring different irregular belief sets for posteriors, priors, likelihoods, and discrepancies.

[0014] Another embodiment uses variational sampling for semi-supervised settings when the dataset contains missing target labels. Yet another embodiment uses ensemble stacking, which combines estimates from multiple different Bayesian models to improve performance. Another embodiment uses a probabilistic graph neural network to exploit the geometric information of the data, and the pruning strategy is assisted by belief propagation across the Bayesian graph to verify relevance. Wasserstein distance can also be used instead of divergence. In some embodiments, the present system and method can be combined with existing test-time online adaptation techniques from zero-shot and few-shot learning frameworks to achieve even better performance that is robust to nuisances. The present system can offer the advantage of learning nuisance-invariant representations by exploring various regularization modules and uncertainty models. Yet another embodiment uses quantum and molecular devices to sample probabilistic nodes by exploiting the randomness of the devices. Some embodiments use a reversible network architecture to simultaneously model the encoder and decoder without having two disjoint models. Some embodiments use probabilistic implicit layers, such as neural ordinary differential equations, convex optimization, deep equilibrium, diffusion models, and quantum mechanics, embedded in DNN architectures. Thus, the present invention provides a method for designing Bayesian machine learning models, where an unknown, uncertain basis dataset is automatically modeled with a certain robustness by exploring various combinations of prior beliefs, posterior beliefs, and likelihood beliefs in probabilistic representation learning, such as VAE, DDPM, and VIB, under incomplete and misspecified knowledge of data statistics.

[0015] Further, according to some embodiments of the present invention, a system for data analysis using a probabilistic DNN block is provided. In this case, the system for signal analysis includes an interface, a probabilistic deep neural network (DNN) block, a memory bank, and a processor. The interface is configured to receive and transmit signals, such as a dataset, which is a multidimensional signal associated with a task label. The probabilistic DNN block is used to identify the task label from the multidimensional signal via probabilistic nodes, each of which is individually specified by irregular beliefs and discrepancy measures for posterior distributions, prior distributions, and likelihood distributions. The memory bank is used to store the dataset, irregular beliefs, other hyperparameters, and trainable parameters to determine the probabilistic DNN block. The processor, in association with the interface and the memory bank, performs probabilistic inference to analyze the dataset by using the probabilistic DNN block. The probabilistic inference is achieved by importance-weighted accumulation after variational sampling at the probabilistic nodes according to the irregular beliefs. Probabilistic DNN blocks include variants of VAE, VIB, and DDPM, which are composed of combinations of transformation layers, interconnections, nonlinear activations, and regularization layers. Variational sampling uses a random number generator based on a reparameterization trick subject to variational parameters including position, scale, shape, and temperature specified by random beliefs. Discrepancy measures include combinations of Renyi's alpha divergence, beta divergence, gamma divergence, Fisher divergence, and Jeffrey divergence.

[0016] Irregular beliefs include mismatched or heterogeneous combinations of discrete univariate distributions, normal-related distributions, exponential-related distributions, extreme value distributions, bounded distributions, heavy-tailed distributions, quantile-based distributions, systematic distributions, multivariate continuous distributions, and multivariate discrete distributions. The probabilistic DNN block uses heterogeneous assignment of different beliefs for at least two disjoint sets of probabilistic nodes in the same layer. Furthermore, the probabilistic DNN block uses mismatched pairs of posterior beliefs and prior beliefs for at least one set of probabilistic nodes. At least two different types of uncertainty models for the posterior beliefs, prior beliefs, and likelihood beliefs are simultaneously imposed at the disjoint probabilistic nodes. Furthermore, different discrepancy measures, such as KLD and Renyi's alpha divergence, are simultaneously imposed at the probabilistic DNN block to regularize the probabilistic nodes. The irregular beliefs can also be flexibly adjusted to use different probability distributions for variational sampling at the probabilistic nodes when analyzing a newly available set of datasets.

[0017] In some embodiments, the system uses the steps of exploring different values ​​for irregular beliefs and hyperparameters by using a supergradient method; constructing a probabilistic DNN by modifying the connectivity between multidimensional signals, task labels, and probabilistic nodes; calculating a loss function by forward propagating the dataset through the probabilistic DNN blocks according to probabilistic inference; modifying the loss function by regularizing the probabilistic nodes according to a discrepancy measure; backpropagating the gradient of the loss function with respect to the trainable parameters; and updating the trainable parameters using a gradient method.

[0018] By allowing irregular beliefs in the probabilistic DNN block, more complex model descriptions for unknown statistics of real-world datasets can be realized. Thus, by exploring irregular combinations of posterior, prior, and likelihood distributions in the probabilistic DNN block, our system provides better variational inference results even when the real-world dataset is not intractable for any closed-form statistical model.

[0019] Furthermore, some embodiments of the present invention provide a computer-implemented method for variational inference using a probabilistic DNN. Specifically, the variational inference method includes: feeding a data signal to a probabilistic DNN; propagating the signal through layers in the probabilistic DNN by employing variational sampling according to irregular beliefs at the probabilistic nodes; and accumulating the output of the probabilistic DNN as a probabilistic inference result. Additionally, a method for automatically designing a variational inference model includes calculating a loss function based on variational bounds to regularize the probabilistic nodes according to a discrepancy measure; backpropagating the gradient of the loss function with respect to trainable parameters; updating the trainable parameters according to a gradient method; and searching for different values ​​for the irregular beliefs according to a supergradient method. The gradient method uses a combination of stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, root-mean-square propagation, and variations thereof. Supergradient methods use a combination of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, Bayesian optimization, and variations thereof.

[0020] The accompanying drawings, which are included to provide a further understanding of the invention, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. [Brief explanation of the drawings]

[0021] [Figure 1A] 1 illustrates an exemplary inference model for predicting Y given data X, according to some embodiments. [Figure 1B]1 illustrates an exemplary inference model for predicting Y given data X under latent variables Z and nuisance factors S, according to some embodiments. [Figure 1C] 1 illustrates an exemplary inference model for predicting Y given data X under latent variables Z and nuisance factors S, according to some embodiments. [Figure 2] 1 illustrates an exemplary schematic diagram of an automated VAE, according to some embodiments. [Figure 3] 10 illustrates an exemplary variation of irregular beliefs, according to some embodiments. [Figure 4A] 1 illustrates exemplary beliefs for posterior, prior, and likelihood distributions used for probabilistic DNNs, according to some embodiments. [Figure 4B] 1 illustrates an exemplary family of standard location scale distributions used for variational sampling based on the reparameterization trick, according to some embodiments. [Figure 5A] 1 illustrates an exemplary landscape of KLD disagreements for various pairs of posterior and prior beliefs at two independent probabilistic nodes, according to some embodiments. [Figure 5B] 1 illustrates an exemplary pair of posterior and prior beliefs quantified by a KLD discrepancy measure, according to some embodiments. [Figure 5C] 10 illustrates exemplary KLD properties for various pairs of posterior and prior beliefs as a function of position value, according to some embodiments. [Figure 6] 1 illustrates an exemplary discrepancy measure based on Renyi's alpha-divergence that covers variants of generalized variational inference, including importance-weighted autoencoders (IWAEs) and standard VAEs, according to some embodiments. [Figure 7] FIG. 1 illustrates an example likelihood belief P of a probabilistic DNN for determining a generalized negative log-likelihood loss, according to some embodiments. [Figure 8A] 1 illustrates exemplary properties of various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and discrepancy measures for a generalized VAE model, according to some embodiments. [Figure 8B] 10 illustrates exemplary ELBO properties for various pairs of posterior and prior beliefs as a function of divergence order for a generalized VAE model, according to some embodiments. [Figure 8C] 10A-10C show example image snapshots generated by a VAE decoder generalized with variational sampling according to prior beliefs after being trained on various pairs of posterior and prior beliefs, according to some embodiments. [Figure 8D] 10A-10C show example image snapshots generated by a VAE decoder generalized with variational sampling according to prior beliefs after being trained on various pairs of posterior and prior beliefs, according to some embodiments. [Figure 9A] 1 illustrates an exemplary probabilistic DNN that sequentially uses multiple different beliefs and disagreement measures in multiple hidden layers, according to some embodiments. [Figure 9B] 1 illustrates an exemplary probabilistic DNN that uses different belief and disagreement measures at different probabilistic nodes, in parallel within one hidden layer, and sequentially across multiple hidden layers, according to some embodiments. [Figure 10] 1 illustrates an exemplary schematic diagram of a system comprising a processor, memory, and an interface, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0022] Various embodiments of the present invention will now be described with reference to the drawings. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by similar reference numerals throughout the drawings. It should also be noted that the drawings are intended only to facilitate the description of particular embodiments of the present invention. They are not intended as an exhaustive description of the invention or as limitations on the scope of the invention. Furthermore, aspects described in connection with a particular embodiment of the present invention are not necessarily limited to that embodiment and may be implemented in any other embodiment of the present invention.

[0023] The present invention provides systems and methods for analyzing signals in a dataset based on a generalized variational inference (GVI) framework, using an underlying probabilistic model of the dataset as an artificial intelligence (AI) pipeline. FIG. 1A illustrates an exemplary inference model for predicting a task label Y given data X, according to some embodiments. The task label is either a categorical identification number or a non-categorical continuous value. In the case of categorical inference, the AI ​​model performs a classification task, and in the case of non-categorical inference, the AI ​​model performs a regression task. In some embodiments, the task label is a mixture of multiple categorical and continuous labels for multi-task inference. In the case of supervised learning, the task label is either a scalar value or a vector of multiple values. In the case of unsupervised learning, the task label is not explicitly provided or, in some embodiments, is equivalent to the input data for reconstruction. The data in the dataset is in tensor format with at least one axis to represent multiple signals and sensor measurements, including, but not limited to, media data such as images, photographs, movies, text, characters, voices, music, audio, and speeches; physical data such as radio waves, optical signals, electrical pulses, temperature, pressure, acceleration, speed, vibration, and force; and physiological data such as heart rate, blood pressure, mass, hydration, electroencephalogram, electromyogram, electrocardiogram, mechanomyogram, electrooculogram, galvanic skin response, magnetoencephalogram, and electrocorticography.

[0024] For example, an AI model predicts emotions from a person's electroencephalogram (EEG) measurements, where the data is a triaxial tensor representing a spatiotemporal spectrogram from a multi-channel sensor over a period of measurement. All available data signals with X and Y pairs are bundled together as a whole batch of data sets for training the AI ​​model, which are called training data or training datasets. In some embodiments, task labels Y are missing for part of the training dataset in semi-supervised learning, and the missing labels are predicted by the AI ​​model in self-supervised learning.

[0025] The AI ​​model can be realized by a deep neural network (DNN) model, whose architecture and behavior are specified by a set of hyperparameters. The set of hyperparameters includes, but is not limited to, training hyperparameters that determine scheduling policies such as learning rate, weight decay, gradient momentum, divergence order, batch size, number of epochs, early stopping, dropout rate, and regularization weight; architecture hyperparameters that determine the DNN architecture, such as connectivity, type of layer operation, type of activation function, size of trainable parameters, selection of probabilistic nodes, number of layer depths, and set of layer widths; and deployment hyperparameters, such as initialization and quantization policies for trainable parameters.

[0026] DNN models are typically based on multi-layer perceptrons that use a combination of cells including, but not limited to: transformation layers such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, implicit layers, attention layers, pooling layers, padding layers, initiation layers, and transformer layers; interconnections such as multi-loop connections, skip-add, skip-multiply, and skip-concatenation; nonlinear activations such as rectified linear transformations, sigmoid transformations, shrinkage transformations, softmax transformations, gated linear, soft-plus, and soft-sine; and regularization layers such as dropout, swap-out, zone-out, block-out, drop-connect, shaking, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

[0027] For example, the residual network architecture uses skip addition from one hidden layer to another, which allows for stable learning of deeper layers. A DNN model has several trainable parameters, such as affine transformation weights and biases.

[0028] The DNN model is trained on a training dataset to minimize or maximize an objective function (also called a loss function or utility function) using gradient methods, including but not limited to stochastic gradient descent, Adam gradient, root-mean-square propagation, ada gradient, ada delta, ada max, ada bound, Nesterov accelerated gradient, elastic backpropagation, and weighted Adam. In some embodiments, the training dataset is divided into multiple sub-batches for local gradient updates. In some embodiments, a portion of the training dataset is reserved for a validation dataset to evaluate the performance of the DNN model. In some embodiments, the validation dataset from the training dataset is rotated for cross-validation. Methods for dividing the training data into sub-batches for cross-validation include, but are not limited to, random sampling; weighted random sampling; one session reservation; one subject reservation; and one region reservation. Typically, the data distribution of each sub-batch is not identical due to domain shifts.

[0029] Gradient-based optimization algorithms have several hyperparameters, such as learning rate and weight decay. The learning rate is an important parameter to select and can be automatically adjusted by several scheduling policies, such as step functions, exponential functions, trigonometric functions, and adaptive decay to plateaus. Other optimization methods, such as evolutionary strategies, genetic algorithms, differential evolution, simulated annealing, particle swarms, Bayesian optimization, and Nelder-Mead, can also be used to optimize trainable parameters. The objective function is a combination of various functions, including but not limited to L1 loss; Lp norm; mean squared error; cross-entropy; connectionist time series classification loss; negative log-likelihood; Kullback-Leiber divergence (KLD); cross-covariance; structural similarity; cosine similarity; clustering loss; margin ranking loss; hinge loss; Huber loss; negative sampling; Waserstein distance; and triplet loss.

[0030] AI models without hidden node guidance often suffer from local minimum trapping due to over-parameterized DNN architectures to solve task problems. Several regularization techniques are used to stabilize training convergence. For example, the L1 / L2 norm is used to regularize affine transformation weights. Batch normalization and dropout techniques are also widely used to prevent overfitting. Other regularization techniques include, but are not limited to, drop connect, drop block, drop pass, shake drop, spatial drop, zone out, stochastic depth, stochastic width, spectral normalization, and shake-shake. However, these well-known regularization techniques do not utilize underlying data statistics. Most datasets have specific probabilistic relationships between X and Y and potential nuisance factors S that interfere with task prediction performance. For example, physiological datasets such as EEG signals are highly dependent on the subject's mental state and measurement conditions, which are treated as nuisance factors S. Nuisance variations include, but are not limited to, subject identification, session number, biological state, environmental state, sensor state, sensor location, sensor orientation, sampling rate, time, and sensitivity set. In yet another example, electromagnetic datasets such as Wi-Fi signals are susceptible to room environment, surrounding users, interference, and hardware imperfections. This disclosure provides a method for efficiently regularizing DNN blocks by considering nuisance factors so that AI models are insensitive to domain shifts caused by changes in those factors.

[0031] The present disclosure is based on a probabilistic DNN that imposes a probabilistic model with hidden layers and latent variables, and the nodes are called probabilistic nodes. Specifically, the hidden layers are randomly sampled from a specific distribution family, such as a normal distribution, and their variational parameters, such as a location factor (or mean) and a scale factor (or standard deviation), are defined by the hidden layers of the probabilistic DNN. To make the variational parameters differentiable, a reparameterization trick is used, in which the random variables sampled from the standard distribution are shifted and scaled so that they are distributed with the desired distribution of the latent variables. To regularize the probabilistic nodes, some embodiments modify the loss function based on discrepancy measures, including but not limited to KLD; Renyi's alpha divergence; beta divergence; gamma divergence; Jeffery divergence; Fisher divergence; Jensen-Shannon divergence; and Wasstern distance.

[0032] In the GVI framework, the target distribution is specified as the prior belief, and the sample distribution is specified as the posterior belief. Additionally, task labels and data predictions are also probabilistically defined under a specific uncertainty model to determine likelihood beliefs. Likelihood beliefs include, but are not limited to, normal distributions; Bernoulli distributions; unspecified normal distributions; continuous Bernoulli distributions; Laplace distributions; Cauchy distributions; beta distributions; unspecified Lapalce distributions; and gamma distributions. Given specific likelihood beliefs, the corresponding negative log-likelihood (NLL) is used to minimize the objective function of the probabilistic DNN. For example, variational autoencoders (VAEs) use encoder and decoder models to infer latent variables and generative models, respectively. Standard VAEs use a mean squared error (MSE)-based reconstruction loss under the assumption of likelihood beliefs based on an unspecified normal distribution with undefined variance. Additionally, latent variables are typically sampled from normal distributions for both the posterior and prior beliefs. Some embodiments of the present invention enable irregular heterogeneous pairing of posterior and prior beliefs to provide additional degrees of freedom for unknown data uncertainty modeling. Yet another embodiment uses irregular heterogeneous discrepancy measures at individual probabilistic nodes in the DNN model to be robust to misspecified uncertainty. In addition to VAEs, probabilistic DNN models in some embodiments include, but are not limited to, variational information bottlenecks (VIBs), denoising diffusion probability models (DDPMs), and variational Bayesian neural networks.

[0033] The present invention uses random beliefs about the posterior, prior, and likelihood distributions, and their discrepancy measures, to determine statistics of probabilistic nodes in a probabilistic DNN block.Irregular beliefs include discrete univariate distributions such as the Bernoulli distribution, binomial distribution, geometric distribution, Pascal distribution, hypergeometric distribution, Waring-Yule distribution, Poisson distribution, Skellam distribution, discrete uniform distribution, Zipf distribution, logarithmic series distribution, Benford distribution, Polya distribution, Tanner distribution, and Poisson-Consul distribution; normally-related distributions such as the normal distribution, lognormal distribution, Johnson distribution, half-normal distribution, skew-normal distribution, Voigt distribution, Tsallis distribution, hyperbolic distribution, variance gamma distribution, Student-t distribution, chi distribution, Rayleigh distribution, Maxwell distribution, chi-squared distribution, F-ratio distribution, Fisher distribution, Hotelling distribution, and Tracy distribution; and exponential distribution, Laplace distribution, Lindley distribution, logistic distribution, sech distribution, exponential power distribution, shifted Gompertz distribution, Erlang distribution, hypoexponential distribution, Coxian distribution, gamma distribution, Nakagami distribution, Moyal Exponentially related distributions such as the Meixner distribution; extreme value distributions such as the max stable, min stable, Gumbel, Frechet, and Weibull distributions; bounded distributions such as the uniform, arcsine, Bates, Kumaraswamy, power, triangular, uniform sum, Mises, Wigner, and Pastur distributions; Pareto, second kind beta, Dagum, Davis, and Singh distributions. This includes mismatched or heterogeneous combinations of various distributions, including, but not limited to, heavy-tailed distributions such as the Maddala, Cauchy, Landau, Levy, Benini, Gibrat, and Suzuki distributions; quantile-based distributions such as the Tukey and Wakeby distributions; systematic distributions such as the Pearson distribution; multivariate continuous distributions such as the multivariate normal, multinomial, multivariate t, Dirichlet, and Copula distributions; and multivariate discrete distributions such as the multivariate hypergeometric, negative multinomial, and multivariate Poisson distributions.

[0034] For example, a probabilistic DNN block may use heterogeneous assignment of different beliefs for at least two disjoint sets of probabilistic nodes in one layer; for example, the first 70% of the probabilistic nodes in a second layer may follow a logistic distribution, while the remaining 30% of the probabilistic nodes in the same layer may follow a Cauchy distribution. In another example, a probabilistic DNN block may use mismatched pairs of posterior beliefs and prior beliefs for at least one set of probabilistic nodes; for example, a probabilistic node in a third layer may use a uniform distribution for the posterior beliefs, while the same probabilistic node uses a Laplace distribution for the prior beliefs. Additionally, in some embodiments, the selection of irregular beliefs is adaptively modified on the fly so that, even though the probabilistic DNN block was previously trained with different irregular beliefs, different probability distributions are used for variational sampling in the probabilistic nodes within the same probabilistic DNN block when analyzing a newly available set of data. For example, a probabilistic node may be trained to follow a normal distribution for a training data set, but the same probabilistic node may use a logistic distribution for a testing data set. This allows further adaptation to reduce model mismatch in unseen testing data sets.

[0035] The probabilistic DNN block provides probabilistic inference to analyze a dataset by using importance-weighted accumulation after variational sampling at probabilistic nodes according to irregular beliefs. Variational sampling uses a random number generator based on a reparameterization trick according to variational parameters such as location, scale, shape, and temperature specified by the irregular beliefs. To optimize the probabilistic DNN block, some embodiments perform the following steps: exploring different values ​​for the irregular beliefs and hyperparameters by using a supergradient method; configuring the probabilistic DNN block by modifying the connectivity between multidimensional signals, task labels, and probabilistic nodes according to the hyperparameters; calculating a loss function by forward-propagating the dataset through the probabilistic DNN block according to probabilistic inference; modifying the loss function by regularizing the probabilistic nodes according to the discrepancy measure; back-propagating the gradient of the loss function with respect to the trainable parameters; and updating the trainable parameters using a gradient method.

[0036] The steps mainly include two methods for inference and optimization. The inference method for a probabilistic DNN model involves feeding a data signal to a DNN with a certain number of layers and probabilistic nodes, propagating the signal along the layers while the probabilistic nodes use variational sampling according to irregular beliefs, and accumulating the DNN output as a probabilistic inference result. In addition to the inference method, the optimization method further involves calculating a loss function based on variational bounds to regularize the probabilistic nodes according to a discrepancy measure, backpropagating the gradient of the loss function with respect to trainable parameters, updating the trainable parameters using a gradient method, and exploring different values ​​for the irregular beliefs and hyperparameters using a supergradient method. The supergradient method enables the efficient construction of a probabilistic DNN model with higher performance than traditional variational inference models that rely on regular beliefs with homogeneous matched assignments for probabilistic nodes. Auxiliary Regularization Module

[0037] 1B and 1C show an exemplary inference model for predicting Y given data X under a latent variable Z and a nuisance factor S, according to some embodiments. The DNN model uses a main pipeline, which is decomposed into an encoder unit 101 and a classifier unit 102 (or a regressor unit for regression tasks). The encoder unit 101 extracts a feature vector as a latent variable Z from data X, and the classifier unit 102 predicts a task label Y from the latent variable Z. For example, the latent variable Z is a vector of hidden nodes in the intermediate layer of the DNN model. To stabilize the training convergence of the DNN model, some embodiments use a regularization technique for the latent variable Z. In addition to the main pipeline of the encoder 101 and the classifier 102, FIG. 1B uses an additional auxiliary regularization module to regularize the latent variable Z. Specifically, the decoder DNN block 103 reconstructs the original data X, appended with the latent variable Z, along with optional conditional information of the nuisance variation S. This conditional decoder 103 can facilitate the disentanglement of nuisance domain information S from latent variables Z. For example, for a physiological dataset, the nuisance domain variations S include subject identification (ID), measurement session ID, noise level, subject height / weight / age information, etc. By separating these nuisance factors S from the latent variables Z, the present invention, in some embodiments, can realize a subject-invariant universal human-machine interface without lengthy calibration sessions. An auxiliary DNN block called decoder 103 is trained to minimize the mean squared error and another loss term, such as a Gaussian negative log-likelihood loss, to recover X from Z.

[0038] In some embodiments, the latent variables are further denoted by Z1, Z2, ..., Z L Each of these factors is a nuisance factor S1, S2, ..., S NIn addition, some nuisance factors are partially known or unknown depending on the dataset. For known labels of nuisance factors, the DNN block can be trained in a supervised manner, while unlabeled nuisance factors require a semi-supervised method. In the semi-supervised case, some embodiments use pseudo-labeling based on variational sampling across all potential labels of the nuisance factors, which is based on a reparameterization trick, such as the Gumbel softmax trick, which uses a variational temperature parameter to control the sharpness of approximately one-hot samples for the classification nuisance factors. For continuous nuisance factors, variational sampling is based on a reparameterization trick that uses variational parameters including location, scale, and shape to determine a probability distribution. For example, some of the data in a dataset lacks subject age information, while the remaining data has age information to be used for supervised regularization.

[0039] The DNN block in FIG. 1B has another auxiliary regularization module 104 attached to the latent variables Z to estimate the nuisance variation S. This regularization DNN block 104 is used to further promote the disentanglement of nuisance factors and make them more robust, and is often referred to as an adversarial network because the main pipeline DNN block is trained to maximize a loss function, while the regularization DNN block is trained to minimize a loss function to estimate S from Z to censor the nuisance information. The adversarial block is trained in an alternating fashion with associated hyperparameters, including adversarial coefficients, adversarial learning rate, interval between adversarial alternations, and architecture specifications.

[0040] The graphical model in FIG. 1B is known as an adversarial conditional variational autoencoder (A-CVAE) for unsupervised feature extraction in conjunction with the downstream task classifier 102. The A-CVAE graphical model has various graph nodes and graph edges to represent connectivity covering the random variable factors X, Y, Z, and S. With the regularization block, the A-CVAE model has higher robustness against nuisance factors. Therefore, the auxiliary regularization module is used as a so-called pre-shot transfer learning or domain generalization technique to make the AI ​​model robust against unseen nuisance factors.

[0041] The present invention provides a method for efficiently assigning multiple distinct beliefs to individual probabilistic nodes Z1, Z2, ..., Zn. If there are 10 distinct posterior-prior pairs for 100 probabilistic nodes, then we can assign up to 10 distinct beliefs to specify a probabilistic DNN. 100 There are potential combinations. Some embodiments use a supergradient-based automated machine learning (AutoML) framework to facilitate the discovery of better combinations of such heterogeneous pairings of posterior / prior beliefs. In addition, some embodiments allow different discrepancy measures for individual probabilistic nodes. For example, Renyi's alpha divergence of order 0.1 is used for Z1, Z2 uses an alpha order of 0.2, and KLD is used for Z3. The present invention enables such irregular probabilistic models through the use of AutoML search, and the search space rapidly expands. AutoML's supergradient method efficiently searches for better combinations of irregular beliefs for posterior, prior, and likelihood distributions, as well as discrepancy measures, simultaneously for individual probabilistic nodes, based on a combination of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, and Bayesian optimization. Architecture Exploration

[0042] There are many possible ways to connect the encoder 101, classifier 102, decoder 103, and adversarial network block 104. For example, FIG. 1C shows one exemplary DNN block in which a separate auxiliary module 105 exists to estimate nuisance factors S from data X. Because the number of possible DNN connections rapidly explodes with the size of the AI ​​model, it is necessary to efficiently build reasonable AI models. Additionally, randomly connected DNN blocks tend to be useless and unreasonable. In some embodiments, graph connectivity is explored using an automated Bayesian graph search method called AutoBayes. At the heart of AutoBayes is the consideration of a graphical Bayesian model that captures probabilistic relationships among random variables representing data features X, task labels Y, nuisance variation labels S, and latent expressions Z. The primary goal is to infer task labels Y from measured data features X, which is hindered by the presence of nuisance variations (e.g., inter-subject / inter-session variations) that are partially labeled by S. The potential representation Z (and optionally Z1, Z2, ..., Z L ) is also optionally introduced into the AI ​​model to help capture the underlying relationships between S, X, and Y.

[0043]

number

[0044] The above graphical models do not impose any assumptions about the inherent independence of the dataset and are therefore the most comprehensive. However, depending on the underlying independence in the dataset, some edges in these graphs can be pruned. For example, if the data has a Markov chain of YX that is independent of S and Z, it automatically results in the AI ​​model shown in Figure 1A. This means that the most complex inference model with a high degree of freedom will not always perform best across any dataset. This encourages us to consider extended AutoML frameworks that automatically search for the best pair of inference factor graph models and corresponding Bayesian graph models that match the dataset, in addition to other hyperparameters.

[0045] AutoBayes begins by exploring any potential Bayesian graph by cutting links in the full-chain graph and imposing possible independence. We then employ the Bayesball algorithm on each hypothetical Bayesian graph to examine conditional independence across different inference strategies. The Bayesball algorithm justifies reasonable pruning of links in the full-chain inference graph and also justifies potential adversarial censorship when Z is independent of S. This process automatically builds good inference connectivity for the inference, generative, and adversarial blocks to build, for example, the A-CVAE classifier in Figure 1B, rather than the arbitrary extended model in Figure 1C. An example probabilistic DNN model

[0046] FIG. 2 shows an exemplary schematic diagram of an automated VAE according to some embodiments. Generalized variational inference (GVI) offers a large degree of freedom, but this, in turn, makes it difficult to design an appropriate choice of probabilistic model in addition to other architectural hyperparameters. To solve this problem, the present invention uses automatic construction of probabilistic DNN models. FIG. 2 shows such an example of an automated VAE, also called AutoVAE, which facilitates finding appropriate choices of posterior, prior, likelihood, and discrepancy measures by using a supergradient-based automated machine learning (AutoML) framework. While normal distributions are often used for potential sampling in the probabilistic nodes of standard VAEs, they are not always optimal when the underlying uncertainty in the dataset is complex or unknown. The present invention provides a method for improving VAE models by allowing irregular beliefs for probabilistic nodes. By exploring diverse settings of irregular beliefs, the variational Renyi (VR) bound can be improved by mismatch pairing, for example, when a logistic distribution is used as the posterior belief in the VAE encoder, even though the VAE decoder assumes a normal distribution as the prior belief.

[0047] The VAE uses an encoder block 201 and a decoder block 203 to reconstruct the data signal from the latent variables 202. The final node of the VAE encoder is a probabilistic node, which provides variational parameters to conform to a specific probability distribution given the data signal. Thus, the VAE encoder expresses a posterior belief, specified by variational parameters such as position, scale, and shape. For example, the first half of the output node of the VAE encoder is the position parameter, and the second half of the output node of the VAE encoder is the scale parameter. The latent variables 202 are then sampled by a random number generator, and the random numbers sampled from the standard distribution are scaled and transformed by the variational parameters generated by the VAE encoder according to the posterior belief through a reparameterization trick 205.

[0048] The VAE decoder takes sampled latent variables 202 and generates an estimated data signal. The final node of the VAE decoder is a probabilistic node, which provides variational parameters to follow a specific probability distribution given the latent samples. Thus, the VAE decoder expresses likelihood beliefs 208 specified by variational parameters such as location, scale, and shape. For example, the output node of the VAE decoder is the location parameter of a normal distribution, but the scale parameter is an unspecified constant; that is, the generalized NLL loss is reduced to the MSE loss.

[0049] The VAE encoder and decoder are jointly trained by gradient descent to maximize the evidence lower bound (ELBO), which is a function of the generalized NLL loss and the discrepancy 207 between the posterior and prior beliefs 206. The discrepancy term acts as a regularizer for the probabilistic nodes that control the probability distribution. Once the VAE encoder and decoder are trained, the VAE decoder can be used as a generative model to generate synthetic signals that are close to the original dataset by feeding random numbers sampled from the prior beliefs. Some embodiments of the present invention use adaptive adaptation of the prior beliefs to sample latent variables from different probability distributions between the training and testing datasets.

[0050] AutoVAE uses the AutoML framework 204 to explore various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and discrepancy measures by using a hypergradient method to achieve the best ELBO. The AutoVAE system enables various irregular pairings for probabilistic nodes, such as: i) matched pairings using the same probability distribution for posterior beliefs and prior beliefs; ii) mismatched pairings using different probability distributions for posterior beliefs and prior beliefs; and iii) heterogeneous pairings using different probability distributions for posterior beliefs and prior beliefs non-uniformly at individual probabilistic nodes. Figure 3 illustrates exemplary variations of irregular beliefs in accordance with some embodiments. The first variation 301 in Figure 3 is when posterior beliefs and prior beliefs are matched to be Laplace-distributed. The second variation 302 in Figure 3 is when posterior beliefs using Laplace distributions are mismatched with prior beliefs using normal distributions. The third variant 303 of Figure 3 is where the probabilistic nodes are heterogeneous, specifically, where the first half of the latent variable uses a different posterior-prior pair than the second half of the latent variable. The use of such irregular beliefs can significantly improve the expressive power of the probabilistic DNN model, thus offering the benefit of being able to efficiently model more complex statistics. In addition, standard KLD regularization can be significantly extended by exploring discrepancy measures across Renyi's alpha-divergence, beta-divergence, gamma-divergence, etc.

[0051] FIG. 4A shows exemplary beliefs for posterior, prior, and likelihood distributions used in probabilistic DNNs, according to some embodiments. The corresponding notations for various random distributions and their probability density function (PDF) representations are listed for convenience. Some distributions are specified by a single variational parameter, e.g., scale for exponential distributions. Some distributions are specified by two variational parameters, e.g., location and scale for Laplace distributions. In some embodiments, variational parameters with more than two shapes are also used for other distributions, including, but not limited to, skew-normal, exponential-power, hyperbolic, gamma, Johnson, Pearson, Student-TT, and Pareto distributions. Some distributions belong to the location-scale family (LSF), which is suitable for variational sampling because nonstandard distributions can be easily generated by standard distributions through the use of reparameterization tricks. For discrete probability distributions, such as the Gumbel distribution, a modified reparameterization trick called the softmax trick is used with a variational parameter called temperature to adjust the sharpness toward one-hot encoding samples.

[0052] As an example, x∈R N Let be the N-dimensional data input to the VAE encoder. The encoder generates variational parameters such as the position, scale, and shape of the posterior belief hypotheses to generate an L-dimensional latent variable z∈R L The latent variables are then input to a VAE decoder to generate variational parameters for the likelihood belief hypotheses, and the reconstructed data x'∈R N The encoder and decoder are constructed using parameterized DNNs, and are respectively φ :x→z and p ψ :z→x', where φ and ψ are the trainable parameters of the DNN, such as weights and biases. The DNN attempts to minimize the reconstruction loss, which would typically be the negative log-likelihood (NLL) under the likelihood-belief hypothesis.

[0053] For a given choice of parameters φ and ψ, the VAE encoder and decoder models, respectively, have the posterior q φ The conditional distribution known as (z|x) and likelihood p ψ This means the conditional distribution known as (x|z). If π(z) is the probability distribution of the latent variable z under the assumption of prior beliefs, then the VAE is

number

number

number

[0054] The VAE encoder and decoder are jointly trained such that the ELBO is maximized under a variational Bayesian framework.

[0055] In ELBO, there are four important factors: likelihood belief P=p ψ (x|z); posterior belief Q=q φ (z|x); prior belief Π = π(z); and discrepancy measure D = D KL(.||.) is specified. Standard VAEs often use a normal distribution (or a Bernoulli distribution for near-binary image reconstruction) for the likelihood beliefs P=N(λ,γ), and a specific KLD D KL We use (Q||Π) to regularize the latent variables by assuming a normal posterior Q = N(mu, sigma) and a normal prior Π = N(0, 1). Based on GVI, the present invention allows for arbitrary selection of the generalized NLL loss, discrepancy measure, prior, and posterior beliefs. While standard VAEs are asymptotically optimal for infinite dimensions, optimality is no longer justified when the likelihood / posterior / prior beliefs are misspecified or misspecified in contrast to the real data distribution.

[0056] While exponential family prior beliefs will result in the same exponential family posterior beliefs under certain conditions, the present invention provides an extended GVI by considering irregular beliefs, such as mismatched pairing of posterior-prior beliefs, e.g., logistic posterior and Laplace prior. Furthermore, the generalized VAE allows for alternative irregular beliefs using heterogeneous pairings, e.g., 30% of latent nodes use Laplace-normal pairings and 70% of nodes use uniform-Cauchy pairings. Some prior art considers non-Gaussian probabilities, such as Laplace and Cauchy distributions, but does not consider irregular beliefs with mismatched or heterogeneous pairings. To design such irregular VAEs, the present invention provides a convenient set of pairings with a closed-form differentiable representation and straightforward reparameterization of KLDs, without requiring the high complexity of information autoregressive flows (IAFs) and Gibbs sampling.

[0057] In the prior art, the normal distribution is the most widely used belief for probabilistic DNNs, allowing for simple sampling, reparameterization tricks, and closed-form expressions for KLD. In some embodiments of this disclosure, we consider various alternatives, such as the location-scale distribution family LSF(μ,σ), which retains the same distribution under transformations involving variational parameters for location μ∈R and scale σ∈R+. For example, given a random variable ε drawn as ε~LSF(0,1), its transformed random variable Z = μ + σ · ε follows the same family as Z~LSF(μ,σ). Here, the notation "~" means "distributed to." This transformation, known as the reparameterization trick for variational sampling, is one of the key methods that enables stochastic DNN training to backpropagate gradients of the variational parameters μ and σ while maintaining the target distribution in forward variational sampling.

[0058] As shown in the exemplary list in FIG. 4A , this disclosure considers a diverse set of beliefs, including LSF distributions: normal N; Laplace La; Cauchy C; logistic Lo; uniform U; Gumbel G; and exponential E (which is not an LSF, but is a scale distribution family). FIG. 4B shows an exemplary standard position scale distribution family used for variational sampling based on the reparameterization trick, according to some embodiments. Several PDFs of its standard form LSF(0,1) are plotted in FIG. 4B . These distributions are useful as a choice of potential posterior beliefs Q for generating a target non-standard distribution of LSF(μ,σ) 404 for a VAE encoder, due to a simple reparameterization trick 400 by transforming a standard random sample LSF(0,1) 403 with two variational parameters: position 401 and scale 402. For example, sampling of the logistic distribution can be done as Z = μ + σε ∼Lo(μ,σ), where V,W ∼E(1), for ε = log(V / W) ∼Lo(0,1). Position vector μ∈R L and the scale vector σ∈R L is generated from the data x by a parametric VAE encoder, and the latent z is a standard random variable ε~LSF(0,1)L It is stochastically sampled by transformation from

[0059] For the choice of prior belief Π = π(z) used in a generative model with a VAE decoder, we typically use a standard normal distribution N(0,1), or for a particular choice of posterior belief Q = LSF(μ,σ), a matched standard prior Π = LSF(0,1). The probabilistic nodes are then regularized to achieve a posterior distribution close to the prior distribution by a discrepancy measure such as KLD. FIG. 5A shows exemplary landscapes of KLD discrepancies for various pairs of posterior and prior beliefs at two independent probabilistic nodes, according to some embodiments. For example, the KLD landscape for a pair of normal posterior and normal prior beliefs 501 is circularly isotropically symmetric across any axis, penalizing samples with larger magnitudes to behave like a Gaussian distribution. For a pair of Laplace posterior and Laplace prior beliefs 502, the KLD landscape becomes anisotropic, with a rounded square shape, which facilitates sparse regularization. For another pair of Cauchy posterior and Cauchy prior beliefs 503, the KLD landscape is shaped like a rounded cross, which generally encourages even sparser regularization.

[0060] Although we can choose any distribution for the prior beliefs, regardless of the posterior beliefs, it is desirable to have a closed-form simple expression for the KLD to measure the discrepancy between the posterior Q and the prior Π. In some embodiments, we consider such pairs of posterior and prior beliefs. Figure 5B shows example pairs of posterior and prior beliefs quantified by a KLD discrepancy measure, according to some embodiments. In Figure 5B, closed-form expressions for the KLD discrepancy measure for several pairs of posterior and prior beliefs are listed for convenience. In some embodiments, the KLD can also be obtained by numerical integration via variational sampling and accumulation for other irregular beliefs.

[0061] Yet another embodiment uses adaptive prior modification on the fly after the VAE is trained. For example, the distribution of latent variables in the optimized VAE encoder may ultimately have a different distribution than the target prior. To solve the mismatch problem, the prior belief is modified when synthetically generating new data, depending on some statistics such as the mean, variance, and skewness of the latent variables that match the newly available dataset.

[0062] Matched posterior-prior pairings generally have smaller KLDs, but mismatched pairs are not necessarily worse than matched posterior-prior pairings. Figure 5C shows exemplary KLD characteristics for various pairs of posterior and prior beliefs as a function of position value, according to some embodiments. For example, mismatched pairs of logistic-Laplace beliefs have slightly smaller KLDs than matched pairs of Laplace-Laplace beliefs at moderate position values. More importantly, such mismatched cases have sufficiently small KLD values ​​for a wide range of positions μ. Because the KLD value cannot be exactly zero even for matched pairs unless μ = 0 and σ = 1, minimizing the KLD term is not necessarily important. This implies that exploring different irregular pairings may ultimately provide a better ELBO than traditional regular pairings, due to the greater degree of freedom in adjusting the KLD landscape and ELBO. The present invention is based on this recognition.

[0063] Note that KLD is the most commonly used discrepancy measure to evaluate the "difference" between the posterior Q and the prior Π for VAEs. However, besides KLD, probabilistic DNNs can use various other discrepancy measures, including Fisher, Jeffrey, and Renyi's α, β, and γ divergences. In particular, Renyi's α-divergence is useful because, as a special case in adjusting the divergence order alpha, it covers many variants, such as importance-weighted autoencoders (IWAEs) and standard VAEs. Figure 6 shows an example discrepancy measure based on Renyi's α-divergence, covering variants of generalized variational inference, including importance-weighted autoencoders (IWAEs) and standard VAEs, according to some embodiments. For example, when α → 1, the α-divergence D α (Q∥Π) is KLD D KL (Q∥Π). In another example, the alpha divergence reduces to a variant of the Hellinger distance and the chi-squared divergence. The Renyi divergence of order α≧0 can be expressed as:

number

[0064] More importantly, it has been shown that the ELBO based on the Renyi divergence has a tighter bound than the KLD case (α → 1). Specifically, the variational Renyi (VR) bound is given by the K-fold variational sample z as follows: k ~p φ It is approximated by an importance-weighted accumulation of (z|x):

number

[0065] The IWAE is a special case of the VR framework with α = 0, converging to the marginal probability as K → ∞. Because the α-divergence is closely related to the KLD, the aforementioned pair in FIG. 5B also has a closed-form expression for the α-divergence. Thus, some embodiments use the Renyi α-divergence as a generalized discrepancy measure, where the Renyi order alpha is adjusted to find the best ELBO.

[0066] For VAE, in practice, any differentiable loss measure can be used as the reconstruction loss, e.g., mean squared error (MSE), mean absolute error (MAE), and binary cross entropy (BCE), in addition to NLL. Nevertheless, most loss functions are based on a particular likelihood belief P=p ψ Under (x|z), we represent the generative model of data through a VAE decoder, closely related to a generalized NLL. Figure 7 shows an example of likelihood beliefs P for a probabilistic DNN that determines a generalized negative log-likelihood loss, according to some embodiments. Various likelihood beliefs P for probabilistic DNNs are listed in Figure 7, where generalized NLL losses are specified for different likelihood beliefs. For example, BCE is often used for nearly binary images, such as the MNIST dataset. BCE is equivalent to NLL under likelihood beliefs based on the Bernoulli distribution B(λ). Some embodiments use the beta distribution B as the appropriate likelihood belief, while some embodiments use its modified belief, called the continuous Bernoulli distribution C, to improve VAE for less strictly binary images. MSE corresponds to NLL under a normal distribution likelihood N(x',*) when omitting unspecified variances. MAE corresponds to NLL under a Laplace distribution likelihood L(x',*) when omitting unspecified measures.

[0067] A VAE decoder may require multiple variational outputs, such as location, scale, and shape, to generate x' given a likelihood belief P. For example, a normal distribution likelihood P = N(λ, γ) provides the mean of the reconstructed data x' = λ and the standard deviation of that γ as the confidence. Similarly, some embodiments use x' = λ in the maximum likelihood sense for a Laplace distribution. In another example, a beta distribution as the likelihood belief also has two variational parameters to be generated by a VAE decoder. Generally, given the decoder output of variational parameters (such as λ and γ), data reconstruction should be performed according to its mode (i.e., the peak of the PDF). For the Bernoulli likelihood B(λ), the mode is binary, since x' = 0 or 1 depending on λ, but some embodiments instead use the mean as the reconstruction x' = λ. In some embodiments, the output of the probabilistic DNN after multiple variational sampling is accumulated using a weighted average to improve the accuracy of variational inference.

[0068] VAE and its variants for probabilistic DNN models require specifying four probabilistic factors: posterior belief Q; prior belief Π; likelihood belief P; and discrepancy measure D. To design these factors, we often require manual effort in exploring the best combinations of different likelihood beliefs, such as those listed in Figure 7, different posterior-prior pairs, such as those listed in Figure 5B, and different divergence orders α > 0, such as those listed in Figure 6. The search space grows rapidly when irregular, heterogeneous, and mismatched pairings are considered at each latent node. This invention facilitates the design of such heterogeneous probabilistic DNNs by using an AutoML framework to explore these factors for VAE design in an automated manner, as shown in Figure 2; we call this concept AutoVAE. Exemplary Model Implementation

[0069] Rare irregular pairings of posterior-prior beliefs, such as logistic-normal or Laplace-normal, can outperform the common choice of normal-normal pairings when searching for the best divergence order α. The irregular heterogeneous pairings explored in AutoVAE can further achieve better performance due to the higher degrees of freedom. For example, as an exemplary model implementation, posterior-prior beliefs are heterogeneously mixed by AutoVAE as logistic-normal (Lo∥N) pairs, Laplace-normal (La∥N) pairs, and Laplace-Laplace (La∥La) pairs for 50%, 30%, and 20% of the latent variables, respectively, to achieve higher ELBO performance than standard VAE when analyzing the MNIST practical benchmark dataset.

[0070] Some embodiments use a simple VAE architecture based on multilayer perceptrons (MLPs) for the encoder and decoder blocks. The MLPs consist of several fully connected linear layers with hundreds of hidden nodes having rectified linear unit (ReLU) activations. For example, a two-layer MLP with 400 hidden nodes is used for both the VAE encoder and decoder. The number of latent nodes is selected to be approximately several tens depending on the dataset in some embodiments. For example, L = 20 stochastic nodes are used. Some embodiments use a gradient method based on Adam gradient optimization with a learning rate of 0.01 over 100 epochs with a mini-batch size of 1000. AutoML can also design such hyperparameters on irregular beliefs by using hypergradient methods such as Bayesian optimization.

[0071] Once the VAE is trained, the VAE decoder can be used as a generative model to reproduce synthetic data by feeding random samples drawn from the prior beliefs Π for some embodiments. For example, the generated images are evaluated by Fréchet initiation distance (FID) and kernel initiation distance (KID) to assess their natural distribution from the original image dataset. To evaluate the initiation score, some embodiments generate 50,000 images by sampling random latent variables z from the prior beliefs Π.

[0072] FIG. 8A illustrates exemplary characteristics of various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and mismatch measures for a generalized VAE model, according to some embodiments. More specifically, FIG. 8A illustrates mismatch VAE results under various combinations of likelihood, posterior, and prior beliefs after training on the benchmark MNIST dataset. For an MSE loss based on likelihood beliefs of an unspecified normal distribution P=N(λ,*), the ELBO performance of regular normal-normal pairs (for alpha order=1 and cumulative number of K=1) can be improved by exploring different pairings, such as logistic-normal pairs, Laplace-Laplace pairs, and logistic-Laplace pairs. The MSE and starting scores are also improved by irregular beliefs, for example, for logistic-normal pairs. When exploring different likelihood beliefs, the benefits of mismatch pairing can be achieved even more clearly. Furthermore, the irregular heterogeneous use of posterior-prior pairs in individual latent nodes (denoted as “Auto”) can, in some cases, further improve performance via automatic exploration of mixed pairs for irregular beliefs.

[0073] Furthermore, exploring the Renyi order α can further refine the VR bounds. FIG. 8B shows exemplary ELBO characteristics for various pairs of posterior and prior beliefs as a function of divergence order for a generalized VAE model, according to some embodiments. Specifically, FIG. 8B illustrates the benefit of exploring different disparity measures by adjusting the divergence order α for some embodiments. FIG. 8B demonstrates that when searching for the best divergence order α, rare pairings of posterior and prior beliefs, such as logistic-normal or Laplace-normal, can outperform the common choice of normal-normal pairings. It has also been confirmed that the irregular belief pairings explored in AutoVAE achieve the best performance on the MNIST dataset. The best posterior-prior mixtures for heterogeneous assignment were the logistic-normal (Lo∥N), Laplace-normal (La∥N), and Laplace-Laplace (La∥La) pairs, respectively, for 50%, 30%, and 20% of the L = 20 latent nodes at Renyi divergence order with α = 0.5. This exemplary result clearly demonstrates the benefit of our method of using irregular beliefs for probabilistic DNN models by enabling mismatched and heterogeneous pairing with an adjustable discrepancy measure.

[0074] 8C and 8D show example image snapshots generated by a generalized VAE decoder according to prior beliefs with variational sampling after being trained with various pairs of posterior and prior beliefs, according to some embodiments. Specifically, FIGS. 8C and 8D show synthetic image snapshots of randomly generated images using a VAE decoder trained on the MNIST dataset. By exploring different combinations of posterior beliefs, prior beliefs, likelihood beliefs, and disparity measures, particularly using heterogeneous pairings designed by the AutoML framework, the generator model can be significantly improved. Similar benefits can be achieved for different datasets. The present invention uses mismatched, heterogeneous, and irregular combinations of posterior-prior-likelihood beliefs for generalized VAE design. A mismatched VAE can outperform a standard VAE by using an automated search for irregular beliefs. Additionally, in some embodiments, the systems and methods of the present invention explore different disparity measures by adjusting the divergence order. The concept of AutoVAE can facilitate exploring appropriate combinations of these probabilistic factors to construct a generalized VAE model for variational inference. The irregular belief pairing technique is also applicable to other probabilistic DNNs besides VAEs. An example probabilistic DNN model

[0075] In addition to VAE, probabilistic DNN models include, but are not limited to, variational information bottleneck (VIB), denoising diffusion probabilistic model (DDPM), and variational Bayesian neural network (VBL). VIB uses a similar but different approach to VAE, with VIB being suitable for supervised learning and VAE being suitable for unsupervised learning. Probabilistic nodes in VIB are identified by irregular beliefs in a similar manner to AutoVAE, and therefore, some embodiments use automated search of irregular beliefs in VIB to improve performance. DDPM is another probabilistic DNN model similar to VAE, but involves multiple variational sampling in the diffusion step. Like AutoVAE, some embodiments use automated search of irregular beliefs to realize mismatches and disparate pairings in probabilistic nodes at each diffusion step. Similarly, in addition to VAE, VIB, and DDPM, some embodiments use different probabilistic DNN models when engaging probabilistic nodes under specific assumptions of underlying probabilistic beliefs.

[0076] 9A illustrates an exemplary probabilistic DNN model that sequentially uses multiple different belief and discrepancy measures in multiple hidden layers, according to some embodiments. This exemplary probabilistic DNN model in FIG. 9A takes in data x 901 and generates variational inference for task label y 905 through multiple DNN blocks, with some hidden layers being probabilistic nodes that use variational sampling according to individually specified beliefs. For example, the first probabilistic node z1 902 uses a normal distribution for posterior beliefs and a uniform distribution for prior beliefs, with KLD discrepancy; the second probabilistic node z2 903 uses a Cauchy distribution with a Renyi alpha divergence of order 0.5 for both posterior and prior beliefs; and the third probabilistic node z3 904 uses a logistic distribution for posterior beliefs and a normal distribution for prior beliefs, with a beta divergence of order 0.2. Such probabilistic DNNs with multiple probabilistic layers in sequence provide more complex uncertainty models, and heterogeneous beliefs can further improve the power of probabilistic modeling for variational inference.

[0077] 9B illustrates an exemplary probabilistic DNN model that uses different beliefs and discrepancy measures in different probabilistic nodes, in parallel within one hidden layer, and sequentially across multiple hidden layers, according to some embodiments. This exemplary probabilistic DNN model implements heterogeneous beliefs within one probabilistic layer to provide variational inference y 915 from data x 911 through multiple DNN blocks with parallel probabilistic nodes z1 912 and z2 913 in addition to sequential probabilistic node z3 914. Specifically, parallel probabilistic nodes z1 912 and z2 913 are heterogeneously identified with different beliefs and divergence metrics. Many other possibilities for connecting multiple DNN blocks with multiple probabilistic nodes are explored using supergradient methods to implement a high-performance generalized variational inference model for some embodiments.

[0078] Each probabilistic DNN block is configured with hyperparameters that specify a set of layers with trainable variables and interconnected neuron nodes to pass signals sequentially from layer to layer. The trainable variables are numerically optimized using gradient methods such as stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, and root-mean-square propagation. The gradient methods use training data to update the trainable parameters of the DNN block so that the output of the DNN block provides smaller loss values, such as mean squared error, cross-entropy, structural similarity, negative log-likelihood, absolute error, cross-covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, and triplet loss. Multiple loss functions are further weighted for combination with some regularization factor according to a training schedule policy.

[0079] In some embodiments, the probabilistic DNN block is reconfigurable according to hyperparameters such that the DNN block is composed of a set of fully connected layers, convolutional layers, graph convolutional layers, recurrent layers, multi-loop connections, skip connections, and inception layers, with a set of nonlinear activations including rectified linear transforms, hyperbolic tangents, sigmoids, gated linear, softmax, and thresholds. The DNN block is further regularized with a set of dropout, swapout, zoneout, blockout, dropconnect, noise injection, shaking, and batch normalization. In yet another embodiment, the layer parameters are further quantized to reduce memory size as specified by the adjustable hyperparameters. In another embodiment of link concatenation, the system uses multidimensional tensor projections together with dimensionally trainable linear filters to convert lower-dimensional signals into higher-dimensional signals for dimensionally mismatched links.

[0080] Another embodiment integrates AutoML with AutoBayes and AutoTransfer for hyperparameter search and training scheduling for each DNN block. Here, AutoTransfer uses a search for different auxiliary regularization modules from a diverse set of regularization methods, including but not limited to adversarial censorship, mutual information gradient estimation, and Wyner distance. Note that AutoTransfer and AutoBayes can be easily integrated with AutoML based on supergradient methods to optimize any hyperparameter of an individual DNN block. More specifically, the system modifies hyperparameters by exploring different combinations of discrete and continuous hyperparameter values ​​using reinforcement learning, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, Bayesian optimization, hyperbanding, and multi-objective Lamarckian evolution.

[0081] The system of the present invention also provides an additional testing step suitable as a post-training step to refine the trained DNN block by unpacking some trainable variables so that the DNN block can be robust to new datasets with new nuisance variations, such as new subjects. This embodiment can reduce the calibration time requirements for new users of the HMI system. Exemplary System

[0082] FIG. 10 shows an exemplary schematic diagram of a system comprising a processor, a memory, and an interface, according to some embodiments. Specifically, FIG. 10 is a block diagram illustrating an example of a system 1000 for signal analysis with automatic configuration of a probabilistic DNN. The system 1000 includes a device 1100 having a set of interfaces and data links 1105 configured to send and receive signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and storage 1140. The processor 1120, in association with the memory 1130, executes computer-executable programs and algorithms stored in the memory 1130 and the storage 1140. The set of interfaces and data links 1105 includes a human-machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 can execute computer-executable programs and algorithms in association with the memory 1130, which uploads the computer-executable programs and algorithms from the storage 1140. The computer-executable programs and algorithms stored in storage 1140 use a probabilistic deep neural network (DNN) 1141, trainable parameters 1142, hyperparameters 1143, scheduling policies 1144, forward pass signal / backward pass gradient and other temporary caches 1445, a regularization module 1146, gradient methods for training 1147, hypergradient methods for AutoML 1148, and irregular belief sets for probabilistic nodes 1149. The processor performs probabilistic inference to analyze a set of datasets by using the probabilistic DNN 1141, which uses importance-weighted accumulation after variational sampling at the probabilistic nodes according to the irregular belief sets 1149.

[0083] The system 1000 receives signals via a set of interfaces and data links 1105 over a network 1190. The signals include a set of datasets 1195 having data for training, validation, and testing. The set of datasets includes a set of multidimensional signals X associated with task labels Y to be identified. In some embodiments, the set of datasets further includes other side information, such as nuisance variations S.

[0084] In some cases, each of the probabilistic DNN blocks 1141 is configured to either encode a multidimensional signal X into latent variables Z, decode the latent variables Z to reconstruct the multidimensional signal X, classify a task label Y, estimate a nuisance variation S, regularize a latent space by estimating the nuisance variation S, or select a graphical model. The memory bank further includes intermediate neuron signals and temporary calculations including forward pass signals and backward pass gradients.

[0085] The processor 1120, in association with the interface and memory bank 1130, is configured to submit the signal and dataset 1195 to the probabilistic DNN block 1141 and explore different irregular beliefs 1149 and different hyperparameters 1143 by using a supergradient method 1148. The processor 1120 further performs the following: configuring the probabilistic DNN 1141 according to the hyperparameters 1143; calculating a loss function by forward-propagating the dataset 1195 through the probabilistic DNN 1141 according to probabilistic inference; modifying the loss function by regularizing the probabilistic nodes according to a discrepancy measure specified in the irregular beliefs 1149; and back-propagating the gradient of the loss function through the probabilistic DNN 1141 with respect to the trainable parameters 1142 to update the trainable parameters 1142 using a gradient method 1147. The hypergradient method for AutoML 1148 also explores different auxiliary regularization modules 1146, different hyperparameters 1143, different scheduling policies 1144, and irregular probabilistic beliefs 1149 to improve robustness to nuisance variations and unspecified uncertainty in the dataset 1195.

[0086] In some embodiments, system 1000 is applied to design an HMI through analysis of a user's physiological data. System 1000 receives signals of physiological data 1195 via network 1190 and set of interfaces and data links 1105. In some embodiments, system 1000 receives electroencephalogram (EEG) and electromyogram (EMG) measurements as user physiological data from a set of sensors 1111 and from other interface modules such as pointing device / medium 1112.

[0087] The above-described embodiments of the present invention may be implemented in any of numerous ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit with one or more processors within an integrated circuit component. However, a processor may be implemented using any suitable form of circuitry.

[0088] Also, embodiments of the present invention may be embodied as methods, for which examples are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that illustrated, which may include performing some acts shown as sequential acts in the exemplary embodiment simultaneously.

[0089] Furthermore, the use of ordinal numbers such as "first" or "second" to modify claim elements in the claims does not, by itself, imply a priority, precedence, or ordering of one claim element over another, or a chronological order in which method operations are performed, but rather serves merely as a label to distinguish one claim element having a particular name from another element having the same name (absent the use of the ordinal number).

[0090] Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. 1. A system for signal analysis, comprising: an interface configured to receive and transmit signals, the signals including a set of datasets including a set of multidimensional signals associated with a set of task labels, the system further comprising: a probabilistic deep neural network (DNN) block configured to identify the set of task labels from the set of multidimensional signals via a set of probabilistic nodes, the set of probabilistic nodes being individually specified by a set of irregular beliefs for posterior distributions, prior distributions, and likelihood distributions quantified by a set of discrepancy measures, the irregular beliefs including heterogeneous combinations of mismatches between posterior distributions and prior distributions or heterogeneous combinations of heterogeneous assignments to probabilistic nodes of a group of distributions including at least two or more of a variety of distributions; The various distributions include discrete univariate distributions, normal-related distributions, exponential-related distributions, extreme value distributions, bounded distributions, heavy-tailed distributions, quantile-based distributions, systematic distributions, multivariate continuous distributions, and multivariate discrete distributions; The system further comprises: a memory bank configured to store the set of datasets, the set of irregular beliefs, a set of hyperparameters, and a set of trainable parameters; and a processor, associated with the interface and the memory bank, configured to perform probabilistic inference to analyze the set of datasets by using the probabilistic DNN block, wherein the probabilistic inference uses importance-weighted accumulation after variational sampling on the set of probabilistic nodes according to the set of irregular beliefs.

2. The probabilistic DNN block includes a variational autoencoder (VAE), a variational information bottleneck (VIB), a denoising diffusion probability model (DDPM), and / or a variational Bayesian neural network, and the probabilistic DNN block is A transformation layer, which can be a fully connected layer, a bilinear layer, a convolutional layer, a graph convolutional layer, a recurrent layer, an implicit layer, an attention layer, a pooling layer, a padding layer, an initiation layer, and a transformer layer; Interconnections that are multi-loop connections, skip additions, skip multiplications, or skip concatenations; Nonlinear activations that are any of rectified linear, sigmoid, shrinkage, softmax, gated linear, soft plus, and soft sine; and 10. The system of claim 1 , configured with a combination of regularization layers that are any of dropout, swap out, zone out, block out, drop connect, shaking, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

3. The variational sampling uses a random number generator based on a reparameterization trick according to a set of variational parameters including position, scale, shape, and temperature specified by the set of random beliefs; and / or The system of claim 1 , wherein the set of discrepancy measures includes a combination of Renyi's alpha divergence, beta divergence, gamma divergence, Fisher divergence, and / or Jeffrey divergence.

4. The set of hyperparameters is: a set of training hyperparameters that determine the scheduling policy, which can be one of the following: learning rate, weight decay, gradient momentum, divergence order, batch size, number of epochs, early stopping, dropout rate, and regularization weight; A set of architecture hyperparameters, which are any of a connectivity for determining the probabilistic DNN block, a type of layer operation, a type of activation function, a size of the set of trainable parameters, a selection of probabilistic nodes, a number of layer depths, and a set of layer widths; and a set of evolution hyperparameters, the set being one of an initialization policy for the set of trainable parameters and a quantization policy for the set of trainable parameters.

5. The discrete univariate distribution is one of a Bernoulli distribution, a binomial distribution, a geometric distribution, a Pascal distribution, a hypergeometric distribution, a Waring-Yule distribution, a Poisson distribution, a Skellam distribution, a discrete uniform distribution, a Zipf distribution, a logarithmic series distribution, a Benford distribution, a Polya distribution, a Tanner distribution, and a Poisson-Consul distribution; the normal-related distribution is any one of a normal distribution, a log-normal distribution, a Johnson distribution, a half-normal distribution, a skew-normal distribution, a Voigt distribution, a Tsallis distribution, a hyperbolic distribution, a variance gamma distribution, a Student-t distribution, a chi distribution, a Rayleigh distribution, a Maxwell distribution, a chi-squared distribution, an F-ratio distribution, a Fisher distribution, a Hotelling distribution, and a Tracy distribution; the exponential-related distribution is one of the exponential distribution, Laplace distribution, Lindley distribution, logistic distribution, sech distribution, exponential power distribution, shifted Gompertz distribution, Erlang distribution, hypoexponential distribution, Coxian distribution, gamma distribution, Nakagami distribution, Moyal distribution, and Meixner distribution; the extreme value distribution is one of a max stable distribution, a min stable distribution, a Gumbel distribution, a Frechet distribution, and a Weibull distribution; the bounded distribution is one of a uniform distribution, an arcsine distribution, a Bates distribution, a Kumaraswamy distribution, a power distribution, a triangular distribution, a uniform sum distribution, a Mises distribution, a Wigner distribution, and a Pastur distribution; the heavy-tailed distribution is one of a Pareto distribution, a beta distribution of the second kind, a Dagum distribution, a Davis distribution, a Singh-Madala distribution, a Cauchy distribution, a Landau distribution, a Levy distribution, a Benini distribution, a Gibrat distribution, and a Suzuki distribution; the quantile-based distribution is one of a Tukey distribution and a Wakeby distribution; the systematic distribution is a Pearson distribution, the multivariate continuous distribution is one of a multivariate normal distribution, a multinomial distribution, a multivariate t-distribution, a Dirichlet distribution, and a Copula distribution; The system of claim 1 , wherein the multivariate discrete distribution is one of a multivariate hypergeometric distribution, a negative multinomial distribution, and a multivariate Poisson distribution.

6. The set of data sets is: Media data, which may be images, photographs, movies, text, characters, voices, music, audio or speeches; Physical data such as radio waves, optical signals, electrical pulses, temperature, pressure, acceleration, speed, vibration, and force; and 10. The system of claim 1, comprising a combination of physiological data selected from the group consisting of heart rate, blood pressure, mass, hydration, electroencephalogram, electromyogram, electrocardiogram, mechanomyogram, electrooculogram, galvanic skin response, magnetoencephalogram, and electrocorticography.

7. The processor further comprises: configuring the probabilistic DNN block by modifying connectivity between the set of multidimensional signals, the set of task labels, and the set of probabilistic nodes according to the set of hyperparameters; computing a loss function by forward propagating the set of data sets through the probabilistic DNN block according to the probabilistic inference; modifying the loss function by regularizing the set of probabilistic nodes according to the set of discrepancy measures; backpropagating the gradient of the loss function with respect to the set of trainable parameters through the stochastic DNN block; and updating the set of trainable parameters of the stochastic DNN block with a gradient method.

8. the loss function is a combination of variational Renyi bound, evidence lower bound, mean squared error, cross entropy, structural similarity, cosine similarity, negative log-likelihood, absolute error, cross-covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, and / or triplet loss; and / or The gradient method uses a combination of stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, and / or root mean square propagation; and / or 8. The system of claim 7, wherein the processor further performs the step of exploring different values ​​for the set of irregular beliefs and the set of hyperparameters in the memory bank by using a supergradient method, the supergradient method using a combination of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, and / or Bayesian optimization.

9. the probabilistic DNN block uses heterogeneous assignment of different beliefs to at least two disjoint sets of probabilistic nodes in a layer; and / or The probabilistic DNN block uses mismatched pairs of posterior beliefs and prior beliefs for at least one set of probabilistic nodes, and / or 2. The system of claim 1, wherein the set of irregular beliefs is modified to use different probability distributions for the variational sampling in the set of probabilistic nodes within the probabilistic DNN block when analyzing a newly available set of datasets.

10. 1. A computer-implemented method for data analysis, comprising: providing a set of data signals to a probabilistic deep neural network (DNN), the probabilistic DNN including a set of layers and a set of probabilistic nodes, the method further comprising: Propagating the set of data signals through the set of layers in the probabilistic DNN, wherein the probabilistic DNN uses variational sampling in the set of probabilistic nodes according to a set of irregular beliefs, the irregular beliefs including mismatched combinations of posterior distributions and prior distributions of a group of at least two or more types of diverse distributions, or heterogeneous combinations of posterior distributions and prior distributions assigned to probabilistic nodes; The various distributions include discrete univariate distributions, normal-related distributions, exponential-related distributions, extreme value distributions, bounded distributions, heavy-tailed distributions, quantile-based distributions, systematic distributions, multivariate continuous distributions, and multivariate discrete distributions; The method further comprises: A computer-implemented method comprising accumulating outputs of the probabilistic DNN as probabilistic inference results.

11. The probabilistic DNN includes a variational autoencoder (VAE), a variational information bottleneck (VIB), a denoising diffusion probability model (DDPM), and / or a variational Bayesian neural network, and the probabilistic DNN includes: A transformation layer, which can be a fully connected layer, a bilinear layer, a convolutional layer, a graph convolutional layer, a recurrent layer, an implicit layer, an attention layer, a pooling layer, a padding layer, an initiation layer, and a transformer layer; Interconnections that are multi-loop connections, skip additions, skip multiplications, or skip concatenations; Nonlinear activations that are any of rectified linear, sigmoid, shrinkage, softmax, gated linear, soft plus, and soft sine; and 11. The method of claim 10, configured with a combination of regularization layers, the regularization layers being any of dropout, swap out, zone out, block out, drop connect, shaking, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

12. the variational sampling uses a random number generator based on a reparameterization trick according to a set of variational parameters including location, scale, shape, and temperature specified by the set of random beliefs; the discrete univariate distribution is one of a Bernoulli distribution, a binomial distribution, a geometric distribution, a Pascal distribution, a hypergeometric distribution, a Waring-Yule distribution, a Poisson distribution, a Skellam distribution, a discrete uniform distribution, a Zipf distribution, a logarithmic series distribution, a Benford distribution, a Polya distribution, a Tanner distribution, and a Poisson-Consul distribution; the normal-related distribution is any one of a normal distribution, a log-normal distribution, a Johnson distribution, a half-normal distribution, a skew-normal distribution, a Voigt distribution, a Tsallis distribution, a hyperbolic distribution, a variance gamma distribution, a Student-t distribution, a chi distribution, a Rayleigh distribution, a Maxwell distribution, a chi-squared distribution, an F-ratio distribution, a Fisher distribution, a Hotelling distribution, and a Tracy distribution; the exponential-related distribution is one of the exponential distribution, Laplace distribution, Lindley distribution, logistic distribution, sech distribution, exponential power distribution, shifted Gompertz distribution, Erlang distribution, hypoexponential distribution, Coxian distribution, gamma distribution, Nakagami distribution, Moyal distribution, and Meixner distribution; the extreme value distribution is one of a max stable distribution, a min stable distribution, a Gumbel distribution, a Frechet distribution, and a Weibull distribution; the bounded distribution is one of a uniform distribution, an arcsine distribution, a Bates distribution, a Kumaraswamy distribution, a power distribution, a triangular distribution, a uniform sum distribution, a Mises distribution, a Wigner distribution, and a Pastur distribution; the heavy-tailed distribution is one of a Pareto distribution, a beta distribution of the second kind, a Dagum distribution, a Davis distribution, a Singh-Madala distribution, a Cauchy distribution, a Landau distribution, a Levy distribution, a Benini distribution, a Gibrat distribution, and a Suzuki distribution; the quantile-based distribution is one of a Tukey distribution and a Wakeby distribution; The systematic distribution is the Pearson distribution, the multivariate continuous distribution is one of a multivariate normal distribution, a multinomial distribution, a multivariate t-distribution, a Dirichlet distribution, and a Copula distribution; The method of claim 10 , wherein the multivariate discrete distribution is one of a multivariate hypergeometric distribution, a negative multinomial distribution, and a multivariate Poisson distribution.

13. moreover, computing a loss function based on variational bounds to regularize the set of probabilistic nodes according to a set of discrepancy measures; backpropagating the gradient of the loss function with respect to a set of trainable parameters; and updating the set of trainable parameters according to a gradient method, wherein the gradient method uses a combination of stochastic gradient descent, Adam, Ada gradient, Ada bound, Nesterov accelerated gradient, and / or root-mean-square propagation.

14. 14. The method of claim 13, wherein the set of discrepancy measures includes a combination of Renyi alpha divergence, beta divergence, gamma divergence, Fisher divergence, and / or Jeffrey divergence, and the loss function is a combination of variational Renyi bound, evidence lower bound, mean squared error, cross-entropy, structural similarity, cosine similarity, negative log-likelihood, absolute error, cross-covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, and / or triplet loss.

15. 14. The method of claim 13, further comprising searching for different values ​​for the set of irregular beliefs according to a supergradient method, wherein the supergradient method uses a combination of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, and / or Bayesian optimization.

Images