Automatic variational inference using stochastic models with irregular belief

Abstract

A system and method for automatically building a random deep neural network (DNN) architecture is provided. The framework of the present invention automatically searches for the most relevant random pattern as a basis for a data set for variational Bayesian inference. The present invention provides an approach to use heterogeneous belief, irregular belief and mismatch belief in stochastic sampling for intermediate representations in DNN with the ability of an auto-tuning mechanism for posterior, priori and likelihood models to achieve an accurate generative model and an uncertainty model for machine learning tasks. The system also allows an adjustable difference metric to regularize the intermediate representation by variants of a divergence metric including Rayleigh alpha divergence, beta divergence, and gamma divergence. The present invention enables various hybrid combinations of random models for misspecified and unspecified probabilistic relationships in an automatic manner. Thus, the representation capabilities of variational autoencoders, variational information bottlenecks, de-noising diffusion probability models, and other random DNNs are improved.

Description

Technical Field

[0001] The present invention relates to automated learning systems for artificial neural networks, and more particularly to automated Bayesian inference systems based on stochastic neural networks with irregular beliefs. Background Art

[0002] Great advances in deep learning techniques based on deep neural networks (DNNs) have solved a variety of problems in data processing, including: media signal processing for video, speech, and images; physical data processing for radio waves, electronic pulses, and light beams; and physiological data processing for heart rate, temperature, and blood pressure. However, DNN architectures are usually handcrafted with insights from experts who know the inherent data model and structure. How to optimize the architecture of DNNs requires a time / resource consuming trial and error method. The framework of automated machine learning (AutoML) is used to automatically explore different DNN architectures to solve problems. Automation of hyperparameter and architecture exploration in the context of AutoML can promote DNN design suitable for specific data processing. AutoML includes architecture search, learning rule design, and enhanced exploration. Most AutoML methods use evolutionary optimization, super gradient, or reinforcement learning frameworks to adjust hyperparameters or build network architectures from pre-selected choices of building blocks. The recent AutoML-Zero considers expansion to exclude expert knowledge and insights from fully automated design from scratch.

[0003] Learning data representations that capture task-relevant features but are invariant to nuisance variations remains a key challenge in machine learning. A variational Bayesian inference method introduced by a random DNN cascade variational autoencoder (VAE) combined with an auto-association architecture, in which the generation and inference models are jointly learned by using a pair of decoder architectures and encoder architectures. This method extends conditional VAEs and regularized VAEs, where conditional VAEs introduce conditional variables that can be used to represent nuisance, and regularized VAEs consider disentangling nuisance variables from latent representations. The concept of adversarial learning is considered in Generative Adversarial Networks (GANs) and has been adopted in countless applications. Adversarially Learned Inference (ALI) and Bi-directional GANs (Bi GANs), which were discovered at the same time, propose adversarial methods for training autoencoders. Adversarial training has also been combined with VAEs to regularize and disentangle latent representations, enabling nuisance-rubost learning.

[0004] For Bayesian inference, VAE is configured with parameter encoders and decoders to learn latent variables underlying the data within the variational inference (VI) framework. There are many variants of this stochastic DNN. For example, β-VAE uses the emphasized Kullback-Leibler divergence (KLD) to regularize the latent distribution more strongly than the typical evidence-lower bound (ELBO)-based loss. Continuous Bernoulli and β distributions are studied as alternative likelihood beliefs for data reconstruction models. Laplace and Cauchy distributions are considered as alternative prior beliefs for sparse potential representations. Normal posterior beliefs are adjusted by inverse autoregressive flow (IAF), importance-weighted autoencoder (IWAE), and Gibbs sampling. IWAE is further extended to variational Renyi (VR) based on Renyi's α divergence. Then, the generalized VI (GVI) discusses any arbitrary choice of loss, divergence, and posterior, and it shows the higher robustness of using different divergence metrics instead of KLD when the data statistics are not specified.

[0005] While generalized VAEs offer large degrees of freedom, it is difficult to design those choices in addition to other architectural hyperparameters. No framework exists for designing variational Bayesian inference and stochastic DNNs when no intrinsic model is specified to capture data statistics and probabilistic uncertainty. For example, stochastic DNNs including VAEs, variational information bottlenecks (VIBs), and denoising diffusion probability models (DDPMs) typically use isomorphic statistics for latent representations based on normal distributions. The normal distribution is computationally convenient because simple sampling, reparameterization techniques, and closed-form divergence expressions are possible for gradient computations. However, when the statistics underlying the real dataset are not known in advance, the normal distribution is usually no longer optimal.

[0006] However, the disadvantage of AutoML is that it requires a lot of exploration time to find the best hyperparameters due to the surge in search space. In addition, without any good inference, most of the search space of link connections will be meaningless. In order to develop a system for automatically building an artificial neural network with rationality, a method called AutoBayes is proposed. The AutoBayes method uses different Bayesian graphs to represent the inherent graphical relationship between random variables used to generate the model, and then constructs the most reasonable inference graph to connect encoders, decoders, classifiers, regressors, opponents, and estimators. By using the so-called Bayesian ball algorithm, the most compact inference graph for a specific Bayesian graph can be automatically constructed, and some factors are identified as variable independence of domain factors to be reviewed by the adversarial block. It is verified that adversarial censoring, which cleans up the nuisance factors from the feature space, is effective for domain generalization in pre-shooting transfer learning and domain adaptation in post-shooting transfer learning. However, adversarial training requires careful selection of hyperparameters because too strong review will impair the main task performance because the main objective function is relatively underweighted. Furthermore, adversarial censoring is not the only regularization method that promotes independence from nuisance variables in feature space.

[0007] Therefore, there is a need to efficiently identify the best probabilistic model for random DNNs that depends on specific problems and datasets. Summary of the invention

[0008] The present invention is based on the recognition that random deep neural networks (DNNs) use specific probability models and impose many different probability models, including normal distribution, Laplace distribution, Cauchy distribution, logistic distribution, Gumbel distribution, student-t distribution, uniform distribution, exponential distribution and hyperexponential distribution, while the true probability model underlying real-world data sets is usually unknown. In addition, the possibilities of representing latent variables and data randomness and correlation measures (such as divergence measures in generalized variational inference (GVI)) are not restricted. For example, there is no formal framework to determine what difference measure to choose from the infinite possibilities including Kullback-Leibler divergence (KLD), Rayleigh's alpha divergence, beta divergence, gamma divergence, Jensen-Shannon divergence, Jeffery divergence and Fisher divergence to regularize random nodes in DNNs. In addition, the data uncertainty of identifying likelihood beliefs is usually not specified. For example, the normal distribution is often used to minimize the weighted mean-square error (MSE) for regression problems, and the Bernoulli distribution is often used to minimize the binary cross entropy as the reconstruction loss for nearly binary pixel images.

[0009] The present invention enables random DNNs to efficiently identify data statistics and uncertainty models by using an AutoML framework to better combine prior beliefs, posterior beliefs, likelihood beliefs, and difference metrics. This provides benefits beyond the automated construction of DNN architectures such as layer size, node size, activation type, and link connectivity of DNNs. Specifically, the present invention provides a method for searching for irregular combinations of random models over conventional architecture hyperparameters. Some embodiments allow random DNNs to impose unmatched posterior beliefs and prior belief pairs to increase the degrees of freedom for modeling capabilities under unspecified model uncertainty. Yet another embodiment provides a method for generalizing random DNNs toward heterogeneous pairs of posterior beliefs and prior beliefs at different hidden nodes and hidden layers. In this way, more robust and complex potential representations can be achieved. In addition, the present invention provides a method for automatically searching for irregular assignments, mismatched assignments, and heterogeneous assignments of difference metrics for each random node. For example, a first latent node uses a logistic distribution as a posterior belief and a normal distribution as a prior belief using a difference measure of KLD, while a second latent node uses a Cauchy distribution as a posterior belief and a uniform distribution as a prior belief using Rayleigh's alpha divergence of order 0.4. This hybrid heterogeneous random model can improve generalization across different data sets following unknown uncertainty. Some embodiments utilize variants of reparameterization techniques (such as the Gumbel softmax technique for latent representations in VAE, VIB, DDPM and their variants) in conjunction with network architecture search to use a categorical search space across different beliefs. Prior beliefs, posterior beliefs, and likelihood beliefs include, but are not limited to, normal distribution, Cauchy distribution, logistic distribution, Laplace distribution, uniform distribution, triangular distribution, Gumbel distribution, exponential distribution, generalized Gaussian distribution, beta distribution, gamma distribution, Poisson distribution, Bernoulli distribution, etc. In addition, different review methods (such as adversarial cleaning) can promote latent representations within random DNN models to be independent of nuisance parameters, so that nuisance robust feature extraction is achieved for some embodiments.

[0010] The present invention provides a way to adjust those hyperparameters and irregular beliefs under the AutoML framework based on super gradient methods such as Bayesian optimization, implicit gradient, reinforcement learning, and heuristic optimization. Another embodiment uses mismatched posterior-prior pairings and heterogeneous distributions in random DNNs. For example, the posterior belief of the VAE encoder uses a logistic distribution, while the prior belief of the VAE decoder uses a normal distribution of random latent variables. The present invention is based on the following recognition: due to model mismatch, smaller divergence is not always better, which shows that heterogeneous posterior-prior pairs have the potential to provide a better Bayesian model for the entire random DNN, especially when the uncertainty model is not specified in advance.

[0011] Another embodiment uses multiple intermediate representations for variational sampling to improve model accuracy. For this case, the number of combinations selected for each potential layer will grow rapidly, so automatic search will be important. In addition, for some embodiments, the present invention provides a way to allow irregular non-uniform posterior-prior pairs at each latent variable. One embodiment implements an integrated method for exploring stacking protocols through cross-validation during AutoML. The present invention provides a method for automatically generating an auxiliary model that directly controls the parameters of the basic inference model by analyzing the consistent evolutionary behavior in the main DNN model. For some embodiments, the type of divergence is also automatically selected in a non-isomorphic and irregular manner. For example, Rayleigh's alpha divergence, beta divergence, gamma divergence, Wasserstein distance and its order are jointly explored to be robust to numerically difficult data statistics occurring in the real world.

[0012] The present disclosure relates to systems and methods for automatically building stochastic DNNs by exploring different uncertainty models and hyperparameters. Specifically, the system of the present invention introduces an automatic variational Bayesian inference framework that explores different sets of posteriors / priors / likelihoods / discrepancies for variational inference models linking classifiers, encoders, decoders, and estimator blocks to optimize a nuisance-invariant machine learning pipeline. In one embodiment, the framework is applied to a series of physiological datasets, where we have access to object and class labels during training, and provides analysis of its ability to transfer learning with / without variational modeling and adversarial training. The framework can be effectively used for semi-supervised multi-class classification, multidimensional regression, and data reconstruction tasks for various dataset forms (such as media signals and electrical signals as well as biological signals).

[0013] Some embodiments of the present disclosure are based on the following recognition: a new concept called AutoBayes is suitable for interference-robust inference systems, which explores a variety of different Bayesian graph models to facilitate the search for the best inference strategy. Using the Bayes-Ball algorithm, the methods and systems in the present invention can automatically construct reasonable link connections between classifiers, encoders, decoders, interference estimators, and opponent DNN blocks, as well as different belief combinations of random nodes. According to the no free lunch theorem, using a specific model without exploring its variants can potentially suffer from poor inference results. In addition, the best model at one data set does not always perform best for different data, which encourages the use of the AutoML framework for adaptive model generation given a target data set. One embodiment extends the AutoBayes framework to integrate random DNNs for unspecified / misspecified uncertainty base data sets by exploring different irregular belief sets for posterior, prior, likelihood, and difference.

[0014] Another embodiment uses variational sampling for a semi-supervised setting for situations when the data set includes missing target labels. Yet another embodiment uses an integrated stack that combines estimates of multiple different Bayesian models to improve performance. Another embodiment uses a random graph neural network to exploit the geometric information of the data, and the pruning strategy is assisted by belief propagation across the Bayesian graph to verify the correlation. The Wasserstein distance can also be used instead of the divergence. For some embodiments, the system and method can be combined with existing test-time online adaptation techniques from zero-shot and few-shot learning frameworks to achieve even better interference robust performance. The system can provide the benefits of learning interference-invariant representations by exploring various regularization modules and uncertainty models. Yet another embodiment uses quantum and molecular devices for sampling random nodes by exploiting device randomness. Some embodiments use a reversible network architecture to model encoders and decoders simultaneously without having two disjoint models. Some embodiments use random hidden layers embedded in DNN architectures, such as neural ordinary differential equations, convex optimization, deep equalization, diffusion models, and quantum dynamics. In this way, the present invention provides a way to design a Bayesian machine learning model that is robust and automatically models unknown uncertainty underlying data sets by exploring various combinations of prior beliefs, posterior beliefs, and likelihood beliefs in random representation learning (such as VAE, DDPM, and VIB) under imperfect and misspecified data statistical knowledge.

[0015] In addition, according to some embodiments of the present invention, a system for data analysis using a random DNN block is provided. In this case, the system for signal analysis includes an interface, a random deep neural network (DNN) block, a storage body, and a processor. The interface is configured to receive and send signals (such as a data set of a multidimensional signal associated with a task label). The random DNN block is used to identify the task label from the multidimensional signal through a random node, and the random node is individually specified by an irregular belief for the posterior distribution, the prior distribution, the likelihood distribution, and the difference metric. The storage body is used to store the data set, the irregular belief, other hyperparameters, and trainable parameters to determine the random DNN block. The processor performs probabilistic inference in conjunction with the interface and the storage body to analyze the data set by using the random DNN block. Probabilistic inference is achieved by performing importance-weighted accumulation after variational sampling at the random node according to the irregular belief. The random DNN block includes a variant of VAE, VIB, and DDPM, which is configured with a combination of transformation layers, interconnections, nonlinear activations, and regularization layers. Variational sampling is based on using a random number generator based on a reparameterization technique of variational parameters including position, scale, shape, and temperature specified by irregular beliefs. The divergence measures include Rayleigh's alpha divergence, beta divergence, gamma divergence, a combination of Fisher's divergence and Jeffrey's divergence.

[0016] Irregular beliefs include discrete univariate distributions, normally correlated distributions, exponentially correlated distributions, extreme value distributions, bounded distributions, heavy-tailed distributions, quantile-based distributions, systematic distributions, multivariate continuous distributions, and mismatched or heterogeneous combinations of multivariate discrete distributions. The random DNN block uses heterogeneous distributions of different beliefs for at least two disjoint sets of random nodes in the same layer. In addition, the random DNN block uses mismatched pairs of posterior beliefs and prior beliefs for at least one set of random nodes. At least two different types of uncertainty models for posterior beliefs, prior beliefs, and likelihood beliefs are simultaneously applied in disjoint random nodes. In addition, different difference metrics (such as KLD and Rayleigh's alpha divergence) are simultaneously applied in the random DNN block to regularize the random nodes. When analyzing new available sets of data sets, irregular beliefs can also be adjusted on the fly to use different probability distributions for variational sampling at random nodes.

[0017] The system of some embodiments employs the following steps: exploring different values ​​of irregular beliefs and hyperparameters by using a hyper-gradient method; configuring a random DNN by modifying the connectivity between multidimensional signals, task labels, and random nodes; computing a loss function by forward propagating a dataset across random DNN blocks based on probabilistic inference; modifying the loss function by regularizing the random nodes based on a discrepancy metric; backward propagating the gradient of the loss function with respect to trainable parameters; and updating the trainable parameters using a gradient method.

[0018] By allowing irregular beliefs at random DNN blocks, more complex model descriptions of unknown statistics for real-world datasets can be achieved. Therefore, even when the real-world dataset is not difficult to handle in any closed-form statistical model, the system of the present invention provides better variational inference results by exploring irregular combinations of posterior distributions, prior distributions, and likelihood distributions in random DNN blocks.

[0019] In addition, some embodiments of the present invention provide a computer-implemented method for variational inference using a random DNN. Specifically, the method of variational inference includes: feeding a data signal into a random DNN; propagating the signal according to the layers in the random DNN by adopting variational sampling at random nodes according to irregular beliefs; and accumulating the output of the random DNN as a probabilistic inference result. In addition, a method for automatic design of a variational inference model includes: calculating a loss function based on a variational bound to regularize random nodes according to a difference metric; back-propagating the gradient of the loss function with respect to trainable parameters; updating the trainable parameters according to a gradient method; and exploring different values ​​of irregular beliefs according to a super gradient method. The gradient method adopts a combination of stochastic gradient descent, adaptive momentum, Ada gradient, Ada bound, Nesterov accelerated gradient, root mean square propagation, and variants thereof. The super gradient method adopts a combination of reinforcement learning, implicit gradient, evolutionary strategy, differential evolution, particle swarm, genetic algorithm, simulated annealing, Bayesian optimization, and variants thereof.

[0020] The accompanying drawings, which are included to provide a further understanding of the invention, illustrate embodiments of the invention and together with the description serve to explain the principle of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] [ Figure 1A ]

[0022] Figure 1A shows an example inference model for predicting Y given data X according to some embodiments;

[0023] [ Figure 1B ]

[0024] Figure 1B shows an example inference model for predicting Y given data X under latent variables Z and nuisance factors S according to some embodiments;

[0025] [ Figure 1C ]

[0026] Figure 1C shows an example inference model for predicting Y given data X under latent variables Z and nuisance factors S according to some embodiments;

[0027] [ Figure 2 ]

[0028] Figure 2 shows an exemplary schematic diagram of an automatic VAE according to some embodiments;

[0029] [ Figure 3 ]

[0030] Figure 3shows exemplary variations of irregular beliefs according to some embodiments;

[0031] [ Figure 4A ]

[0032] Figure 4A shows exemplary beliefs of posterior distribution, prior distribution, and likelihood distribution for a random DNN according to some embodiments;

[0033] [ Figure 4B ]

[0034] Figure 4B shows an exemplary standard position-scale family for variational sampling based on the reparameterization trick according to some embodiments;

[0035] [ Figure 5A ]

[0036] Figure 5A shows an exemplary scenario of KLD differences for various pairs of posterior beliefs and prior beliefs at two independent random nodes according to some embodiments;

[0037] [ Figure 5B ]

[0038] Figure 5B shows an exemplary posterior belief and prior belief pair quantified by a KLD difference metric according to some embodiments;

[0039] [ Figure 5C ]

[0040] Figure 5C shows exemplary KLD properties of various a posteriori belief and a priori belief pairs as a function of position value according to some embodiments;

[0041] [ Figure 6 ]

[0042] Figure 6 shows an exemplary difference metric based on Rayleigh's alpha divergence covering variants of generalized variational inference including importance-weighted autoencoder (IWAE) and standard VAE according to some embodiments;

[0043] [ Figure 7 ]

[0044] Figure 7 shows an exemplary likelihood belief P for a stochastic DNN determining a generalized negative log-likelihood loss according to some embodiments;

[0045] [ Fig. 8A ]

[0046] Fig. 8A shows exemplary properties of various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and difference measures for a generalized VAE model according to some embodiments;

[0047] [ Figure 8B ]

[0048] Figure 8B shows exemplary ELBO properties for various pairs of posterior beliefs and prior beliefs as a function of the divergence order of a generalized VAE model according to some embodiments;

[0049] [ Figure 8C ]

[0050] Figure 8C shows exemplary image snapshots generated by a generalized VAE decoder using variational sampling according to prior beliefs after training with various posterior belief and prior belief pairs in accordance with some embodiments;

[0051] [ Fig.8D ]

[0052] Fig.8D shows example image snapshots generated by a generalized VAE decoder using variational sampling according to prior beliefs after training with various posterior belief and prior belief pairs in accordance with some embodiments;

[0053] [ Fig.9A ]

[0054] Fig.9A An exemplary stochastic DNN using multiple different beliefs and variance metrics at multiple intermediate layers in a sequence is shown in accordance with some embodiments;

[0055] [ Fig. 9B ]

[0056] Fig. 9B shows an exemplary stochastic DNN that uses different beliefs and variance metrics in parallel at different stochastic nodes within an intermediate layer and sequentially at multiple intermediate layers in accordance with some embodiments;

[0057] [ Fig.10 ]

[0058] Fig.10 An exemplary schematic diagram of a system configured with a processor, memory, and interface according to some embodiments is shown. DETAILED DESCRIPTION

[0059] Various embodiments of the present invention are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and in all drawings, elements of similar structure or function are represented by the same reference numerals. It should also be noted that the drawings are intended only to facilitate the description of specific embodiments of the present invention. They are not intended to serve as a detailed description of the present invention or as a limitation to the scope of the present invention. In addition, the aspects described in conjunction with a specific embodiment of the present invention are not necessarily limited to that embodiment, and may be practiced in any other embodiment of the present invention.

[0060] The present invention provides a system and method for analyzing signals in a data set based on a generalized variational inference (GVI) framework, which uses a probabilistic model that uses the data set as the basis for an artificial intelligence (AI) pipeline. Figure 1A An example inference model for predicting a task label Y given data X according to some embodiments is shown. The task label is a classification identification number or a non-classification continuous value. For classification inference, the AI ​​model performs a classification task, while the AI ​​model performs a regression task for non-classification inference. For some embodiments, the task label is a mixture of multiple classification and continuous labels as multi-task inference. The task label is a scalar value or a vector of multiple values ​​for supervised learning. For unsupervised learning, the task label is not explicitly provided, or for some embodiments, the task label is equivalent to the input data for reconstruction. The data in the data set is in the form of a tensor with at least one axis to represent many signals and sensor measurements, including but not limited to: media data such as images, pictures, movies, text, letters, voice, music, audio, and speech; physical data such as radio waves, optical signals, electrical impulses, temperature, pressure, acceleration, speed, vibration, and force; and physiological data such as heart rate, blood pressure, mass, moisture, electroencephalogram, electromyogram, electrocardiogram, mechanomyogram, electrooculogram, galvanic skin response, magnetoencephalogram, and electrocorticogram.

[0061] For example, an AI model predicts emotions from brainwave measurements of a person, where the data is a three-axis tensor representing a space-time spectrogram from a multi-channel sensor at a specific measurement time. All available data signals with X and Y pairs are bundled into a whole batch of data sets for training the AI ​​model, and they are referred to as training data or training data sets. For some embodiments, for a portion of the training data set in semi-supervised learning, the task label Y is missing, where the missing label is predicted by the AI ​​model for self-supervision.

[0062] The AI ​​model can be implemented by a deep neural network (DNN) model, whose architecture and behavior are specified by a set of hyperparameters. The set of hyperparameters includes, but is not limited to: training hyperparameters that determine the scheduling strategy, such as learning rate, weight decay, gradient momentum, divergence order, batch size, number of epochs, early stopping, dropout rate, and regularization weights; architectural hyperparameters that determine the DNN architecture, such as connectivity, type of layer operations, type of activation function, size of trainable parameters, selection of random nodes, number of layer depths, and set of layer widths; and deployment hyperparameters for trainable parameters, such as initialization and quantization strategies.

[0063] DNN models are typically based on multi-layer perception using a combination of units, including but not limited to: transformation layers, such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, hidden layers, attention layers, pooling layers, padding layers, inception layers, and transformer layers; interconnections, such as recurrent connections, skip additions, skip multiplications, and skip cascading; non-linear activations, such as rectified linear variants, sigmoid variants, shrinkage variants, softmax variants, gated liner, soft addition, and soft sign; and regularization layers, such as dropout, swap out, zone out, block out, drop connections, jitter, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

[0064] For example, the residual network architecture uses skip addition from one hidden layer to another, which enables stable learning of deeper layers. The DNN model has multiple trainable parameters, such as affine transformation weights and biases.

[0065] The DNN model is trained on the training data set to minimize or maximize the objective function (also called loss function or utility function) by a gradient method, including but not limited to stochastic gradient descent, adaptive momentum gradient, root mean square propagation, Ada gradient, adaptive increment, adaptive maximum, Ada boundary, Nesterov accelerated gradient, elastic backpropagation and weighted adaptive momentum. For some embodiments, the training data set is divided into multiple sub-batches for local gradient updates. For some embodiments, a portion of the training data set is taken out for the validation data set to evaluate the performance of the DNN model. In some embodiments, the validation data set from the training data set is cyclically used for cross-validation. The manner of dividing the training data into sub-batches for cross-validation includes but is not limited to: random sampling; weighted random sampling; one session is taken out; one object is taken out; one region is taken out. Typically, due to domain shift, the data distribution of each sub-batch is different.

[0066] Gradient-based optimization algorithms have some hyperparameters, such as learning rate and weight decay. The learning rate is an important parameter to choose and can be automatically adjusted by some scheduling strategies (such as step functions, exponential functions, trigonometric functions, and adaptive decay on the platform). Other optimization methods (such as evolutionary strategies, genetic algorithms, differential evolution, simulated annealing, particle swarms, Bayesian optimization, and Nelder-Mead can also be used to optimize trainable parameters. The objective function is a combination of various functions, including but not limited to: L1 loss; Lp norm; mean squared error; cross entropy; connectionist temporal classification loss; negative log-likelihood; Kullback-Leibler divergence (KLD); mutual covariance; structural similarity; cosine similarity; clustering loss; margin ranking loss; hinge loss; Huber loss; negative sampling; Wasserstein distance; triplet loss.

[0067] Due to the over-parameterized DNN architecture to solve the task problem, AI models without guidance for hidden nodes often suffer from local minimum capture. In order to stabilize the training convergence, some regularization techniques are used. For example, the L1 / L2 norm is used to regularize the affine transformation weights. Batch normalization and discarding techniques are also widely used to prevent overfitting. Other regularization techniques include, but are not limited to: discarding connections; discarding blocks; discarding paths; jittering discarding; spatial discarding; ignoring; random depth; random width; spectral normalization; jitter-jittering. However, those well-known regularization techniques do not utilize the underlying data statistics. Most data sets have a specific probabilistic relationship between X and Y and a potential interference factor S that interferes with the task prediction performance. For example, physiological data sets (such as brain wave signals) are highly dependent on the subject's mental state and measurement conditions, which are considered to be interference factors S. Interference variations include, but are not limited to, a collection of object identity, session number, biological state, environmental state, sensor state, sensor position, sensor orientation, sampling rate, time, and sensitivity. For another example, electromagnetic data sets (such as Wi-Fi signals) are sensitive to room environment, surrounding users, interference, and hardware defects. The present disclosure provides a way to efficiently regularize DNN blocks by considering those interference factors, so that the AI ​​model is insensitive to domain shifts caused by changes in interference factors.

[0068] The present disclosure is based on a random DNN that applies a probabilistic model at intermediate layers and latent variables, whose nodes are referred to as random nodes. Specifically, a random intermediate layer is randomly sampled from a specific distribution family (such as a normal distribution), and the variational parameters of the specific distribution family (such as a location factor (or mean) and a scale factor (or standard deviation)) are defined by the intermediate layers of the random DNN. In order to achieve differentiable variational parameters, a reparameterization technique is used, in which random variables sampled from a standard distribution are shifted and scaled to distribute the expected distribution of the latent variables. In order to regularize the random nodes, some embodiments modify the loss function based on a difference metric, the difference metric including but not limited to: KLD; Rayleigh's alpha divergence; beta divergence; gamma divergence; Jeffrey divergence; Fisher divergence; Jensen-Shannon divergence; Wassassern distance.

[0069] In the GVI framework, the target distribution is specified as a prior belief, and the sample distribution is specified as a posterior belief. In addition, the task label and data prediction are also randomly defined under a specific uncertainty model to determine the likelihood belief. Likelihood beliefs include, but are not limited to: normal distribution; Bernoulli distribution; unspecified normal distribution; continuous Bernoulli distribution; Laplace distribution; Cauchy distribution; β distribution; unspecified Laplace distribution; γ distribution. Given the specified likelihood belief, the corresponding negative log-likelihood (NLL) is used to minimize the objective function of the random DNN. For example, the variational autoencoder (VAE) uses an encoder model and a decoder model to infer latent variables and a generative model, respectively. For the standard VAE, a reconstruction loss based on the mean squared error (MSE) is used under the assumption of a likelihood belief based on an unspecified normal distribution with an undefined variance. In addition, the latent variables are usually sampled from the normal distribution of the posterior belief and the prior belief. Some embodiments of the present invention allow irregular heterogeneous pairing of posterior beliefs and prior beliefs to give additional degrees of freedom to the unknown data uncertainty model. Yet another embodiment uses irregular heterogeneous difference metrics at various random nodes in the DNN model to be robust to mis-specified uncertainty. In addition to VAE, for some embodiments, the random DNN model includes but is not limited to variational information bottleneck (VIB), denoising diffusion probability model (DDPM), and variational Bayesian neural network.

[0070] The present invention uses irregular beliefs for posterior distribution, prior distribution and likelihood distribution and their difference measures to determine the statistics of random nodes in a random DNN block.Irregular beliefs include mismatched or heterogeneous combinations of different distributions, including but not limited to: discrete univariate distributions such as Bernoulli distribution, binomial distribution, geometric distribution, Pascal distribution, hypergeometric distribution, Waringol distribution, Poisson distribution, Skellam distribution, discrete uniform distribution, Zipf distribution, logarithmic series distribution, Benford distribution, Polya distribution, Tanner distribution, and Poisson-Kombucha distribution. Consul distribution; normally related distributions, such as normal distribution, lognormal distribution, Johnson distribution, half normal distribution, skew normal distribution, Voigt distribution, Tsallis distribution, hyperbolic distribution, variance gamma distribution, Student-t distribution, chi distribution, Rayleigh distribution, Maxwell distribution, chi-square distribution, F-ratio distribution, Fisher distribution, Hotelling distribution, and Tracy distribution; exponential related distributions, such as exponential distribution, Laplace distribution, Lindley distribution, logistic distribution, hyperbolic secant (sech) distribution, exponential power distribution, shift distribution, Gompertz, Erlang, subexponential, Coxian, gamma, Nakagami, Moyal, and Meixner distributions; extreme value distributions such as maximally stable, minimally stable, Gumbel, Frechet, and Weibull; bounded distributions such as uniform, inverse sine, Bates, Coomaraswamy, power, triangular, uniform and, Mises, Wigner, and Pastur; heavy-tailed distributions such as Pareto, Beta prime, prime, Dagum, Davis, SingMaddalala, Cauchy, Landau, Levy, Benini, Gibrat, and Suzuki distributions; quantile-based distributions, such as Tukey and Wakeby; systematic distributions, such as Pearson; multivariate continuous distributions, such as multivariate normal, multivariate multinomial, multivariate-t, Dirichlet, and Coula; and multivariate discrete distributions, such as multivariate hypergeometric, negative multinomial, and multivariate Poisson.

[0071] For example, the random DNN block uses a heterogeneous distribution of different beliefs for at least two disjoint sets of random nodes within a layer; for example, the first 70% of the random nodes at the second layer follow a logistic distribution, and the remaining 30% of the random nodes at the same layer follow a Cauchy distribution. For another example, the random DNN block uses unmatched pairs of posterior beliefs and prior beliefs for at least one set of random nodes; for example, the random nodes at the third layer use a uniform distribution for the posterior beliefs, while the same random nodes use a Laplace distribution for the prior beliefs. In addition, for some embodiments, when analyzing a new available set of data sets, although the same random DNN block was previously trained with different irregular beliefs, the selection of irregular beliefs is dynamically and adaptively modified to use different probability distributions for variational sampling at random nodes in the random DNN block. For example, the random nodes are trained to follow a normal distribution for the training data set, while the same random nodes use a logistic distribution for the test data set. This allows further adaptation to reduce model mismatch in unseen test data sets.

[0072] The stochastic DNN block provides probabilistic inference to analyze a dataset by using importance-weighted accumulation after variational sampling at random nodes according to irregular beliefs. Variational sampling uses a random number generator based on a reparameterization trick of variational parameters such as location, scale, shape, and temperature specified by irregular beliefs. To optimize the stochastic DNN block, some embodiments perform the following steps: explore different values ​​of irregular beliefs and hyperparameters by using a hypergradient method; configure the stochastic DNN block by modifying the connectivity between multidimensional signals, task labels, and random nodes according to the hyperparameters; compute a loss function by forward propagating the dataset across the stochastic DNN block according to probabilistic inference; modify the loss function by regularizing the random nodes according to a discrepancy metric; backpropagate the gradient of the loss function with respect to trainable parameters; and update the trainable parameters using a gradient method.

[0073] The steps mainly include two inference and optimization methods. The inference method for the random DNN model adopts feeding the data signal into a DNN with multiple layers and random nodes, propagating the signal according to the layer, while the random nodes use variational sampling according to irregular beliefs, and accumulating the output of the DNN as the probabilistic inference result. On top of the inference method, the optimization method also adopts a loss function calculated based on the variational boundary to regularize the random nodes according to the difference metric, back-propagate the gradient of the loss function with respect to the trainable parameters, update the trainable parameters using the gradient method, and explore different values ​​of irregular beliefs and hyperparameters using the super gradient method. The super gradient method enables the efficient configuration of high-performance random DNN models on conventional variational inference models that rely on conventional beliefs with isomorphic allocations and matching allocations for random nodes.

[0074] Auxiliary Regularization Module

[0075] Figure 1B and Figure 1C An example inference model for predicting Y given data X under latent variables Z and nuisance factors S according to some embodiments is shown. The DNN model uses a main pipeline and is decomposed into an encoder part 101 and a classifier part 102 (or a regressor part for regression tasks), where the encoder part 101 extracts feature vectors from data X as latent variables Z, and the classifier part 102 predicts the task label Y from the latent variable Z. For example, the latent variable Z is a vector of hidden nodes in the middle layer of the DNN model. In order to stabilize the training convergence of the DNN model, some embodiments use regularization techniques for the latent variable Z. In addition to the main pipeline of the encoder 101 and the classifier 102, Figure 1B An additional auxiliary regularization module is used to regularize the latent variable Z. Specifically, the decoder DNN block 103 is attached with the latent variable Z to reconstruct the original data X using the optional conditional information of the nuisance variation S. This conditional decoder 103 can facilitate the cleaning of the nuisance domain information S and the latent variable v. For example, the nuisance domain variation S includes the object identification (ID) of the physiological data set, the measurement session ID, the noise level, the height / weight / age information of the object, etc. For some embodiments, by cleaning those nuisance factors S from the latent variable Z, the present invention can achieve a subject-invariant generalized human-computer interface without a long calibration session. The auxiliary DNN block referred to as the decoder 103 is trained to minimize another loss term, such as the mean square error and the Gaussian negative log-likelihood loss to reproduce X from Z.

[0076] For some embodiments, the potential change is further decomposed into multiple potential factors, such as Z1, Z2, ..., Z L , each latent factor consists of a set of nuisance factors S1, S2, ..., S N Regularize individually. In addition, depending on the dataset, some nuisance factors are partially known or unknown. For known nuisance factor labels, the DNN block can be trained in a supervised manner, while it requires a semi-supervised approach for unlabeled nuisance factors. For the semi-supervised case, pseudo-labeling based on variational sampling of all potential labels for nuisance factors is used in some embodiments, for example, based on a reparameterization technique that uses a variational temperature parameter to control the sharpness of nearly one-hot samples of classification nuisance factors, such as the Gumbel softmax technique. For continuous nuisance factors, variational sampling is based on a reparameterization technique that uses variational parameters including position, scale, and shape to determine a probability distribution. For example, a portion of the data in the dataset (object age information) is missing, while the rest of the data has age information to be used for supervised regularization.

[0077] Figure 1B The DNN block in has another auxiliary regularization module 104 attached to the latent variable Z to estimate the nuisance variation S. This regularized DNN block 104 is used to further facilitate the cleaning of the nuisance factor to be robust and is often referred to as an adversarial network because the regularized DNN block is trained to minimize the loss function to estimate S from Z while the mainstream pipeline DNN block is trained to maximize the loss function to censor nuisance information. The adversarial block is trained in an alternating manner, where the associated hyperparameters include adversarial coefficients, adversarial learning rates, intervals of adversarial alternations, and architectural specifications.

[0078] Figure 1B The graphical model in is called an adversarial conditional variational autoencoder (A-CVAE) for unsupervised feature extraction in conjunction with a downstream task classifier 102. The graphical model of A-CVAE has various graph nodes and graph edges to represent connectivity covering random variable factors X, Y, Z, and S. With the regularization block, the A-CVAE model has higher robustness to nuisance factors. Therefore, the auxiliary regularization module is used as a so-called pre-shot transfer learning or domain generalization technique to make the AI ​​model robust to unseen nuisance factors.

[0079] The present invention provides a way to efficiently assign multiple different beliefs to each random node Z1, Z2, ..., Zn. If there are 10 different posterior-prior pairs for 100 random nodes, there are at most 10 100 potential combinations to specify a random DNN. Some embodiments use an automated machine learning (AutoML) framework based on super gradient methods to facilitate finding better combinations of such heterogeneous pairings of posterior / prior beliefs. In addition, some embodiments allow different difference measures for each random node. For example, Rayleigh's α-divergence of order 0.1 is used for Z1, Z2 uses an α order of 0.2, and KLD is used for Z3. The search space grows rapidly, and the present invention explores random models that allow such irregularities by using AutoML. The super gradient method for AutoML is based on a combination of reinforcement learning, implicit gradients, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, simulated annealing, and Bayesian optimization to efficiently search for better combinations of irregular beliefs for posterior distributions, prior distributions, and likelihood distributions, as well as difference measures for individual random nodes simultaneously.

[0080] Architecture Exploration

[0081] There are many possible ways to connect the encoder 101, the classifier 102, the decoder 103, and the adversarial network block 104. For example, Figure 1CAn example DNN block is shown, in which there is another auxiliary module 105 to estimate the nuisance factor S from the data X. Since the possible number of achievable DNN connections explodes rapidly with the size of the AI ​​model, it is necessary to efficiently build reasonable AI models. In addition, randomly connected DNN blocks are often useless and unreasonable. For some embodiments, graph connectivity is explored by using an automatic Bayesian graph exploration method called AutoBayes. At the core of AutoBayes is to consider a graphical Bayesian model that captures the probabilistic relationship between random variables representing data features X, task labels Y, nuisance variation labels S, and latent representation Z. The main goal is to infer the task label Y from the measured data features X, which is hindered by the presence of nuisance variations (e.g., object / session changes) that are partially labeled by S. The latent representation Z (and as needed also consists of Z1, Z L , …, Z L Representation) is also optionally introduced into the AI ​​model to help capture the underlying relationship between S, X, and Y.

[0082] Our four factors for the random variables Y, S, Z, and X represent the joint probability distribution underlying the dataset. The chain rule of probability calculations can yield the following factorization from Y to X for the generative model:

[0083] p(y,s,z,x)=p(y)p(s|y)p(z|s,y)p(x|z,s,y)

[0084] It is a full-chain Bayesian graph. The probability conditional on X can be factored, for example, as follows:

[0085]

[0086] They are marginalized to obtain the likelihood of task class Y given data X. When more nodes with multiple nuisance and latent variables are considered, the number of possible Bayesian and inference graphs increases rapidly.

[0087] The above graphical models do not impose any assumptions about the underlying inherent independence in the dataset and are therefore the most general. However, depending on the underlying independence in the dataset, we may be able to prune some of the edges in those graphs. For example, if the data has a Markov chain of YX that is independent of S and Z, then it automatically yields Figure 1A AI models shown. This means that the most complex inference models with high degrees of freedom do not always perform optimally across arbitrary datasets. We are encouraged to consider extending the AutoML framework to automatically explore the best pair of inference factor graphs and corresponding Bayesian graph models that match the dataset in addition to other hyperparameters.

[0088] AutoBayes begins by exploring any potential Bayesian graph by cutting links in the full-chain graph, imposing possible independence. We then employ the Bayesian Sphere algorithm on each hypothesized Bayesian graph to check conditional independence on different inference strategies. The Bayesian Sphere algorithm justifies reasonable pruning of links in the full-chain inference graph and also proves potential adversary censoring when Z is independent of S. This process automatically constructs connectivity of inference blocks, generation blocks, and adversary blocks with good reasoning, for example to construct Figure 1B The A-CVAE classifier in Figure 1C Any extended model in .

[0089] Example Stochastic DNN Model

[0090] Figure 2 An exemplary schematic diagram of an automated VAE according to some embodiments is shown. Although generalized variational inference (GVI) provides large degrees of freedom, it is difficult to design the correct choice of probabilistic model in addition to other architectural hyperparameters. To solve this problem, the present invention uses automated construction of random DNN models. Figure 2 Such an example of an automated VAE (also called AutoVAE) is shown, which motivates finding appropriate choices of posterior, prior, likelihood, and difference metrics by using an automated machine learning (AutoML) framework based on super gradient methods. Although the normal distribution is often used for potential sampling in random nodes of standard VAEs, it is not always optimal when the underlying uncertainty in the dataset is complex or unknown. The present invention provides a way to improve VAE models by allowing irregular beliefs for random nodes. For example, when the logistic distribution is used as the posterior belief at the VAE encoder despite the VAE decoder assuming a normal distribution as the prior belief, the variational Rayleigh (VR) bound can be improved by exploring various settings of irregular beliefs over mismatched pairings.

[0091] The VAE uses an encoder block 201 and a decoder block 203 to reproduce the data signal from the latent variables 202. The last node of the VAE encoder is a random node that provides variational parameters to follow a specific probability distribution of a given data signal. Therefore, the VAE encoder represents a posterior belief specified by variational parameters such as location, scale, and shape. For example, the first half of the output nodes of the VAE encoder are location parameters, and the second half of the output nodes of the VAE encoder are scale parameters. The latent variables 202 are then sampled by a random number generator to follow the posterior belief through a reparameterization trick 205, where random numbers sampled from a standard distribution are scaled and transformed by the variational parameters generated by the VAE encoder.

[0092] The VAE decoder employs the sampled latent variables 202 to generate the estimated data signal. The final node of the VAE decoder is a random node that provides variational parameters to follow a specific probability distribution given a latent sample. Thus, the VAE decoder represents a likelihood belief 208 specified by variational parameters such as location, scale, and shape. For example, the output node of the VAE decoder is the location parameter of a normal distribution, while the scale parameter is an unspecified constant, i.e., the generalized NLL loss reduces to the MSE loss.

[0093] The VAE encoder and decoder are jointly trained by gradient methods so that the evidence lowerbound (ELBO) is maximized, where the evidence lower bound is a function of the generalized NLL loss and the difference 207 between the posterior belief and the prior belief 206. The difference term is used as a regularization of the random nodes to control the probability distribution. Once the VAE encoder and decoder are trained, the VAE decoder can be used as a generative model to generate a synthetic signal close to the original data set by feeding random numbers sampled from the prior belief. Some embodiments of the present invention use on-the-fly adaptation of prior beliefs to sample latent variables from different probability distributions between the training data set and the test data set.

[0094] AutoVAE uses the AutoML framework 204 to achieve the highest ELBO by exploring various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and difference metrics using a super gradient method. The AutoVAE system implements various irregular pairings for random nodes: for example, i) matching pairings use the same probability distribution for posterior beliefs and prior beliefs; ii) mismatched pairings use different probability distributions for posterior beliefs and prior beliefs; iii) heterogeneous pairings use different probability distributions for posterior beliefs and prior beliefs in individual random nodes non-uniformly. Figure 3 Exemplary variations of irregular beliefs are shown in accordance with some embodiments. Figure 3 The first variant 301 in is the case when the posterior belief and the prior belief are matched to a Laplace distribution. Figure 3 The second variation 302 in is the case where the posterior belief using the Laplace distribution does not match the prior belief using the normal distribution. Figure 3 The third variant 303 in is the case where the random nodes are heterogeneous, specifically, the first half of the latent variables use different posterior-prior pairs than the second half of the latent variables. The use of such irregular beliefs provides the following benefits: the expressive power of the random DNN model can be greatly enhanced, and thus more complex statistics can be efficiently modeled. In addition, the standard KLD regularization can be significantly extended by exploring the difference measures on Rayleigh's alpha divergence, beta divergence, gamma divergence, etc.

[0095] Figure 4A Exemplary beliefs of posterior distribution, prior distribution and likelihood distribution for random DNN according to some embodiments are shown. The corresponding symbols of various random distributions and expressions of their probability density functions (PDF) are listed for convention. Some distributions are specified with a single variational parameter (e.g., scale for exponential distribution). Some distributions are specified with two variational parameters (e.g., location and scale of Laplace distribution). For some embodiments, more than 2 variational parameters of shape are also used for other distributions, including but not limited to skewed normal distribution, exponential power distribution, hyperbolic distribution, gamma distribution, Johnson distribution, Pearson distribution, Student-t distribution and Pareto distribution. Some distributions belong to the location-scale family (LSF), which is suitable for variational sampling because non-standard distributions can be easily generated by standard distributions using reparameterization techniques. For discrete probability distributions such as Gumbel distribution, a modified reparameterization technique called softmax technique is used together with a variational parameter called temperature to adjust sharpness toward one-hot encoded samples.

[0096] As an example, let x∈R N is the N-dimensional data input to the VAE encoder. The encoder generates variational parameters such as the location, scale, and shape of the posterior belief hypothesis to determine the L-dimensional latent variable z∈R L The latent variables are then fed into the VAE decoder to generate the variational parameters of the likelihood belief hypothesis to determine the reconstructed data x′∈R N The encoder and decoder are configured with parameterized DNNs, mapping q φ : x→z and p φ : z→x', φ and are the trainable parameters of the DNN, such as weights and biases. The DNN attempts to minimize the reconstruction loss, which will typically be the negative log-likelihood (NLL) under the likelihood belief assumption.

[0097] For parameters Given a choice of and ψ, the VAE encoder and decoder models imply what are called the posterior The conditional distribution of ψ Let π(z) be the probability distribution of the latent variable z under the assumption of prior beliefs. VAE attempts to maximize the marginal distribution Pr(x), which is given by:

[0098] Pr(x)=∫p ψ (x|z)π(z)dz

[0099] It is usually difficult to calculate accurately. Although the integral with the samples of z can be approximated, the key to the VAE method is to utilize the generative model p through the VAE decoder ψ The posterior implied by (x|z) The variational lower bound of q ψ (z|x) denotes the variational approximation of the posterior using the VAE encoder, and the evidence lower bound (ELBO) is given by:

[0100]

[0101] Among them, D KL (Q|Π) denotes KLD, which measures the difference between posterior and prior beliefs and is defined as:

[0102]

[0103] The VAE encoder and decoder are jointly trained such that ELBO is maximized under a variational Bayesian framework.

[0104] In ELBO, there are four important factors to specify: likelihood belief P = p ψ (x|z); posterior belief Q = q φ (z|x); prior belief Π = π(z); and difference measure D = D KL (.||.). Standard VAEs typically use a normal distribution for the likelihood belief P = N(λ,γ) (or a Bernoulli distribution for almost binary image reconstruction), and a specific KLD D KL (Q||Π) to regularize the latent variables by assuming a normal posterior Q=N(mu, sigma) and a normal prior Π=N(0,1). The present invention is based on GVI to allow any arbitrary choice of generalized NLL loss, difference measure, prior belief and posterior belief. Standard VAE is asymptotically optimal for infinite dimensions, but the optimality is no longer justified when the likelihood belief / posterior belief / prior belief is misspecified or unspecified instead of compared to the true data distribution.

[0105] Although under some conditions, exponential family prior beliefs will lead to the same exponential family posterior beliefs, the present invention provides an extended GVI by considering irregular beliefs, such as mismatched pairings of posterior-prior beliefs (e.g., logistic posterior and Laplace prior). In addition, the generalized VAE allows another irregular belief to be used with heterogeneous pairings, for example, 30% of potential nodes use Laplace-normal pairs and 70% of nodes uniform-Cauchy pairs. Although non-Gaussian beliefs such as Laplace distribution and Cauchy distribution have been considered in some prior arts, irregular beliefs with mismatched or heterogeneous pairings have not been considered. In order to design such irregular VAEs, the present invention provides a set of convenient pairings with closed-form differentiable expressions and direct reparameterization of KLD without the need for high complexity like information autoregressive flow (IAF) and Gibbs sampling.

[0106] In the prior art, the normal distribution is the most widely used belief for random DNNs, allowing simple sampling, reparameterization techniques, and closed-form expressions for KLD. In some embodiments of the present disclosure, we consider various alternatives such as the location-scale family LSF(μ, σ), which maintains the same distribution within a transformation with variational parameters of location μ∈R and scale σ∈R+. For example, given a random variable ε drawn as ε~LSF(0,1), its transformed random variable Z=μ+σ·ε follows the same family as Z~LSF(μ,σ). Here, the symbol "~" means "distributed to". This transformation is called the reparameterization technique for variational sampling (one of the key implementation methods for random DNN training) to back-propagate the gradients of the variational parameters μ and σ while maintaining the target distribution in forward variational sampling.

[0107] like Figure 4A As shown in the exemplary list of , the present disclosure considers different belief sets that include LSF distributions: Normal N; Laplace La; Cauchy C; Logistic Lo; Uniform U; Gumbel G; and Exponential E (which is not an LSF but a family of scales). Figure 4B An exemplary standard position-scale family for variational sampling based on the reparameterization trick is shown in accordance with some embodiments. Figure 4BSome PDFs of their standard form LSF(0,1) are plotted in . Due to the simple reparameterization trick 400 of the transformation of the standard random sample LSF(0,1)403 by exploiting two variational parameters of position 401 and scale 402, these distributions can be used as a choice of latent posterior beliefs Q to generate the target non-standard distribution of LSF(μ,σ)404 for the VAE encoder. For example, for ε=log(V / W)~Lo(0,1), and V,W~E(1), sampling the logistic distribution can be performed as Z=μ+σε~Lo(μ,σ). The position vector μ∈R L and the scale vector σ∈R L The parameter VAE encoder is generated from the data x, while the potential z is transformed from a standard random variable ε to LSF(0,1) L Random sampling.

[0108] For the choice of prior belief Π=π(z) used in the generative model with VAE decoder, for a particular choice of posterior belief Q=LSF(μ,σ), we typically use a standard normal distribution N(0,1), or a matching standard prior Π=LSF(0,1). The random nodes are then regularized by a difference metric such as KLD to achieve a posterior distribution close to the prior distribution. Figure 5A An exemplary landscape of KLD differences for various posterior belief and prior belief pairs at two independent random nodes according to some embodiments is shown. For example, the KLD landscape for the normal posterior and normal prior belief pair 501 is circularly isotropically symmetric on any axis to penalize samples with larger amplitudes to behave like a Gaussian distribution. For the Laplace posterior and Laplace prior belief pair 502, the KDL landscape becomes non-anisotropic to become a rounded square, which promotes sparse regularization. For another Cauchy posterior and Cauchy prior belief pair 503, the KLD landscape is shaped like a rounded cross, which typically promotes even sparser regularization.

[0109] Although we can choose any arbitrary distribution for the prior belief regardless of the posterior belief, it is desirable to have a closed-form simple expression for KLD to measure the difference between the posterior Q and the prior Π. In some embodiments, we consider such a posterior belief and prior belief pair. Figure 5B An exemplary pair of posterior beliefs and prior beliefs quantified by the KLD difference metric according to some embodiments is shown. Figure 5B In , closed-form expressions of KLD difference measures for some posterior belief and prior belief pairs are listed for convention. For some embodiments, KLD is also obtained by numerical integration through parity sampling and accumulation for other irregular beliefs.

[0110] Yet another embodiment uses on-the-fly adaptive prior modification after training the VAE. For example, the distribution of latent variables of the optimized VAE encoder may end up having a different distribution than the target prior distribution. To address the mismatch, when new data is synthetically generated, the prior beliefs are modified based on some statistics (such as the mean, variance, and skewness of the latent variables) to match the new available dataset.

[0111] While matched posterior-prior pairs usually have smaller KLDs, unmatched pairs are not necessarily worse than matched posterior-prior pairs. Figure 5C Exemplary KLD properties of various posterior beliefs and prior belief pairs as a function of position value according to some embodiments are shown. For example, unmatched logistic-Laplacian belief pairs have slightly smaller KLD than matched Laplace-Laplacian belief pairs at moderate position values. More importantly, those unmatched cases have sufficiently small KLD values ​​for a wide range of positions μ. Because the KLD value cannot be exactly zero unless μ=0 and σ=1, minimizing the KLD term is not necessarily important even for matching pairs. This means that, due to the higher degrees of freedom in adjusting the KLD landscape and ELBO, exploring different irregular pairings may have a higher chance of providing a better ELBO in the end than conventional regular pairings. The present invention is based on this recognition.

[0112] Note that KLD is the most commonly used difference metric to evaluate the "difference" between the posterior Q and the prior Π for VAE. However, in addition to KLD, various other difference metrics can be used for stochastic DNNs, including Fisher divergence, Jeffrey divergence, Rayleigh's α-divergence, β-divergence, and γ-divergence. In particular, Rayleigh's α-divergence is useful because it covers many variants such as importance-weighted autoencoders (IWAE) and standard VAEs as a special case when adjusting the divergence order α. Figure 6 An exemplary difference measure based on Rayleigh's α-divergence is shown, covering variants of generalized variational inference including importance-weighted autoencoders (IWAE) and standard VAEs, according to some embodiments. For example, when α→1, the α-divergence D α (Q / / Π) is reduced to KLD D KL (Q / / Π). For another example, the α divergence is reduced to a variant of the Hellinger distance and the chi-square divergence. The Rayleigh divergence of order α ≥ 0 is expressed as

[0113]

[0114] More importantly, we show that the Rayleigh divergence-based ELBO has a tighter bound than the KLD case (α→1). Specifically, by K-fold variational samples The importance-weighted accumulation of is used to approximate the variational Rayleigh (VR) boundary as follows:

[0115]

[0116] IWAE is a special case of the VR framework when α = 0, and it converges to the marginal probability when K → ∞. Since the α divergence is closely related to KLD, Figure 5B The above pair in also has a closed form expression for the α-divergence. Therefore, some embodiments use Rayleigh's α-divergence as a generalized difference measure, where the Rayleigh order α is adjusted to find the optimal ELBO.

[0117] For VAE, any differentiable loss metric can actually be used as the reconstruction loss; for example, mean squared error (MSE), mean-absolute error (MAE), and binary cross-entropy (BCE) in addition to NLL. However, most loss functions are limited to a certain likelihood belief P = p ψ (x|z) is closely related to generalized NLL to represent a generative model of data via a VAE decoder. Figure 7 An exemplary likelihood belief P for a stochastic DNN determining a generalized negative log-likelihood loss is shown in accordance with some embodiments. Figure 7 Various likelihood beliefs P for random DNNs are listed in , where generalized NLL losses are specified for different likelihood beliefs. For example, BCE is commonly used for almost binary images, such as the MNIST dataset. BCE is equivalent to NLL under a likelihood belief based on a Bernoulli distribution B(λ). Some embodiments use a beta distribution Be as the appropriate likelihood belief, while some embodiments use its modified belief, called a continuous Bernoulli distribution CB, to improve VAEs for non-strictly binary images. When the unspecified variance is omitted, the MSE corresponds to the NLL under the normal distribution likelihood N(x', *). When the unspecified scale is omitted, the MAE corresponds to the NLL under the Laplace distribution likelihood La{x', *).

[0118] The VAE decoder may require multiple variational outputs (such as location, scale, and shape) to generate x' given a likelihood belief P. For example, the normal distribution likelihood N(λ, γ) provides the mean as the reconstructed data x'=λ and its standard deviation γ as confidence. Similarly, some embodiments use x'=λ for the Laplace distribution in the sense of maximum likelihood. For another example, the beta distribution as a likelihood belief also has two variational parameters to be generated by the VAE decoder. Typically, given the decoder output of the variational parameters (λ, γ, etc.), data reconstruction should be done by its mode (i.e., the peak of the PDF). Although for the Bernoulli likelihood B(λ), the mode is binary, because x'=0 or 1 depends on λ, some embodiments use the mean instead to reconstruct x'=λ. For some embodiments, the output of the random DNN after multiple variational samplings is accumulated with the weighted mean to improve the accuracy of the variational inference.

[0119] VAE and its variants of random DNN models require the specification of four random factors: posterior belief Q; prior belief Π; likelihood belief P; and difference measure D. To design these factors, we usually need to manually search for Figure 7 The different plausible beliefs listed in Figure 5B The different posterior-prior pairs listed in Figure 6 The search space grows rapidly when we consider irregular, uneven, and mismatched pairings at each potential node. This paper facilitates the design of such heterogeneous random DNNs by using the AutoML framework to explore those factors of VAE design in an automatic manner, such as Figure 2 As shown - we call this concept AutoVAE.

[0120] Example Model Implementation

[0121] When exploring the optimal divergence order α, uncommon irregular pairings of posterior-prior beliefs such as logistic-normal or Laplace-normal may outperform the common choice of normal-normal pairs. The irregular heterogeneous pairings explored in AutoVAE can further achieve better performance due to higher degrees of freedom. For example, as an exemplary model implementation, when analyzing the real benchmark dataset of MNIST, for 50%, 30% and 20% of the latent variables, the posterior-prior beliefs are mixed by AutoVAE in a heterogeneous manner into logistic-normal (Lo / / N), Laplace-normal (La / / N) and Laplace-Laplacian (La / / La), respectively, to achieve higher ELBO performance than the standard VAE.

[0122] Some embodiments use a simple VAE architecture based on a multi-layer perceptron (MLP) for the encoder block and the decoder block, where the MLP consists of several fully connected linear layers and hundreds of hidden nodes have rectified linear units (ReLU) activations. For example, a two-layer MLP with 400 hidden nodes is used for the VAE encoder and the VAE decoder. For some embodiments, the number of potential nodes is chosen to be about tens, depending on the dataset. For example, L=20 random nodes are used. Some embodiments use a gradient method based on adaptive momentum gradient optimization, where the learning rate is 0.01 for 100 epochs at a mini-batch size of 1000. AutoML can also design such hyperparameters on top of irregular beliefs by using hyper-gradient methods such as Bayesian optimization.

[0123] For some embodiments, once the VAE is trained, the VAE decoder can be used as a generative model to reproduce synthetic data by feeding random samples provided from the prior belief Π. For example, the generated images are evaluated by the Frechet inception distance (FID) and the kernel inception distance (KID) to evaluate their natural distribution according to the original image dataset. To evaluate the initial score, some embodiments generate 50,000 images by sampling random latent variables z from the prior belief ΠΠ.

[0124] Fig. 8A exemplarily characterizes various combinations of posterior beliefs, prior beliefs, likelihood beliefs, and difference measures for a generalized VAE model according to some embodiments. More specifically, Fig. 8A Results of unmatched VAEs under various combinations of likelihood beliefs, posterior beliefs, and prior beliefs after training on the benchmark MNIST dataset are shown. For the MSE loss of likelihood beliefs based on an unspecified normal distribution P = N (λ, *), the ELBO performance of conventional normal-normal pairs can be improved by exploring different pairings (such as logistic-normal, Laplace-Laplacian, and logistic-Laplacian pairs) (for an order of α = 1 and an accumulation of K = 1). MSE and initial scores are also improved, for example, by irregular beliefs of logistic-normal pairs. When exploring different likelihood beliefs, we can even more clearly realize the benefits of unmatched pairings. In addition, the irregular uneven use of posterior-prior pairs at various potential nodes (denoted as "automatic") can further improve performance in some cases by automatically exploring mixed pairs for irregular beliefs.

[0125] In addition, exploring the Rayleigh order α can further improve the VR boundary. Figure 8B exemplarily ELBO properties of various posterior belief and prior belief pairs as a function of the divergence order of a generalized VAE model according to some embodiments. Specifically, Figure 8B The benefits of exploring different dissimilarity metrics by adjusting the divergence order α are shown for some embodiments. Figure 8B It is shown that uncommon pairings of posterior beliefs and prior beliefs such as logistic-normal or Laplace-normal may outperform the common choice of normal-normal pairs when exploring the optimal divergence order α. It is also confirmed that the irregular belief pairings explored in AutoVAE achieve the best performance on the MNIST dataset. For 50%, 30%, and 20% of L=20 potential nodes at the Rayleigh divergence order of α=0.5, the best posterior-prior mixtures in heterogeneous distribution are logistic-normal (Lo / / N), Laplace-normal (La / / N), and Laplace-Laplace (La / / La) pairs. This exemplary result clearly shows the benefits of the present invention, which uses irregular beliefs for stochastic DNN models by allowing mismatched pairings and heterogeneous pairings with adjustable difference metrics.

[0126] Figure 8C and Fig.8D 1 shows example image snapshots generated by a generalized VAE decoder using variational sampling according to prior beliefs after training with various posterior beliefs and prior belief pairs in accordance with some embodiments. Specifically, Figure 8C and Fig.8D Snapshots of synthetic images randomly generated by using a trained VAE decoder for the MNIST dataset are shown. By exploring different combinations of posterior beliefs, prior beliefs, likelihood beliefs, and difference metrics, the generator model can be significantly improved, especially by utilizing heterogeneous pairings designed by the AutoML framework. Similar benefits can be achieved for different datasets. The present invention uses mismatched, heterogeneous, and irregular combinations of posterior-prior-likelihood-beliefs for generalized VAE design. By using automated exploration of irregular beliefs, mismatched VAEs can outperform standard VAEs. In addition, for some embodiments, the systems and methods of the present invention explore different difference metrics by adjusting the divergence order. The concept of AutoVAE can facilitate the search for appropriate combinations of those random factors to configure a generalized VAE model for variational inference. The irregular belief pairing technique is also applicable to other random DNNs besides VAE.

[0127] Example Stochastic DNN Model

[0128] In addition to VAE, for some embodiments, random DNN models include, but are not limited to, variational information bottleneck (VIB), denoising diffusion probability model (DDPM), and variational Bayesian neural network. VIB uses a similar but different approach to VAE, where VIB is suitable for supervised learning and VAE is suitable for unsupervised learning. The random nodes of VIB are specified by irregular beliefs in a manner similar to AutoVAE, so some embodiments use automatic exploration of irregular beliefs at VIB to improve performance. DDPM is another random DNN model similar to VAE, but with a large amount of variational sampling in the diffusion step. Similarly, AutoVAE, some embodiments use automatic exploration of irregular beliefs to achieve mismatches and heterogeneous pairings at random nodes at each diffusion step. Similarly, in addition to VAE, VIB and DDPM, some embodiments use different random DNN models when involving random nodes under specific assumptions of underlying probabilistic beliefs.

[0129] Fig.9A An exemplary stochastic DNN is shown that sequentially uses multiple different beliefs and variance metrics at multiple intermediate layers in accordance with some embodiments. Fig.9A The exemplary random DNN model in obtains data x901 through multiple DNN blocks in sequence to generate variational inference of task label y 905, some of which are random nodes using variational sampling according to individually specified beliefs. For example, the first random node z1 902 uses a normal distribution for posterior beliefs and a uniform distribution for prior beliefs with KLD differences; the second random node z2 903 uses a Cauchy distribution for posterior beliefs and prior beliefs with a Rayleigh α-divergence of order 0.5; the third random node z3 904 uses a logistic distribution for posterior beliefs and a normal distribution for prior beliefs with a β-divergence of order 0.2. Such a random DNN with multiple random layers in sequence provides a more complex uncertainty model, and heterogeneous beliefs can further improve the ability of random modeling for variational inference.

[0130] Fig. 9BAn exemplary random DNN using different beliefs and divergence metrics in parallel at different random nodes within an intermediate layer and sequentially at multiple intermediate layers according to some embodiments is shown. The exemplary random DNN model implements heterogeneous beliefs within a random layer to provide variational inference y 915 from data x 911 through multiple DNN blocks having parallel random nodes z1 912 and z2 913 in addition to sequential z3 914. Specifically, the parallel random nodes z1 912 and z2 913 are specified differently by different beliefs and divergence metrics. For some embodiments, super gradient methods are utilized to explore many other possibilities of connecting multiple DNN blocks with multiple random nodes to achieve a high-performance generalized variational inference model.

[0131] Each stochastic DNN block is configured with hyperparameters to specify a set of layers with neuron nodes, interconnected with trainable variables to sequentially pass signals from layer to layer. The trainable variables are numerically optimized using gradient methods such as stochastic gradient descent, adaptive momentum, Ada gradient, Ada bound, Nesterov accelerated gradient, and root mean square propagation. The gradient method updates the trainable parameters of the DNN block by using the training data so that the output of the DNN block provides a smaller loss value, such as mean square error, cross entropy, structural similarity, negative log likelihood, absolute error, mutual covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, and triple loss. Multiple loss functions are further weighted to be combined with some regularization coefficients according to the training scheduling strategy.

[0132] In some embodiments, the random DNN block is reconfigurable according to hyperparameters such that the DNN block is configured with a set of fully connected layers, convolutional layers, graph convolutional layers, recurrent layers, recurrent connections, skip connections, and initial layers, and a set of nonlinear activations includes modified linear variants, hyperbolic tangents, sigmoids, gated linears, softmax, and thresholds. The DNN block is further regularized with a set of dropouts, swap-outs, ignores, occlusions, drop connections, noise injections, jitter, and batch normalizations. In yet another embodiment, the layer parameters are further quantized to reduce the size of the memory specified by the adjustable hyperparameters. For another embodiment of link cascading, the system uses multi-dimensional tensor projections with dimensionally trainable linear filters to convert lower dimensional signals into larger dimensional signals for dimensionally mismatched links.

[0133] Another embodiment integrates AutoML into AutoBayes and AutoTransfer for hyperparameter exploration and learning scheduling for each DNN block. Here, AutoTransfer uses different auxiliary regularization modules to explore from a different set of regularization methods, including but not limited to adversarial review, mutual information gradient estimation, and Wyner distance. Note that AutoTransfer and AutoBayes can be easily integrated with AutoML based on super gradient methods to optimize any hyperparameters of various DNN blocks. More specifically, the system modifies hyperparameters by using reinforcement learning, evolutionary strategies, differential evolution, particle swarms, genetic algorithms, annealing, Bayesian optimization, hyperbands, and multi-objective Lamarckian evolution to explore different combinations of discrete hyperparameter values ​​and continuous hyperparameter values.

[0134] The system of the present invention also provides a further testing step to adapt as a post-training step, which refines the trained DNN block by unfreezing some trainable variables so that the DNN block can be robust to new data sets (such as new subjects) with new nuisance variations. This embodiment can reduce the calibration time requirements for new users of the HMI system.

[0135] Example System

[0136] Fig.10 An exemplary schematic diagram of a system configured with a processor, a memory, and an interface according to some embodiments is shown. Specifically, Fig.101 is a block diagram illustrating an example of a system 1000 for signal analysis using automatic configuration of a random DNN. The system 1000 includes a device 1100 having a set of interfaces and data links 1105 configured to receive and send signals, at least one processor 1120, a memory (or a set of memory banks) 1130, and a storage device 1140. The processor 1120 executes computer executable programs and algorithms stored in the memory bank 1130 and the storage device 1140 in conjunction with the memory bank 1130. The set of interfaces and data links 1105 includes a human-machine interface (HMI) 1110 and a network interface controller 1150. The processor 1120 can execute computer executable programs and algorithms in conjunction with the memory bank 1130, and the memory bank 1130 uploads computer executable programs and algorithms from the storage device 1140. The computer executable programs and algorithms stored in the storage device 1140 use a random deep neural network (DNN) 1141, trainable parameters 1142, hyperparameters 1143, a scheduling policy 1144, forward pass signal / backward-pass gradient and other temporary caches 1445, a regularization module 1146, a gradient method for training 1147, a super gradient method for AutoML 1148, and an irregular belief 1149 set for random nodes. The processor performs probabilistic inference using the random DNN 1141 to analyze the data set, wherein the probabilistic inference uses importance-weighted accumulation after variational sampling at random nodes according to the irregular belief 1149 set.

[0137] The system 1000 receives signals via a set of interfaces and data links 1105 through a network 1190. The signals include a dataset set 1195 having data for training, validation, and testing. The set of datasets includes a multidimensional signal set X associated with a task label Y to be identified. For some embodiments, the dataset set also includes other auxiliary information, such as a nuisance variation S.

[0138] In some cases, each of the stochastic DNN blocks 1141 is configured to encode the multidimensional signal X into a latent variable Z, decode the latent variable Z to reconstruct the multidimensional signal X, classify the task label Y, estimate the nuisance variation S, regularize the latent space by estimating the nuisance variation S, or select a graphical model. The memory bank also includes intermediate neuron signals, and temporary calculation values ​​including forward-propagated signals and backward-propagated gradients.

[0139] The processor 1120 is configured to submit the signal and the data set 1195 to the random DNN block 1141 in conjunction with the interface and the storage body 1130 to perform exploration of different irregular beliefs 1149 and different hyperparameters 1143 by using the hyper gradient method 1148. The processor 1120 also performs: configuring the random DNN 1141 according to the hyperparameters 1143; calculating the loss function by forward propagating the data set 1195 across the random DNN 1141 according to the probabilistic inference; modifying the loss function by regularizing the random nodes according to the difference metric specified in the irregular belief 1149; back-propagating the gradient of the loss function with respect to the trainable parameters 1142 across the random DNN 1141 to update the trainable parameters 1142 using the gradient method 1147. The super gradient method for AutoML 1148 also explores different auxiliary regularization modules 1146, different hyperparameters 1143, different scheduling strategies 1144, and irregular random beliefs 1149 to improve robustness against nuisance variation and unspecified uncertainty in the dataset 1195.

[0140] For some embodiments, the system 1000 is applied to design HMI by analyzing the physiological data of the user. The system 1000 receives signals of physiological data 1195 via a network 1190 and a set of interfaces and data links 1105. In some embodiments, the system 1000 receives electroencephalogram (EEG) and electromyogram (EMG) measurements as the physiological data of the user from a set of sensors 1111 and other interface modules (such as a pointing device / medium 1112).

[0141] The above-described embodiments of the present invention may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or set of processors, whether provided in a single computer or distributed in multiple computers. Such processors may be implemented as integrated circuits of one or more processors in an integrated circuit assembly. However, the processor may be implemented using circuits of any suitable format.

[0142] Moreover, embodiments of the present invention may be embodied as a method, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown, which may include performing some actions simultaneously, even though shown as sequential actions in illustrative embodiments.

[0143] The use of ordinal terms such as "first", "second" and the like to modify claim elements in the claims does not itself imply any priority, precedence or order of one claim element relative to another claim element or the temporal order of the actions of performing the method, but is merely used as a label to distinguish one claim element with a specific name from another element with the same name (but for the use of ordinal terms) to distinguish the claim elements.

[0144] Although the present invention has been described by examples of preferred embodiments, it should be understood that various other changes and modifications may be made within the spirit and scope of the present invention. Therefore, it is the purpose of the appended claims to cover all such changes and modifications that fall within the true spirit and scope of the present invention.

Claims

1. A system for signal analysis, the system comprising: an interface configured to receive and send signals, wherein the signals include a set of data sets, the set of data sets including a set of multidimensional signals associated with a set of task tags; a random deep neural network (DNN) block configured to identify the set of task labels from the set of multidimensional signals via a set of random nodes, wherein the set of random nodes is individually specified by a set of irregular beliefs for a posterior distribution, a prior distribution, and a likelihood distribution quantified by a set of difference metrics; A storage body configured to store the data set, the irregular belief set, the hyperparameter set, and the trainable parameter set; and A processor connected to the interface and the memory bank, the processor being configured to perform probabilistic inference to analyze the set of data sets by using the stochastic DNN blocks, wherein the probabilistic inference uses importance-weighted accumulation after variational sampling at the set of random nodes according to the irregular belief set.

2. The system according to claim 1, wherein: The stochastic DNN block includes a variational autoencoder, a variational information bottleneck, a denoising diffusion probability model, and a variational Bayesian neural network variant, wherein the stochastic DNN block is configured with the following combination: Transformation layers, such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, hidden layers, attention layers, pooling layers, padding layers, inception layers, and transformer layers; interconnects such as loop connections, skip additions, skip multiplications, and skip cascades; Non-linear activations such as rectified linear variants, sigmoid variants, shrinkage variants, softmax variants, gated linear, soft additive, and soft sign; and Regularization layers such as dropout, swap out, ignore, occlude, drop connections, jitter, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

3. The system according to claim 1, wherein: The variational sampling is based on using a random number generator according to a reparameterization technique according to a set of variational parameters including location, scale, shape and temperature specified by the irregular belief set.

4. The system according to claim 1, wherein: The set of difference metrics includes Rayleigh's alpha divergence, beta divergence, gamma divergence, Fisher divergence, Jeffrey divergence and combinations of their variants.

5. The system according to claim 1, wherein: The hyperparameter set includes: A set of training hyperparameters that determine the scheduling strategy, such as learning rate, weight decay, gradient momentum, divergence order, batch size, number of epochs, early stopping, dropout rate, and regularization weights; determining a set of architectural hyperparameters of the random DNN block, such as connectivity, type of layer operations, type of activation function, size of the set of trainable parameters, selection of random nodes, number of layer depths, and set of layer widths; and A set of hyperparameters is deployed, such as an initialization strategy for the set of trainable parameters and a quantization strategy for the set of trainable parameters.

6. The system according to claim 1, wherein: The irregular belief set includes mismatched or heterogeneous combinations of: Discrete univariate distributions such as Bernoulli, binomial, geometric, Pascal, hypergeometric, Waringol, Poisson, Schiram, discrete uniform, Zipf, log-sequential, Benford, Polya, Tanner, and Poisson-Consur; Normal-related distributions, such as normal, lognormal, Johnson, half-normal, skew-normal, Voigt, Saliss, hyperbolic, variance-gamma, Student-t, chi, Rayleigh, Maxwell, chi-square, F-ratio, Fisher, Hotelling, and Tracy; Exponential-related distributions, such as exponential, Laplace, Lindley, logistic, hyperbolic secant, exponential power, shifted Gompertz, Erlang, subexponential, Coxian, gamma, Nakagami, Moyal, and Meisner; Extreme value distributions, such as the maximally stable distribution, the least stable distribution, the Gumbel distribution, the Fréchet distribution, and the Weibull distribution; Bounded distributions such as uniform, arcsine, Bates, Coomaraswamy, power, triangular, uniform and, Mises, Wigner, and Pasteur; Heavy-tailed distributions, such as the Pareto distribution, beta prime distribution, Dagum distribution, Davis distribution, Singh Maddala distribution, Cauchy distribution, Landau distribution, Levy distribution, Bernini distribution, Gibbright distribution, and Suzuki distribution; Quantile-based distributions, such as Tukey distribution and Wakeby distribution; Systematic distributions, such as the Pearson distribution; Multivariate continuous distributions, such as the multivariate normal distribution, the multivariate-t distribution, the Dirichlet distribution, and the Copula distribution; and Multivariate discrete distributions, such as the multivariate hypergeometric distribution, the negative multinomial distribution, and the multivariate Poisson distribution.

7. The system according to claim 1, wherein: The dataset set includes a combination of the following: media data, such as images, pictures, movies, text, letters, voice, music, audio and speech; Physical data, such as radio waves, light signals, electrical impulses, temperature, pressure, acceleration, velocity, vibration, and force; as well as Physiological data such as heart rate, blood pressure, mass, water content, electroencephalogram, electromyogram, electrocardiogram, mechanomyograph, electrooculogram, galvanic skin response and magnetoencephalogram, electrocorticogram.

8. The system according to claim 1, wherein: The processor also performs the following steps: configuring the random DNN block by modifying connectivity between the set of multidimensional signals, the set of task labels, and the set of random nodes according to the set of hyperparameters; computing a loss function by forward propagating the set of data sets across the random DNN blocks according to the probabilistic inference; modifying the loss function by regularizing the set of random nodes according to the set of difference metrics; Back-propagating the gradient of the loss function with respect to the set of trainable parameters across the random DNN blocks; as well as The set of trainable parameters of the stochastic DNN block is updated using a gradient method.

9. The system according to claim 8, wherein: The loss function is a combination of variational Rayleigh bound, evidence lower bound, mean square error, cross entropy, structural similarity, cosine similarity, negative log likelihood, absolute error, cross covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, triplet loss and their variants.

10. The system according to claim 8, wherein: The gradient method uses a combination of stochastic gradient descent, adaptive momentum, Ada gradient, Ada bound, Nesterov accelerated gradient, RMS propagation and their variants.

11. The system according to claim 8, wherein: The processor also performs the step of exploring different values ​​of the irregular belief set and the hyperparameter set in the memory bank by using a super gradient method, wherein the super gradient method adopts a combination of reinforcement learning, implicit gradient, evolutionary strategies, differential evolution, particle swarm, genetic algorithm, simulated annealing, Bayesian optimization and their variations.

12. The system of claim 1, wherein: The random DNN block uses a heterogeneous assignment of different beliefs for at least two disjoint sets of random nodes within a layer.

13. The system of claim 1, wherein: The stochastic DNN block uses unmatched pairs of posterior beliefs and prior beliefs for at least one set of random nodes.

14. The system of claim 1, wherein: When analyzing a new set of available data sets, the irregular belief set is modified on the fly to use different probability distributions for the variational sampling at the set of random nodes in the random DNN block.

15. A computer-implemented method for data analysis, the computer-implemented method comprising the steps of: Feeding the data signal set into a random deep neural network DNN, wherein the random DNN includes a layer set and a random node set; propagating the set of data signals according to the set of layers in the stochastic DNN, wherein the stochastic DNN employs variational sampling at the set of random nodes according to an irregular belief set; and The outputs of the random DNN are accumulated as the probability inference results.

16. The method according to claim 15, wherein: The random DNN includes a variational autoencoder, a variational information bottleneck, a denoising diffusion probability model, and a variational Bayesian neural network variant, wherein the random DNN configuration has the following combination: Transformation layers, such as fully connected layers, bilinear layers, convolutional layers, graph convolutional layers, recurrent layers, hidden layers, attention layers, pooling layers, padding layers, inception layers, and transformer layers; interconnects such as loop connections, skip additions, skip multiplications, and skip cascades; Non-linear activations such as rectified linear variants, sigmoid variants, shrinkage variants, softmax variants, gated linear, soft additive, and soft sign; and Regularization layers such as dropout, swap out, ignore, occlude, drop connections, jitter, shuffle, batch normalization, layer normalization, local response normalization, group normalization, and instance normalization.

17. The method according to claim 15, wherein: The variational sampling is based on using a random number generator according to a reparameterization technique of a set of variational parameters including position, scale, shape, and temperature specified by the irregular belief set, wherein the irregular belief set includes mismatched or heterogeneous combinations of: Discrete univariate distributions such as Bernoulli, binomial, geometric, Pascal, hypergeometric, Warinyuer, Poisson, Schirram, discrete uniform, Zipf, log-sequential, Benford, Polya, Tanner, and Poisson-Consonant; Normal-related distributions, such as normal, lognormal, Johnson, half-normal, skew-normal, Voigt, Saliss, hyperbolic, variance-gamma, Student-t, chi, Rayleigh, Maxwell, chi-square, F-ratio, Fisher, Hotelling, and Tracy; Exponential-related distributions, such as exponential, Laplace, Lindley, logistic, hyperbolic secant, exponential power, shifted Gompertz, Erlang, subexponential, Coxian, gamma, Nakagami, Moyal, and Meisner; Extreme value distributions, such as the maximally stable distribution, the least stable distribution, the Gumbel distribution, the Fréchet distribution, and the Weibull distribution; Bounded distributions such as uniform, arcsine, Bates, Coomaraswamy, power, triangular, uniform and, Mises, Wigner, and Pasteur; Heavy-tailed distributions, such as the Pareto distribution, beta prime distribution, Dagum distribution, Davis distribution, Singh Maddala distribution, Cauchy distribution, Landau distribution, Levy distribution, Bernini distribution, Gibbright distribution, and Suzuki distribution; Quantile-based distributions, such as Tukey distribution and Wakeby distribution; Systematic distributions, such as the Pearson distribution; Multivariate continuous distributions, such as the multivariate normal distribution, the multivariate-t distribution, the Dirichlet distribution, and the Copula distribution; and Multivariate discrete distributions, such as the multivariate hypergeometric distribution, the negative multinomial distribution, and the multivariate Poisson distribution.

18. The method according to claim 15, further comprising the steps of: calculating a loss function based on a variational bound to regularize the set of random nodes according to a set of difference metrics; Back-propagating the gradient of the loss function with respect to the set of trainable parameters; as well as The set of trainable parameters is updated according to a gradient method, wherein the gradient method adopts a combination of stochastic gradient descent, adaptive momentum, Ada gradient, Ada bound, Nesterov accelerated gradient, root mean square propagation and their variants.

19. The method according to claim 18, wherein: The set of difference metrics includes Rayleigh's alpha divergence, beta divergence, gamma divergence, Fisher divergence, Jeffrey divergence and a combination of their variants, wherein the loss function is a combination of variational Rayleigh bound, evidence lower bound, mean square error, cross entropy, structural similarity, cosine similarity, negative log-likelihood, absolute error, mutual covariance, clustering loss, divergence, hinge loss, Huber loss, negative sampling, Wasserstein distance, triplet loss and a combination of their variants.

20. The method of claim 18, further comprising the step of exploring different values ​​of the irregular belief set according to a super gradient method, wherein: The super gradient method uses a combination of reinforcement learning, implicit gradient, evolutionary strategies, differential evolution, particle swarm, genetic algorithm, simulated annealing, Bayesian optimization and their variants.

Images