Methods and systems for training latent variable models

Abstract

A method and system for training a latent variable model are provided. The method includes generating, by a short-term Markov chain Monte Carlo (MCMC) inference, a latent vector for each of a plurality of observed examples generated from the latent variable model to obtain an inferred latent vector population; moving, by an optimization transport (OT) correction, the inferred latent vector population to a prior distribution; and updating, based on the corrected latent vectors and corresponding observed examples, model parameters of the latent variable model by gradient descent.

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for computer learning, which can provide improved computer performance, features, and uses. More specifically, this disclosure relates to systems and methods for improving the performance of learning deep latent variable models. Background Technology

[0002] Deep generative models have achieved great success in many fields, such as image generation, image restoration, image representation, image decoupling, and anomaly detection. These models typically consist of simple and expressive generator networks that are latent variable models, assuming that each observed example is generated by a low-dimensional vector of latent variables, and that the latent vectors follow a non-informative prior distribution, such as a Gaussian distribution. Since high-dimensional visual data (e.g., images) often reside on low-dimensional manifolds embedded in high-dimensional spaces, learning latent variable models of visual data is crucial for unsupervised representation learning in computer vision. The main challenge lies in inferring the latent variables for each observation, which typically relies on Markov chain Monte Carlo (MCMC) methods to draw appropriate samples from a difficult-to-analyze posterior distribution (i.e., the conditional distribution of latent variables given the observed samples). Because the posterior distribution of latent variables is parameterized by highly nonlinear deep neural networks, MCMC-based inference can encounter non-convergence and inefficiency, thus affecting the accuracy of model parameter estimation.

[0003] Therefore, there is a need for systems and methods to learn deep latent variable models with improved efficiency. Summary of the Invention

[0004] Firstly, a computer implementation method for training a latent variable model is provided, including:

[0005] The latent vectors are obtained by inferring the latent vectors for each of the multiple observation examples generated from the latent variable model through short-term Markov chain Monte Carlo (MCMC).

[0006] By optimizing transport (OT) correction, the inferred latent vector group is moved to the prior distribution; and based on the corrected latent vectors and corresponding observation examples, the model parameters of the latent variable model are updated by gradient descent.

[0007] Secondly, a system for training a latent variable model is provided, comprising:

[0008] One or more processors; and

[0009] A non-transitory computer-readable medium comprising one or more sets of instructions, which, when executed by at least one of one or more processors, cause the execution of the steps of the method as described in the first aspect.

[0010] Thirdly, a non-transitory computer-readable medium comprising one or more sequences of instructions, which, when executed by at least one processor, cause the steps of a method for training a latent variable model as described in the first aspect.

[0011] Fourthly, a computer program product is provided, comprising a computer program that, when executed by a processor, causes the processor to perform the method described in the first aspect.

[0012] Implementations of short-term MCMC, such as short-term Langevin dynamics, are used in this paper as approximate flow-based inference engines. Bias present in the output distribution of non-convergent short-term Langevin dynamics can be corrected using optimal transfer (OT), which aims to transform the bias distribution generated by finite-step MCMC into a prior distribution with minimal transfer cost. Experimental results validate the effectiveness of OT correction for short-term MCMC and demonstrate that latent variable models trained using a public policy outperform variational autoencoders in image reconstruction, image generation, and anomaly detection. Attached Figure Description

[0013] Reference will be made to embodiments of this disclosure, examples of which may be illustrated in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although this disclosure has been generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of this disclosure to these particular embodiments. Items in the drawings may not be to scale.

[0014] Figure 1 A block diagram depicts a deep latent variable model according to embodiments of the present disclosure.

[0015] Figure 2 Different methods for training deep latent variable models are described.

[0016] Figure 3 Long-term and short-term MCMC inference frameworks according to embodiments of this disclosure are described.

[0017] Figure 4 The process of learning a deep latent variable model using short-term MCMC inference and optimal transfer correction according to embodiments of the present disclosure is described.

[0018] Figure 5 The variation of the output distribution corrected by optimal transmission according to an embodiment of the present disclosure is depicted.

[0019] Figure 6The process of short-term MCMC inference according to embodiments of the present disclosure is described.

[0020] Figure 7 The process of optimal transmission correction according to embodiments of the present disclosure is described.

[0021] Figure 8 The description depicts the marginal distribution q at different iterations and prior distributions according to embodiments of the present disclosure. θ (z K Visualization of the underlying code sampled.

[0022] Figure 9 The output edge distribution of z of different models trained on images of classes "0" and "1" from dataset A is depicted according to embodiments of the present disclosure.

[0023] Figure 10A The effects of the hyperparameter α, which controls the percentage of optimal transmission according to embodiments of the present disclosure, on the optimal transmission cost at different iterations are described.

[0024] Figure 10B The effects of the hyperparameter α, which controls the percentage of optimal transmission according to embodiments of the present disclosure, on the mean squared error (MSE) loss at different iterations are depicted.

[0025] Figure 10C The effects of the hyperparameter α, which controls the percentage of optimal transmission according to embodiments of the present disclosure, on the Frechet Inception Distance (FID) at different iterations are depicted.

[0026] Figure 11 A simplified block diagram of a computing device / information processing system according to embodiments of the present disclosure is depicted. Detailed Implementation

[0027] In the following description, specific details are set forth for purposes of explanation in order to provide an understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of this disclosure described below can be implemented in various ways, such as as processes, apparatuses, systems, devices, or methods on tangible computer-readable media.

[0028] The components or modules shown in the figures are illustrative of exemplary embodiments of this disclosure and are intended to avoid obscuring this disclosure. It should also be understood that throughout the discussion, components can be described as individual functional units, which may include subunits; however, those skilled in the art will recognize that various components or portions thereof may be divided into individual components or may be integrated together, including, for example, in a single system or component. It should be noted that the functions or operations discussed herein can be implemented as components. Components can be implemented in software, hardware, or a combination thereof.

[0029] Furthermore, the connections between components or systems in the diagram are not intended to be limited to direct connections. Instead, data between these components can be modified, reformatted, or otherwise altered by intermediate components. Additionally, more or fewer connections can be used. It should also be noted that the terms “coupled,” “connection,” “communication coupling,” “interface,” “access,” or any derivative thereof should be understood to include direct connections, indirect connections via one or more intermediate devices, and wireless connections. It should also be noted that any communication, such as signals, responses, replies, acknowledgments, messages, queries, etc., can include one or more exchanges of information.

[0030] References to "one or more embodiments," "preferred embodiments," "embodiments," "some embodiments," etc., in the specification mean that a particular feature, structure, characteristic, or function described in connection with an embodiment is included in at least one embodiment of the invention and may be included in more than one embodiment. Furthermore, the above phrases appearing in multiple places in the specification do not necessarily refer to the same embodiment or multiple embodiments.

[0031] The use of certain terms in different places in this specification is for illustrative purposes and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; the use of these terms may refer to a group of related services, functions, or resources that may be distributed or integrated. The terms “including,” “contains,” “has,” and “comprises,” and any variations thereof, should be understood as open-ended terms, and any list below is illustrative and does not imply limitation to the listed items. A “layer” may include one or more operations. The terms “optimal,” “optimized,” “optimization,” etc., refer to an improvement in a result or process and do not require that the specified result or process has reached an “optimal” or peak state. The use of memory, database, repository, data storage, table, hardware, cache, etc., in this document may refer to one or more system components that can input or otherwise record information.

[0032] In one or more embodiments, the stopping condition may include: (1) a set number of iterations have been performed; (2) a certain processing time has been reached; (3) convergence (e.g., the difference between successive iterations is less than a first threshold); (4) divergence (e.g., performance degradation); (5) an acceptable result has been achieved; and (6) all data has been processed.

[0033] Those skilled in the art should recognize that: (1) certain steps may be selectively performed; (2) the steps may not be limited to the specific order set herein; (3) certain steps may be performed in different orders; and (4) certain steps may be performed simultaneously.

[0034] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Every reference / document mentioned in this patent document is incorporated herein by reference in its entirety.

[0035] It should be noted that any experiments and results provided herein are provided in an illustrative manner and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of disclosure of this patent document.

[0036] A. Overview

[0037] Deep generative models have achieved great success in many fields such as image generation, image restoration, image representation, image decoupling, and anomaly detection. Typical examples of such models mainly include simple and expressive generator networks, which are latent variable models that assume each observed example is generated by a low-dimensional vector of latent variables, and that the latent vectors follow a non-informative prior distribution, such as a Gaussian distribution. Figure 1 A block diagram of a deep latent variable model 100 according to an embodiment of the present disclosure is depicted. The model includes multiple convolutional layers, such as a first layer 120, a second layer 130, a third layer 140, and a fourth layer 150, to project and reshape latent vectors 110 onto observed samples I 160, such as images, via a nonlinear transformation g(z). One or more layers in the model can implement convolutions with strides of the desired operations.

[0038] Since high-dimensional visual data (e.g., images) are typically located on low-dimensional manifolds embedded in high-dimensional spaces, learning latent variable models of visual data is crucial for unsupervised representation learning in computer vision. However, learning such models is challenging due to the nonlinear parameterization of g.

[0039] Figure 2Different approaches to training deep latent variable models are described. The main challenge of Markov Chain Monte Carlo (MCMC)-based maximum likelihood estimation (MLE) lies in inferring the latent variables for each observation, which typically relies on MCMC methods drawing appropriate samples from a difficult-to-analyze posterior distribution (i.e., the conditional distribution of latent variables given examples of observations). Since the posterior distribution of latent variables is parameterized by highly nonlinear deep neural networks, MCMC-based inference can encounter non-convergence and inefficiency, thus affecting the accuracy of model parameter estimation.

[0040] Variational autoencoders (VAEs) and generative adversarial networks (GANs) are currently popular methods for training deep latent variable models. Both models train the generator by recruiting an additional model to assist training, which is then ignored during testing. To avoid inefficient MCMC sampling from the posterior, variational inference has become an attractive alternative by approximating the intractable posterior via a tractably tractable network. Despite the increasing popularity and acceptance of VAEs, their drawbacks are becoming increasingly apparent. First, it parameters the internal iterative inference process through an external feedforward inference model. These additional parameters, resulting from reparameterization, must be estimated along with the parameters of the generator network. Second, this joint training is accomplished by maximizing the variational lower bound. Therefore, the accuracy of a VAE largely depends on the accuracy of the inference model, which approximates the true posterior distribution. Variational inference is only equivalent to the desired maximum likelihood estimation when the Kullback-Leibler (KL) divergence between the inference and the posterior distribution is equal to 0. This objective is often impractical in practice. Third, designing inference models for VAEs requires additional effort, especially for generators with complex dependency structures of latent variables. For example, some have proposed top-down generators with multi-level latent variables, while others have proposed dynamic generators with time series of latent variables. Designing inference models to infer the latent variables of such models is not a simple task. Arbitrary design of the inference model does not guarantee performance. GAN training methods involve discriminators in addition to the generator, thus having two sets of parameters during training. Model collapse may occur during training. Furthermore, it is difficult to design effective inference models for GAN methods.

[0041] This disclosure completely abandons the idea of ​​reparameterizing the inference process. Instead, it discloses embodiments of MCMC-based inference for training deep latent variable models. Specifically, embodiments of short-term MCMC, such as short-term Langevin dynamics, are used to perform inference of latent vectors during training. However, considering that convergence of finite-step Langevin dynamics in each iteration can be problematic, embodiments of optimal transport (OT) are used to correct for biases that may exist in such short-term MCMC. OT can be used to transform an arbitrary probability distribution into a desired distribution with minimum transport cost. Therefore, OT cost can be used to measure the difference between two probability distributions. In one or more embodiments of this disclosure, short-term MCMC is considered as a streaming model whose parameters are derived from the learning of the latent variable model. Bias in short-term MCMC can be corrected by performing an optimal transport from the resulting distribution obtained from short-term MCMC to the prior distribution. This operation aims to minimize the OT cost between the inferred distribution and the prior distribution, where the parameters in the streaming model are updated rather than optimized. The parameters of the latent variable model can be updated more accurately with the corrected inference output.

[0042] Figure 3 Long-term and short-term MCMC inference frameworks according to embodiments of this disclosure are described. Conventional long-term MCMC inference frameworks involve only inference step 310 and learning step 320. As previously mentioned, such long-term MCMC-based inference may encounter non-convergence and inefficiency, thus affecting the accuracy of model parameter estimation. On the other hand, this document discloses embodiments of a short-term MCMC inference framework including inference step 330, correction step 340, and learning step 350, details of each step being provided in [the following text is missing from the original extract]. Figure 4 As shown in the image.

[0043] Figure 4 A process for learning a deep latent variable model using short-term MCMC inference and optimal transfer correction according to embodiments of the present disclosure is described. The process iterates through one or more rounds of the following three steps: (1) In inference step 405: a group of inferred latent vectors is obtained by using latent vectors from each of a plurality of observation examples generated from the latent variable model through short-term MCMC inference of Langevin dynamics. Langevin dynamics samples from a posterior distribution. (2) In correction step 410: the group of all inferred latent vectors is moved to a prior distribution through optimal transfer correction. (3) In learning step 415: the model parameters are updated by gradient descent based on the corrected latent vectors and the corresponding observation examples.

[0044] Using the disclosed short-term MCMC inference with OT correction has several advantages: (1) Efficiency: Model learning and inference are efficient using short-term MCMC; (2) Convenience: Approximate inference models represented by short-term MCMC are automatic, eliminating the need to worry about designing and training separate inference models. Bottom-up inference and top-down generation are controlled by the same set of parameters; (3) Accuracy: Optimal transfer corrects for the errors of non-convergent short-term MCMC inference, thereby improving the accuracy of model parameter estimation.

[0045] The contributions disclosed in this patent include at least the following: (1) an embodiment of training a deep latent variable model by non-convergent short-term MCMC inference with OT correction; (2) an embodiment of the semi-discrete OT method to approximate a one-to-one mapping between the inferred latent vectors and samples drawn from the prior distribution; and (3) strong empirical results in various experiments to verify the effectiveness of the disclosed strategy for training deep latent variable models.

[0046] B. Some related work

[0047] 1. Variational inference

[0048] VAEs are a popular method for learning generator networks that approximate the tractable posterior distribution of latent variables by simultaneously training an easily tractable inference network. In VAEs, designing an inference model for the latent variables is a meaningful task in generator networks with complex architectures. However, the method disclosed in this patent document does not rely on an additional inference model to aid training. It infers from the posterior distribution through Langevin sampling, followed by optimal transfer correction.

[0049] Alternating Backpropagation (ABP) Algorithm. Maximum likelihood learning of generator networks, including its dynamic version, can be achieved using the alternating backpropagation (ABP) algorithm without relying on an inference model. The ABP algorithm trains the generator model by alternating the following two steps: (1) Inference step: inferring latent variables from the posterior distribution through Langevin sampling, and (2) Learning step: updating the model parameters based on the training data and the inferred latent variables through gradient descent. Both steps compute gradients with the aid of backpropagation. The ABP algorithm has been successfully applied to saliency detection, zero-order learning, decoupled representation learning, etc.

[0050] 2. Optimal transmission

[0051] Optimal transfer (OT) is used to compute the distance between two metrics and to propagate a source distribution toward a target distribution. Recently, OT has been widely used in generative models to help generate high-quality samples. For example, by replacing the original KL divergence in a GAN model with the W1 distance, some have proposed the Wasserstein GAN (WGAN) model to achieve better convergence and generate higher-quality samples. Others have proposed Wasserstein VAEs that minimize the Wasserstein distance between the inference model and the posterior distribution. Besides the Wasserstein distance, optimal transfer is also used to transfer a simple uniform distribution to a complex latent feature distribution extracted by an autoencoder to generate images.

[0052] C. Implementation Examples of Maximum Likelihood Learning for Deep Latent Variable Models

[0053] Suppose I is a D-dimensional example of observed data, such as an image. Let z be a d-dimensional vector of continuous latent variables. Generalizing from traditional factor analysis models, generator networks assume that the observed example I is obtained through a nonlinear transformation I = g θ (z)+∈ is generated from the latent vector z, where g θ It is a top-down convolutional neural network (sometimes called a deconvolutional neural network), where the parameters θ include all trainable weights and biases in the network. It is observation error. I d and I D Let be d-dimensional and D-dimensional identity matrices, respectively, assuming d << D. The generator network may essentially be a nonlinear latent variable model, defining a joint distribution of (I, z).

[0054] p θ (I, z) = p θ (I|z)p(z) (1)

[0055] Where a prior distribution is assumed and The standard deviation σ is assumed to be a specific value. Following Bayes' rule, the marginal distribution p can be obtained. θ (I)=∫p θ (I, z)dz and the posterior distribution p θ (z|I)=p θ (I, z) / p θ (I).

[0056] Given a set of training examples {I i , i = 1, ..., n} ~ p data (I), where p data (I) represents the unknown data distribution. p θTraining can be performed by maximizing the log-likelihood of the training samples:

[0057]

[0058] When the number of training examples n is large enough, this is equivalent to KL(p data ||p θ Minimize ).

[0059] In one or more embodiments, maximizing the log-likelihood function presented in equation (2) can be achieved using an iterative gradient ascent algorithm.

[0060]

[0061] Where γ t The gradient of the logarithmic probability is given by the following formula, which depends on the learning rate at time t:

[0062]

[0063] In order to calculate equation (4) Need to estimate According to equation (1), the logarithm of the joint distribution is given by the following equation:

[0064]

[0065] The constant term is independent of z or θ, therefore in (z) can be efficiently calculated through backpropagation.

[0066] D. Examples of short-term MCMC inference

[0067] 1. Long-term Langevin dynamics example

[0068] To learn the model parameters θ using equation (3), the key is to compute the expectation term, which is difficult to handle in equation (4). This can be achieved by first working with p... θ This is achieved by sampling from (I, z) and then approximating it using Monte Carlo sample averaging. Given a step size s > 0 and an initial value z... 0 Langevin dynamics, a gradient-based MCMC method, can recursively compute from the posterior density p θ (z|I) Generate samples

[0069]

[0070] In equation (6), k indexes the time steps of Langevin dynamics. It is random noise diffusion. Furthermore, in It can be efficiently computed through backpropagation.

[0071] In one or more embodiments, K is used to represent the number of Langevin steps. When s→0 and K→∞, regardless of z... 0 What is the initial distribution of z? K It will converge to the posterior distribution p θ (z|I), and become p θ Appropriate samples of (z|I).

[0072] 2. Short-term Langevin dynamics example

[0073] Using long-term MCMC to train deep latent variable models may be unwise or impractical. In each iteration, a finite number of Langevin dynamical steps are run to infer p. θ (z|I) appears feasible. Therefore, the short-run K-step Langevin dynamics are given by the following equation:

[0074] z 0 ~p0(z)

[0075]

[0076] In one or more embodiments, the initial distribution p0 is assumed to be a Gaussian distribution. This dynamic can be viewed as a conditional generator that generates random noise z under condition I. 0 The transformation is converted to the target distribution. The transformation itself can also be viewed as a K-layer residual network, where each layer shares the same parameters θ and has noise injection. κ θ Used to represent a K-step MCMC transfer kernel. Given I, z k The conditional distribution is:

[0077] q θ (z K |I)=∫p0(z 0 )κ θ (z K |z 0 ,I)dz 0 (8)

[0078] z K The corresponding edge distribution is

[0079] q θ (z K )=∫q θ (z K |I)p data (I)dI (9)

[0080] If MCMC converges, q θ (z KThe distribution should be close to the prior distribution p(z); otherwise, there is a gap between them.

[0081] Equation (7) is also known as the short-term MCMC with noise initialization, where for each step of parameter update, the short-term MCMC is derived from the noise distribution z. 0 ~p0(z) begins. If a short-term MCMC is initialized with the inference result obtained from the previous iteration, it is called a persistent short-term MCMC.

[0082] Although the short-term MCMC inference in equation (8) is highly efficient, it may not converge to the true posterior distribution p. θ (z|I). Some people treat short-term MCMC as an approximate inference model and optimize the step size s through variational inference, where the step size s is optimized via grid search or gradient descent, such that the short-term MCMC q s (z|I) (where s is the learning parameter) may best approximate the posterior distribution p. θ (z|I).

[0083] E. Implementation examples of MCMC inference with OT correction

[0084] In one or more embodiments, optimal transfer is used to correct for bias in short-term inference results. In one or more embodiments, instead of minimizing the difference between the short-term inference model and the true posterior, i.e., KL(q) θ (z K |I)|p θ (z|I)), instead, OT is used to minimize the marginal distribution q of the latent variables inferred from short-term Langevin dynamics. θ (z K The transmission cost between the prior distribution p0(z) and the prior distribution p0(z).

[0085] 1. Example of OT correction for biased short-term MCMC

[0086] In one or more embodiments, in order to learn a top-down latent variable model I = g from the latent vector z to generate the observed image I, θ (z), iterate through the following three steps.

[0087] (1) Inference steps: First, perform K-step short-term MCMC on each observed image I i Inferring the latent vector, i.e. Then for all observed data {I i} The group that obtains the inferred latent vectors in

[0088] (2) Calibration steps: OT is used to... Move to the desired prior distribution to narrow the gap between them caused by non-convergent inference. Figure 5 The variation in the output distribution corrected by optimal transmission according to embodiments of the present disclosure is depicted. For example... Figure 5 As shown, OT reshapes the biased group 510 to the prior distribution 520 with minimal movement cost. Using a more accurate inferred latent vector, subsequent parameter updates can be more accurate.

[0089] (3) Learning steps: Given the observed image and its corresponding inferred latent vector, θ is updated by equations (3) and (4). As θ is trained better, the inference engine q... θ (z K The accuracy becomes more precise, and the corrections made by OT become smaller. An illustration of the publicly available strategy using OT correction is provided above. Figure 3 Presented in the middle. For example... Figure 3 As shown, the disclosed framework using short-term MCMC with OT correction 304 is compared with a framework using conventional long-term MCMC inference 302.

[0090] In practice, either noisily initialized short-term MCMC or persistent short-term MCMC can be used in the inference step. In one or more experiments, the latter is chosen for faster convergence. For the calibration phase, from To {z i Learn one-to-one OT mapping, {z i} is a group sampled from a prior Gaussian distribution, and The sizes are the same. Calculating the optimal transfer in each iteration is time-consuming and unnecessary in practice. In one or more embodiments, to make the entire pipeline more efficient, a correction step can be performed after every L iterations. This yields a bijective OT mapping. After that, without directly using paired data Instead of updating the model, a mixed correction using OT results and old results is used. To avoid due to Sudden changes can lead to learning instability, i.e.

[0091]

[0092] In equation (10), α∈[0,1] is a hyperparameter used to control the percentage of OT results used for correction. Therefore, the corrected paired data can be obtained. To update the model parameters θ. It should be noted that when α = 0, the model embodiment of this disclosure can be considered to degenerate into a traditional ABP model. If α is set to 1, the short-term output is fully corrected using the OT results. A moderate 0 < α < 1 generally helps to minimize the marginal distribution q. θ (z KThe algorithm gradually pulls the data towards the prior distribution p(z) to ensure smooth correction. Method 1 summarizes the entire pipeline of the learning strategy implementation using a detailed process for short-term MCMC inference and OT correction, respectively. Figure 6 and Figure 7 As shown in the image.

[0093] Method 1: Example of short-term MCMC inference with OT correction

[0094]

[0095]

[0096] Method 2: Short-term MCMC inference example with OT correction

[0097]

[0098] Figure 6 The process of short-term MCMC inference according to embodiments of the present disclosure is described. In step 605, the inference is derived from the Gaussian prior distribution. An initial distribution p0 is randomly sampled. In step 610, short-term MCMC inference is performed using finite-step Langevin dynamics to infer multiple latent vectors from observation examples generated from the latent variable model g0. In each Langevin step, the short-term MCMC can be initialized with random noise or the inference results obtained in the previous step.

[0099] Figure 7 The process of optimal transmission correction according to embodiments of the present disclosure is described. In step 705, given a plurality of inferred latent vectors and a plurality of samples randomly sampled from the prior (e.g., Gaussian distribution N(0, 1),... d The bijective OT mapping, which involves mapping multiple inferred latent vectors to multiple transport paths of multiple samples, is iteratively optimized using gradient descent. In step 710, the bijective OT mapping is established when a stopping condition is met. To obtain OT results that include potential vectors from multiple mappings. The stopping condition can be that the number of iterations is met or the optimization gradient becomes less than a predetermined threshold. In step 715, multiple OT-corrected latent vectors are obtained by mixing multiple mapped latent vectors and multiple inferred latent vectors, where the percentage of mapped latent vectors is controlled by a hyperparameter between 0 and 1. Subsequently, the multiple OT-corrected latent vectors can be used to update the model parameters of the deep latent variables in the learning step.

[0100] It should be noted that although Method 2 shows the update process according to the Adam method, where β1 = 0.9 and β2 = 0.5, the parameters β1 (the exponential decay rate estimated by the first moment) and β2 (the exponential decay rate estimated by the second moment) can be other values ​​and other methods can be used. Such variations should still be within the scope of this patent document.

[0101] 2. Optimal transmission

[0102] Given from q θ (z K The underlying code for sampling, i.e. and from a priori Randomly generated samples from arrive The one-to-one mapping is calculated through optimal transmission. Specifically, in one or more embodiments, the cost function is set as the squared Euclidean distance. Because it has good geometric meaning, the following assignment problem can then be solved:

[0103]

[0104] in

[0105] According to linear programming theory, each row / column of π contains only one non-zero element. In fact, all non-zero elements should equal 1 / n. Therefore, from... To {z j The mapping of} can be defined as: if π ij ≠0, then When n is very large, directly solving the above problem using linear programming will be problematic because the computational complexity is extremely high, O(n^2). 2.5 Similarly, the classic Hungarian algorithm for the allocation problem suffers from high computational complexity O(n^2). 3 The approximation algorithm cannot be used to solve this problem. Approximate OT solvers, such as the Sinkhorn algorithm, also fail to solve the above problem because these solvers tend to give dense transport schedules, making it impossible to recover the OT mapping. Furthermore, approximation algorithms are not suitable for large-scale problems with n > 20,000. Therefore, the dual problem of equation (11) is used. In one or more embodiments, the original dual formula for semi-discrete OT can be extended to the following minimization problem in a discrete setting:

[0106]

[0107] The above problem is convex because it is the maximum value of the sum of n hyperplanes. Therefore, it can be solved by gradient descent optimization. The gradient is given by... Calculation, where #J j It is J j The number of elements in h. Assume h * If h is the optimal solution for E(h), then h = h * +(c, c, ..., c) T This is also the optimal solution. To omit the need for movement, Defined as Using gradient information, the energy E(h) can be minimized using the Adam gradient descent algorithm.

[0108] Since equation (12) is the dual of the allocation problem, in the optimal solution h * It is easy to pass. from To {z j Reconstruct the one-to-one OT mapping. During the optimization process, when the gradient... The process stops when the norm is less than a predetermined threshold ε. Ideally, if ε = 0, the mapping T becomes injective and surjective, and each J... j It includes only one element, i.e., the corresponding i. In this case, the OT mapping T is well-defined. In practice, ε is usually set to ε > 0, so T is neither injective nor surjective. In this case, for some z... j s, may have one or more corresponding And for some other z j s, corresponding It may not exist. To disambiguate and reconstruct the one-to-one mapping, it is necessary to handle sets J that will be empty or contain one or more elements. j Therefore, the approximate OT mapping Given the following: (1) If J j If there is only one element, i, then (2) When J j When there is more than one element, randomly select i∈J j And discard other elements, then define (3) Will be abandoned and empty J j The corresponding z j s respectively from Remove from the domain and range. In this way, new injective and surjective maps approximating the OT map T can be constructed.

[0109] It should be noted that in the embodiments of the disclosed OT method, the prior distribution is not limited to a Gaussian distribution. In fact, any prior distribution can be chosen as long as it is easy to sample. Furthermore, the computational complexity of solving the non-smooth dual problem in equation (12) is... In the context of training complex neural networks with a large number of parameters, the time spent optimizing the OT problem is negligible. Finally, from The number of samples removed should not exceed nε. In one or more experiments, ε is typically set to ε = 0.05. Using such a small ε, a good approximation of the OT mapping can be obtained.

[0110] E. Experimental Results

[0111] It should be noted that these experiments and results are provided for illustrative purposes and were conducted under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of disclosure of this patent document.

[0112] In the experiment, the embodiments of the disclosed model were examined to determine whether they could (1) successfully correct the marginal distribution q of the latent vectors inferred from short-term Langevin dynamics. θ (z K (2) Learn an expressive generator to synthesize visually realistic images from a prior distribution, and (3) successfully perform anomaly detection for testing. To demonstrate the performance of the published method, experiments were conducted on various datasets. For details on the generator architecture design, model hyperparameter selection, and optimization methods for each dataset, please refer to the supplementary material. Furthermore, to investigate the impact of different hyperparameters, dataset A was primarily used due to its simplicity and representativeness. To quantify the model's performance, mean squared error (MSE) and Frechet initiation distance (FID) scores were used to measure the quality of the reconstructed and generated images. The FID score is a metric used to evaluate the quality of the generated images.

[0113] Datasets: Various datasets were used for training and / or testing in one or more experiments. For faster convergence, some sample data, such as image data, were randomly selected. All training images were resized and scaled to the range [-1, 1].

[0114] Model architecture: The model architecture is shown in Table 1, where the potential dimensions of datasets A, B and C are set to 30, 64 and 64, respectively.

[0115] Table 1. Generator architecture for different datasets

[0116]

[0117] Optimization: The generator parameters were initialized using Xavier normalization and then optimized using the Adam optimizer, with β1 = 0.5 and β2 = 0.99. The batch size was set to 2,000 for all experiments. In Method 1, both L and K were set to 50. For dataset A, the hyperparameter α was set to 0.5, and for datasets B and C, α was set to 0.3. The step size s for datasets A, B, and C was set to 0.3, 3.0, and 3.0, respectively. For all models, ∑ was also set to σ = 0.3.

[0118] Computational Cost: Due to the involvement of short-term MCMC and optimal transfer, the overall pipeline runtime needs to be considered. Here, we take a dataset B consisting of multiple 32×32×3 images as an example. The disclosed model implementation was trained on two NVIDIA Titan X GPUs. For each iteration, the inference step with K=30 takes approximately 124 minutes, the optimal transfer correction step takes approximately 10 minutes, and the L2=2 learning step takes approximately 5 minutes. Generally, the model needs to run 10–15 iterations, consuming approximately one day.

[0119] 1. Potential Space Analysis

[0120] To verify that the proposed method does indeed correct the short-term marginal distribution q of the latent variables θ (z K In dataset A, categories “0” and “1” are picked. From these categories, for better visualization, an embodiment of the disclosed model is learned by setting the latent space dimension to 2. Figure 8 Implementations of the public model are shown in different iterations of q. θ (z K The evolution of ). In the figure, iteration represents the number of OT corrections. From Figure 8 It can be clearly seen that q θ (z K As OT correction gradually shifts toward the prior distribution and eventually matches it. Figure 9 A comparison of latent vectors inferred by embodiments of the VAE model, the ABP model, and the disclosed model (shown as "current") is presented. The distributions of the latent vectors inferred by the VAE and ABP models deviate significantly from the prior (Gaussian) distribution, while the marginal distribution q of the disclosed model... θ (z K It looks closer to a Gaussian distribution.

[0121] 2. Image Modeling

[0122] The quality of the reconstructed and generated images was evaluated in one or more experiments. Using a well-learned model, q θ (z KThe marginal distribution should match the prior distribution well. In this case, the generator will be a probability transformation from the prior Gaussian distribution to the image distribution, and can be obtained through I = g θ (z) is used to synthesize a high-quality image with a latent vector z sampled from a prior distribution. Furthermore, this model may be useful for reconstruction. Below, embodiments of the disclosed model are compared with VAE, its variant two-stage VAE (2sVAE), and regularized autoencoder (RAE). It is also compared with the ABP model and its variant Short-Term Inference (SRI), whose generator has multi-layered latent variables. The last model used for comparison is the latent space energy-based model (LEBM), which uses energy-based short-term MCMC to infer latent variables for each observed image.

[0123] Given reconstructed and generated images with latent vectors sampled from a given prior distribution, it is clear that the generated images are realistic and comparable to real images in the training dataset. In Table 2, MSE is used to test the quality of the reconstructed images, and FID scores are used to quantify the quality of the generated images. The table shows that embodiments of the publicly disclosed method (shown in the "Current" column) outperform other methods in both reconstruction and generation tasks.

[0124] Table 2. Comparison results of different datasets. MSE and FID (lower is better) are used to test the quality of reconstructed and generated images, respectively.

[0125]

[0126] Table 3. AUPRC scores (higher is better) for unsupervised anomaly detection on dataset A. Results for the public model implementation are averaged over 10 experiments for variance.

[0127]

[0128] 3. Anomaly Detection

[0129] Anomaly detection is another task that can help evaluate implementations of publicly available models. Using a model that has learned well from normal data, anomalous data can be detected by: firstly, through short-term Langevin dynamics from the conditional distribution q… θ (z K In |I), sample the latent code z of the given test image I, and then calculate the joint probability log p in equation (5). θ The logarithm of (I,z). Based on this theory, the joint probability of normal images should be high, and the joint probability of abnormal images should be low.

[0130] In the following experiments, one class in dataset A was treated as the anomalous class, while the other classes were considered normal. The model was trained using only normal data and then tested using both normal and anomalous data. To evaluate performance, log p was used. θ (I,z) is used as the decision function to calculate the area under the precision-recall curve (AUPRC). During the testing phase, each experiment was run 10 times to obtain the mean and variance. Table 3 compares the embodiments of the disclosed method (shown in the "Current" column) with relevant models in this task, including VAE, MEG, BiGAN-σ, LEBM, and ABP models, which can be considered special cases requiring no OT calibration. As can be seen from the table, the tested method embodiments achieve better results than other methods.

[0131] 4. The impact of the number of potential dimensions

[0132] This section illustrates the impact of the dimension of the latent space under the same architecture. Dataset B is used for different numbers of dimensions in the latent space, for example, 20, 40, and 64. As shown in Table 4, better results can be obtained in both reconstruction and generation as the latent dimension increases.

[0133] Table 4. Performance of the proposed method on dataset B, with the same architecture but different numbers of potential dimensions. (For MSE and FID, smaller is better.)

[0134]

[0135] 5. Ablation Research

[0136] This section explores the performance of the proposed model under different values ​​of the parameter α introduced in Equation (10), different step sizes of Langevin dynamics (s in Equation (7), different numbers of Langevin steps (K in Equation (7), and different number of iterations of the learning step, which is designed to use paired data. Maximize the joint probability in equation (5).

[0137] The effect of α. First, the effect of α in equation (10) is studied, and the results are as follows. Figure 10A , 10B As shown in 10C. Figure 10A It shows from To {z j The OT cost of}, it is used as q θ (z K The distance between the short-term Langevin dynamics and the prior distribution p(z) is used. It is clear that a larger α can more quickly influence the marginal distribution q. θ (z K(Pulling towards the prior distribution) Figure 10B Recommendation: To achieve a smaller MSE loss, it is best to choose a smaller α. For example... Figure 10C As shown, the optimal FID is obtained using a moderate value of α, i.e., α = 0.5. Therefore, to balance OT cost, MSE loss, and FID, α = 0.5 is set in the following experiments. The curves also reveal that as the algorithm progresses, the marginal distribution q... θ (z K As the reconstructed image gets closer to the prior distribution p0(z), the quality of both the reconstructed and generated images also improves.

[0138] The effect of Langevin step size. Table 5 shows the performance of the disclosed model examples under different Langevin step sizes (s in Equation (7)). In the table, "before" means the model was used before OT correction, and "after" means the trained model was used after OT correction. When s is small, the MSE loss is small, but the FID is relatively large, which means the quality of the generated images is not very good. When s is large, for example, s = 6e in the last column. -2 The MSE loss and FID are both large, meaning that even high-quality reconstructed images cannot be obtained. In this case, the model does not actually converge well. Only with an appropriate Langevin step size (in this experiment, s = 3e) can the model achieve good convergence. -2 Only by achieving a good balance between MSE and FID can the desired reconstruction and generation results be obtained.

[0139] Table 5. Effect of Langevin dynamics step size.

[0140]

[0141] The impact of the Langevin step count. The Langevin step count K in Equation (7) is another key factor affecting the performance of the proposed method. Theoretically, a larger K will lead to more convergent MCMC inferences, thus helping to obtain more accurate latent variables. To demonstrate this, K was set to K = 30, 50, and 100, respectively, while other parameters remained unchanged. The results are shown in Table 6. Indeed, a larger K leads to better results. However, a larger K also linearly increases the overall pipeline runtime. Therefore, to achieve a good balance between runtime and performance, an appropriate K needs to be selected for different datasets.

[0142] Table 6. The effect of Langevin step count K.

[0143]

[0144] The impact of the number of iterations within the learning step. In Method 1, the several iterations of gradient ascent, denoted by L2, are actually run within the learning step to pass through paired data. Maximize the joint probability in equation (5). The results are shown in Table 7. As can be seen from the table, image reconstruction and generation can achieve better performance by increasing L2.

[0145] Table 7. The impact of the number of learning iterations.

[0146]

[0147] G. Some conclusions

[0148] This document discloses embodiments of using OT (Operational Theory) to correct biases in short-term inference based on MCMC (Multi-Channel Multi ...

[0149] H. Computing System Implementation

[0150] In one or more embodiments, aspects of this patent document may be directed to, may include, or may be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any tool or set of tools operable for calculating, calculating, determining, classifying, processing, sending, receiving, retrieving, initiating, routing, switching, storing, displaying, communicating, detecting, recording, copying, processing, or utilizing information, intelligence, or data of any form. For example, a computing system may be or may include a personal computer (e.g., a laptop computer), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a phablet, a tablet computer, etc.), a smartwatch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more disk drives (e.g., hard disk drives, solid-state drives, or both), one or more network ports for communicating with external devices and various input and output (I / O) devices, such as keyboards, mice, touchscreens, styluses, microphones, cameras, touchpads, displays, etc. The computing system may also include one or more buses operable to transmit communication between various hardware components.

[0151] Figure 11 A simplified block diagram of an information processing system (or computing system) according to embodiments of the present disclosure is depicted. It should be understood that the functionality shown in system 1100 can operate to support various embodiments of the computing system—although it should be understood that the computing system can be configured differently and include different components, including… Figure 11 The number of components described is less or more.

[0152] like Figure 11 As shown, the computing system 1100 includes one or more CPUs 1101 that provide computing resources and control the computer. The CPU 1101 may be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 1102 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, the one or more GPUs 1102 may be incorporated into a display controller 1109, such as part of one or more graphics cards. The system 1100 may also include system memory 1119, which may include RAM, ROM, or both.

[0153] Multiple controllers and peripherals can also be provided, such as Figure 11As shown. Input controller 1103 represents an interface to various input devices 1104. The computing system 1100 may also include a storage controller 1107 for interfacing with one or more storage devices 1108, each storage device including a storage medium such as magnetic tape or a disk, or an optical medium that can be used to record instruction programs for operating systems, utilities, and applications, which may include embodiments of programs implementing various aspects of this disclosure. According to this disclosure, storage device 1108 can also be used to store processed data or data to be processed. System 1100 may also include a display controller 1109 for providing an interface to a display device 1111, which may be a cathode ray tube (CRT) display, a thin-film transistor (TFT) display, an organic light-emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 1100 may also include one or more peripheral controllers or interfaces 1105 for one or more peripheral devices 1106. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. The communication controller 1114 can interface with one or more communication devices 1115, enabling the system 1100 to connect to remote devices via any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Ethernet Fibre Channel (FCoE) / Data Center Bridge (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN), or via any suitable electromagnetic carrier signal, including infrared signals. As shown in the depicted embodiment, the computing system 1100 includes one or more fans or fan trays 1118 and one or more cooling subsystem controllers or controllers 1117 that monitor the thermal temperature of the system 1100 (or its components) and operate the fans / fan trays 1118 to help regulate the temperature.

[0154] In the illustrated system, all major system components can be connected to bus 516, which can represent more than one physical bus. However, the various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. Furthermore, programs implementing various aspects of this disclosure can be accessed from a remote location (e.g., a server) via a network. Such data and / or programs can be transmitted through any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as optical discs (CDs) and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0155] Various aspects of this disclosure may be encoded on one or more non-transitory computer-readable media having instructions for one or more processors or processing units to result in execution steps. It should be noted that the one or more non-transitory computer-readable media should include volatile and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functionality of a hardware implementation can be implemented using an ASIC, a programmable array, digital signal processing circuitry, etc. Therefore, the term "means" in any claim is intended to cover both software and hardware implementations. Similarly, the term "computer-readable medium" as used herein includes software and / or hardware, or a combination thereof, on which a program of instructions is contained. In consideration of these alternative implementations, it should be understood that the accompanying drawings and description provide functional information necessary for those skilled in the art to write program code (i.e., software) and / or assemble circuitry (i.e., hardware) to perform the desired processing.

[0156] It should be noted that embodiments of this disclosure may further relate to computer products having a non-transitory tangible computer-readable medium having computer code thereon for performing operations of various computer implementations. The medium and computer code may be those specifically designed and constructed for the purposes of this disclosure, or they may be of types known or available to those skilled in the art. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CDs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as ASICs, PLDs, flash memory devices, other non-volatile memory (NVM) devices (such as 3DXPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code generated by a compiler, and files containing higher-level code executed by a computer using an interpreter. Embodiments of this disclosure may be implemented wholly or partially as machine-executable instructions that may reside in program modules executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in a local, remote, or both setting.

[0157] Those skilled in the art will recognize that no computing system or programming language is essential to the practice of this disclosure. They will also recognize that the aforementioned components can be physically and / or functionally separated into modules and / or submodules or combined together.

[0158] Those skilled in the art will understand that the foregoing examples and embodiments are exemplary and do not limit the scope of this disclosure. All arrangements, enhancements, equivalents, combinations, and modifications that will be apparent to those skilled in the art upon reading the specification and studying the accompanying drawings are intended to be included within the true spirit and scope of this disclosure. It should also be noted that the elements of any claim may be arranged differently, including having multiple dependencies, configurations, and combinations.

Claims

1. A computer-based method for training a latent variable model, comprising: The latent vectors are inferred from each of the multiple image examples generated from the latent variable model by short-term Markov chain Monte Carlo (MCMC) to obtain the inferred latent vector group; By optimizing transport (OT) correction, the inferred latent vector group is moved to the prior distribution; and Based on the corrected latent vectors and corresponding image examples, the model parameters of the latent variable model are updated by gradient descent. The latent variable model is used for any of the following: image reconstruction, image generation, or anomaly detection. The OT calibration includes the following steps: Given a group of inferred latent vectors and multiple samples randomly sampled from the prior, a bijective OT mapping is iteratively optimized using gradient descent optimization. The bijective OT mapping includes multiple transmission paths to map multiple inferred latent vectors to multiple samples. When the stopping condition is met, construct a bijective OT mapping to obtain the OT result, which includes the latent vectors of multiple mappings; and Multiple corrected latent vectors are obtained by mixing multiple mapped latent vectors and multiple inferred latent vectors.

2. The computer-implemented method according to claim 1, wherein, The short-term MCMC is an iterative process that runs a finite number of Langevin dynamics steps in each iteration to make inferences.

3. The computer-implemented method of claim 2, wherein for each iteration, the short-term MCMC begins with a noise distribution randomly sampled from a Gaussian distribution, or with an inference obtained in a previous iteration.

4. The computer-implemented method according to claim 2, wherein, The OT correction is performed after a predetermined number of iterations of the short-term MCMC.

5. The computer-implemented method according to claim 1, wherein, The stopping condition is that the number of iterations or the optimization gradient becomes less than a predetermined threshold.

6. The computer-implemented method according to claim 1, wherein, The percentage of the latent vectors in the mixture is controlled by a hyperparameter with a value between 0 and 1.

7. The computer-implemented method according to claim 1, wherein, The OT correction has a cost function set as the squared Euclidean distance.

8. A system for training a latent variable model, comprising: One or more processors; as well as A non-transitory computer-readable medium comprising one or more sets of instructions, which, when executed by at least one of one or more processors, cause the execution of the steps of the method as claimed in any one of claims 1 to 7.

9. A non-transitory computer-readable medium comprising one or more sequences of instructions, which, when executed by at least one processor, cause the steps of a method for training a latent variable model as claimed in any one of claims 1 to 7.

10. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1 to 7.

Images