energy-based generative model via coarse-to-fine expansion and sampling

Abstract

Embodiments of an energy-based model (EBM) are presented herein that can be trained via embodiments of a multi-stage coarse-to-fine expansion and sampling strategy. Embodiments of the training method start by learning a coarse-level EBM from low-resolution images and then gradually transition to learning finer-level EBMs from higher-resolution images by expanding the energy function as learning progresses. Embodiments have both smooth learning and computational efficiency of sampling. Across all EBMs, the tested embodiments achieve state-of-the-art performance in image generation and successfully synthesize high-fidelity images. Embodiments are also applicable to image restoration and out-of-distribution detection. Framework embodiments can be further generalized for one-sided unsupervised image-to-image translation and outperform baseline methods in terms of model size and training budget. Embodiments of a gradient-based generative saliency method that explains the dynamics of the translation are also presented herein.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for computer learning that can provide improved computer performance, features, and uses. More particularly, the present disclosure relates to energy-based models. Background Art

[0002] Energy-based models (EBMs), parameterized by modern neural networks such as convolutional networks, have recently attracted significant attention in the deep learning community. Applications of EBMs include generating images, videos, 3D volumetric shapes, unordered point clouds, text, molecules, etc., as well as image-to-image translation, out-of-distribution detection, and inverse optimization control. EBMs are characterized by: (i) simplicity—the maximum likelihood learning of EBMs unifies representation and generation in a single model, and (ii) explicitness—EBMs provide an explicit density distribution of the data by training an energy function that assigns low values ​​to observed data and high values ​​to unobserved data.

[0003] However, training EBMs to synthesize diverse and high-fidelity images is not easy. The maximum likelihood estimation (MLE) learning process typically involves Markov chain Monte Carlo (MCMC) to sample from the current model and then update the model parameters given the differences between these samples and the observed data. This "analysis-by-synthesis" learning scheme is challenging because the sampling step is inefficient and unstable when iterating over different modes of the learned model, especially when the energy function is multimodal due to high variation or high-resolution training data.

[0004] Therefore, improved ways to train energy-based models are needed. Summary of the Invention

[0005] The present disclosure relates to computer-implemented methods, non-transitory computer-readable media, and computer program products for training energy-based models.

[0006] One aspect of the present disclosure proposes a computer-implemented method for training an energy-based model, the method comprising: obtaining a set of real images having a current-stage resolution; obtaining a set of current-stage initial synthesized images having a current-stage resolution; obtaining a set of previous-stage synthesized images, wherein the previous-stage synthesized images have a previous-stage resolution lower than the current-stage resolution; given a set of previous-stage synthesized images and an input previous-stage energy-based model having the previous-stage resolution, generating a set of updated previous-stage synthesized images via Markov Chain Monte Carlo (MCMC) sampling; and The method further comprises: generating a set of updated current-stage initial synthesized images at the current-stage resolution by combining the updated set of previous-stage synthesized images upsampled to the current-stage resolution; generating a set of updated current-stage synthesized images via Markov Chain Monte Carlo (MCMC) sampling given the updated set of current-stage initial synthesized images and the current-stage energy-based model having the current-stage resolution as input; updating the current-stage energy-based model based on the difference between the expected values ​​obtained using the set of real images and the current-stage energy-based model and the expected values ​​obtained using the set of updated current-stage synthesized images and the current-stage energy-based model; in response to the current-stage resolution not being the final resolution: modifying the updated current-stage energy-based model to generate an extended energy-based model having a resolution higher than the current-stage resolution; setting the updated current-stage energy-based model as the previous-stage energy-based model for the next iteration and setting its resolution to the previous-stage resolution for the next iteration; setting the extended energy-based model as the current-stage energy-based model for the next iteration and setting its resolution to the current-stage resolution for the next iteration; and returning to the step of obtaining a set of real images for iteration. In response to the current stage resolution being the final resolution, outputting the updated current stage energy-based model as a trained energy-based model with the final resolution.

[0007] Another aspect of the present disclosure provides a computer-implemented method for training an energy-based model, the method comprising: starting from a minimum resolution energy-based model, continuously improving the model resolution capability in stages until a final resolution capability for the energy-based model is reached, each stage comprising the following steps, the steps comprising: updating the current stage energy-based model based on the difference between: (1) expected values ​​obtained using a set of non-synthetic images at the current stage resolution and the current stage energy-based model, and (2) expected values ​​using a set of synthetic images at the current stage resolution and the current stage energy-based model; in response to the current stage energy-based model not having the final resolution capability: adding one or more neural network layers to the updated current stage energy-based model to create a higher resolution energy-based model for the next stage, setting the higher resolution energy-based model as the current stage energy-based model for the next iteration; and returning to the step of updating the current stage energy-based model for the next iteration; and in response to the current stage energy-based model having the final resolution capability, outputting the updated current stage energy-based model as a trained energy-based model with the final resolution.

[0008] Another aspect of the present disclosure provides a non-transitory computer-readable medium comprising one or more instruction sequences, which, when executed by at least one processor, causes the execution of steps for training an energy-based model, the steps comprising: starting from a minimum resolution energy-based model and continuously improving the model resolution capability in stages until a final resolution capability for the energy-based model is reached. Each stage includes the following steps: updating a current stage energy-based model based on the difference between: (1) expected values ​​obtained using a set of non-synthetic images at the current stage resolution and the current stage energy-based model, and (2) expected values ​​obtained using a set of synthetic images at the current stage resolution and the current stage energy-based model; in response to the current stage energy-based model not having final resolution capability: adding one or more neural network layers to the updated current stage energy-based model to create a higher resolution energy-based model for the next stage, setting the higher resolution energy-based model as the current stage energy-based model for the next iteration, and returning to the step of updating the current stage energy-based model for the next iteration; and in response to the current stage energy-based model having final resolution capability, outputting the updated current stage energy-based model as a trained energy-based model with the final resolution.

[0009] A computer program product comprises a computer program, which, when executed by a processor, can implement the above-mentioned method for training an energy-based model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Reference will be made to embodiments of the present disclosure, examples of which may be shown in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although the present disclosure is generally described in the context of these embodiments, it should be understood that this is not intended to limit the scope of the present disclosure to these specific embodiments. Items in the drawings may not be drawn to scale.

[0011] Figure 1 An implementation of an energy-based model (EBM) with gradient-based, coarse-to-fine sampling according to an embodiment of the present disclosure is depicted.

[0012] Figure 2 A multi-stage coarse-to-fine EBM training strategy according to an embodiment of the present disclosure is shown.

[0013] Figure 3 The figure illustrates a smooth sampling process according to an embodiment of the present disclosure.

[0014] Figure 4 A computer-implemented method for training an energy-based model according to an embodiment of the present disclosure is depicted.

[0015] Figure 5 The diagram illustrates resolution expansion for energy-based models according to an embodiment of the present disclosure.

[0016] Figure 6 Depicted are methods for generating images using a trained energy-based model in accordance with embodiments of the present disclosure.

[0017] Figure 7 The diagram illustrates image conversion according to an embodiment of the present disclosure.

[0018] Figure 8 The above activation function and its derivative according to an embodiment of the present disclosure are visually presented.

[0019] Figure 9 Depicts a simplified block diagram of a computing device / information processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In the following description, for purposes of explanation, specific details are set forth to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of the present disclosure described below can be implemented in various ways (such as processes, apparatuses, systems, devices, or methods) on tangible computer-readable media.

[0021] The components or modules shown in the accompanying drawings are illustrative of embodiments of the present disclosure and are intended to avoid obscuring the present disclosure. It should be understood that throughout this discussion, components may be described as separate functional units (which may include subunits), but those skilled in the art will recognize that various components or portions thereof may be separated into separate components or integrated together (including, for example, in a single system or component). It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.

[0022] In addition, the connections between components or systems in the drawings are not intended to be limited to direct connections. Instead, the data between these components may be modified, reformatted, or otherwise changed by intermediate components. In addition, additional connections or fewer connections may be used. It should also be noted that the terms "couple," "connect," "communicatively coupled," "interface," "interface," or any of their derivatives should be understood to include direct connections, indirect connections through one or more intermediate devices, and wireless connections. It should also be noted that any communication, such as a signal, response, reply, confirmation, message, query, etc., may include one or more information exchanges.

[0023] References in this specification to "one or more embodiments," "preferred embodiments," "embodiments," "some embodiments," etc., indicate that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the present disclosure, and may be included in more than one embodiment. Furthermore, the appearances of these phrases in various places in this specification are not necessarily all referring to the same embodiment or multiple embodiments.

[0024] The use of certain terms in various places in this specification is for illustrative purposes and should not be construed as limiting. Services, functions, or resources are not limited to a single service, a single function, or a single resource; the use of these terms may refer to a distributable or aggregated grouping of related services, functions, or resources. The terms "include," "comprising," "includes," and "comprising" or any of their variations should be understood as open-ended terms, and any listings thereafter are examples and are not intended to be limited to the listed items. A "layer" may include one or more operations. The words "optimal," "optimize," "optimize," and the like refer to improvements in results or processes without requiring that a specific result or process reach an "optimal" or peak state. As used herein, memory, database, repository, data store, table, hardware, cache, and the like may be used to refer to system components or components to which information may be input or otherwise recorded.

[0025] In one or more embodiments, stopping conditions may include: (1) a set number of iterations have been performed; (2) a certain amount of processing time has been reached; (3) convergence (e.g., the difference between consecutive iterations is less than a first threshold); (4) divergence (e.g., performance deteriorates); (5) an acceptable result has been achieved; (6) all data has been processed.

[0026] Those skilled in the art will recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order described herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0027] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference / document mentioned in this patent document is incorporated herein by reference in its entirety.

[0028] It should be noted that any experiments and results provided herein are provided by way of illustration and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of this patent document.

[0029] It should also be noted that although the embodiments described herein may be within the context of image generation, aspects of the present disclosure are not limited thereto. Therefore, aspects of the present disclosure may be applied or adapted to other contexts.

[0030] A. Overview

[0031] Training energy-based models to synthesize diverse and high-fidelity images presents significant challenges. As discussed above, "analysis-by-synthesis" learning schemes are challenging due to inefficient and unstable sampling steps. In particular, when the energy function is multimodal due to high variation or high-resolution training data, it is non-trivial for the MCMC chain to traverse the modes of the learned model. However, it is well known that manifolds residing in the downsampled low-dimensional image space are smoother than their original high-dimensional counterparts. Therefore, learning EBMs from low-dimensional data is more stable and faster in terms of convergence than learning from high-dimensional data.

[0032] This paper proposes an implementation of a multi-stage, coarse-to-fine energy-based modeling and sampling methodology, as well as an implementation of a method for training the model; for convenience, this implementation may generally be referred to as CF-EBM.

[0033] Figure 1 The figure illustrates a coarse-to-fine EBM with gradient-based MCMC sampling according to an embodiment of the present disclosure. Figure 1As presented in , the implementation starts by learning a coarse-level EBM on easier low-resolution images, and then smoothly transitions to learning finer-level EBMs as learning progresses by adding new layers that take into account higher-resolution information. Gradient-based short-term MCMC (e.g., Langevin dynamics) can be used for efficient sampling. In terms of modeling, coarse-level training can be used to discover the global structure of the image, while fine-level training will then gradually refine the image details. Some recent work has shown the advantages of such incremental learning. However, no work has yet focused on incremental learning of EBMs that combines bottom-up representations with top-down sampling in a single network. In addition, as Figure 1 As shown, top-down gradient information 130 for synthesis flows (dashed lines) from a coarse-level layer (e.g., 110) toward a fine-level layer (e.g., 115). Thus, during expansion from coarse to fine, embodiments may use coarse-level synthesis to help fine-level synthesis stabilize sampling. x E (x) 130 flow can be viewed as mapping a low-dimensional latent “prior” without multimodality to a high-dimensional target by adding finer details, e.g. Figure 1 As shown in Figure 3, the proposed method is insensitive to the multimodal manifold structure embedded in the high-dimensional environment space. In addition, this coarse-to-fine expansion and sampling scheme can be used for high-fidelity synthesis in several vision tasks.

[0034] Furthermore, embodiments of a unilateral energy-based unsupervised image-to-image translation methodology are presented herein that is scalable to high resolution. In one or more embodiments, the methodology is immediately usable with CF-EBM embodiments by using iterative Langevin dynamics without requiring cycle consistency or geometric constraints. Specifically, embodiments can learn an EBM for a target domain using Langevin dynamics initialized with examples from a source domain. In one or more embodiments, the resulting converter is a short-term MCMC. Compared to prior work that learned black-box encoder-decoder networks between domains, embodiments herein are more interpretable in the sense that they can be interpreted by visualization methods that use gradients to visualize the most critical regions (i.e., generate saliency) when translating an image from a source domain to a target domain.

[0035] Some contributions provided by the embodiments herein include, but are not limited to, the following:

[0036] To the best of the inventors' knowledge, this paper is the first to train an EBM using an "analysis-by-synthesis" approach, using a multi-stage coarse-to-fine dilation and sampling strategy. Furthermore, several techniques for improving the EBM, such as smoothed activation, are proposed. In particular, embodiments are the first to train a pure EBM on synthetic 512×512 images.

[0037] In this paper, we propose a novel implementation of an energy-based unsupervised image-to-image translation method that is substantially different from all other existing generative adversarial network (GAN)-based methods. We demonstrate significant results in terms of translation quality as well as time and memory efficiency.

[0038] Extensive experiments are conducted to validate our implementation and various applications, including image generation, denoising, inpainting, out-of-distribution detection, and unsupervised image translation. Strong results show that our implementation outperforms or is competitive with existing methods.

[0039] B. Related Work

[0040] 1. Energy-based generative models

[0041] One of the main challenges in training EBMs via maximum likelihood estimation (MLE) is extracting appropriate samples from the model, especially when the energy function is parameterized by a highly nonlinear convolutional neural network (CNN or ConvNet). Contrastive divergence (CD) can be an efficient way to train EBMs when the MCMC chain is initialized from the data distribution, but it suffers from bias. Another proposed approach is to train EBMs using energy-based corrections that employ more tractable models. Noise contrastive estimation (NCE) and introspective neural networks (INN) fall into this category. Generative Collaborative Networks (CoopNet) attempt to train EBMs using MCMC teaching with a generator or variational autoencoder (VAE) as an amortized sampler. Triangle Divergence attempts to train EBMs via amortized MCMC via VAEs without MCMC. However, these frameworks struggle to scale to and model multimodal data. Several strategies have been proposed to improve EBM training. Gao et al. (2018) (Ruiqi Gao, Yang Lu, Junpei Zhou, Song-ChunZhu, and Ying NianWu, Learning GenerativeConvNets via Multi-Grid Modeling and Sampling), IEEE Conference on Computer Vision and Pattern Recognition Conference Proceedings ( CVPR), pp. 9155-9164, 2018, which is incorporated herein by reference in its entirety) adopts a multi-grid approach to train multiple EBMs at different grids simultaneously, where the EBMs of the coarser grids are used to initialize the image generation of the EBMs at the finer grids. However, optimizing multiple EBMs and sampling from them will result in time and memory inefficiencies. To make the training stable, other methods attempt to add Gaussian white noise to the observed data to obtain noisy synthetic images. In contrast, in one or more embodiments, a single EBM is trained via a coarse-to-fine growing strategy. In addition, embodiments may also adopt some of the improved techniques discussed below. Through smooth parameter training and image sampling, embodiments can maintain the compatibility of EBMs and synthesize high-fidelity images.

[0042] Some other work aims to improve score-based generative models. Although not classified as EBMs, the use of Langevin dynamics for sampling makes the proposed techniques potentially useful. For example, Block et al. (Adam Block, Youssef Mroueh, Alexander Rakhlin, and Jerret Ross, Fast Mixing Of Multi-Scale Langevin Dynamics Under the Manifold Hypothesis, arXiv preprint) arXiv:2006.11166 , 2020, which is incorporated herein by reference in its entirety) proposed mr-Langevin, which applies multi-resolution Langevin dynamics to help improve the slow mixing problem of using pre-trained super-resolution networks. Song and Ermon (Yang Song and Stefano Ermon, Improved techniques for training score-based generative models, arXiv preprint arXiv:2006.09011 , 2020, which is incorporated herein by reference in its entirety, proposes applying an exponential moving average to the parameters at the time of sampling. However, these techniques still struggle to scale and model multimodal data. Worse still, consistently adding Gaussian noise to the observed data to stabilize training results in high-variance learning and noisy synthetic samples. In contrast, embodiments include new neural architectures with improved techniques. These embodiments of the learned model preserve the compatibility of EBM and synthesize high-fidelity images.

[0043] 2. Unsupervised Image-to-Image Translation

[0044] GAN-based solution. The goal of image-to-image translation is to learn the mapping direction between two unpaired domains (i.e., F : X → Y and G: Y → X ). Recent successes are all based on adversarial learning, e.g., CycleGAN, UNIT, MUNIT, DRIT, and U-GAT-IT. These methods typically train two GANs with two-level learning objectives: (i) distribution level: using two adversarial losses to capture the style differences between the source and target domains; and (ii) instance level: to address the difficulties of the unpaired setting, a cycle consistency loss is adopted for content preservation. This loss enables instance-level supervision to regularize its training by enforcing the two mappings to be bijective functions between the two domains. One of the main limitations of this line of work is that F and G The cross-correlation between the two domains is enhanced by optimizing them simultaneously towards roughly reversible directions. To some extent, this may exacerbate the long-standing instability problem of GANs and increase training time costs. It also requires sophisticated designs, such as instance normalization, position normalization, and attention mechanisms, as well as the use of large-scale architectures to ensure that the model discovers the inherent differences between the two domains and preserves the source content during the translation. There is also research on unilateral unsupervised image translation, such as DistanceGAN, GcGAN, and CUT, which apply geometric or contrast constraints. The embodiments herein address this problem from the perspective of EBM, which differs from GAN-based methods. The concise EBM solution embodiments herein utilize EBM as a built-in object for distribution-level statistical matching to achieve unilateral image translation. It transfers style through MCMC and preserves the source content without using a cycle consistency loss. The embodiments demonstrate better performance with less time and memory. Another difference between the method embodiments of this patent document and GAN-based methods is the natural interpretability of Langevin dynamics. It provides a gradient-based saliency map to visualize the essential regions that distinguish the two domains.

[0045] C. Methodology Implementation

[0046] In this section, we propose an EBM learning framework and then describe the implementation of the CF-EBM method. Afterwards, we show how the model implementation can be generalized for the task of unsupervised image-to-image translation.

[0047] 1. MCMC-based Maximum Likelihood Learning with Energy-based Model Implementation

[0048] Let x∈ Dis an observed example, such as an image. The energy-based model can be defined as follows:

[0049] (1)

[0050] in E θ (x): D → is the energy function defined by the bottom-up ConvNet, and the parameters of ConvNet are given by θ express. is an intractable normalizing constant or partitioning function. Given N observed data points ,in is the unknown data distribution, which can be obtained by maximizing the log-likelihood To train the model. The derivative of the negative log-likelihood is given by:

[0051] (2)

[0052] Among them, according to The second expected term of is intractable and can be approximated via MCMC. Based on this, the EBM can be updated by gradient descent. In one or more embodiments, in order to approximate For sampling, we can use the gradient-based Langevin dynamics that recursively computes the following steps:

[0053] (3)

[0054] in, is the step size of the Langevin step and also the variance of the Gaussian noise In theory, to ensure convergence, MCMC is usually performed with an infinite number of steps and infinitesimal step sizes. However, this is not practical for training EBMs. In one or more embodiments, short-term MCMC can be used, which starts with a fixed noise distribution and runs a fixed MCMC for both the training and testing phases. T Langevin. Training with short-term MCMC may result in biased estimates of the EBM, but the learned short-term MCMC is still an effective generator, which enables embodiments to synthesize realistic images and train models efficiently. In one or more embodiments, the step size is kept constant and the noise variance is linearly decayed to 0; however, it should be noted that different methods, different values, or both can be used.

[0055] 2. CF-EBM: Implementation of EBM learning from coarse to fine

[0056] One of the key contributions of this paper is a multi-stage, coarse-to-fine expansion and sampling approach for training EBMs. The key concept is to incrementally scale the EBM from low resolution (coarse model) to high resolution (fine model) by gradually adding new layers to the energy function. In this way, both the stability and time efficiency of training the EBM benefit. Ultimately, in one or more embodiments, the highest-resolution EBM is retained for image generation using short-term MCMC sampling. Figure 2 A multi-stage coarse-to-fine EBM training strategy according to an embodiment of the present disclosure is shown. Methodology 1 also depicts a pseudo-code implementation of an embodiment of the multi-stage coarse-to-fine EBM training strategy.

[0057] Methodology 1: Implementation of CF-EBM training

[0058]

[0059] For the purpose of illustration, let S represents the total number of training phases, and (x (s) , s = 1, …, S ) represents a multi-resolution version of image x, where x （1） is the minimum resolution version of x and x （S） = x. For example, Figure 2 As shown, the resolution of the input image continues to increase x （1） 225, x （2） 230 and x （3） 235. For each x （s） , which can be easily generated by average pooling with pooling size x （s–1） In the stage s , model and energy function E （s） Associated to x （s） In one or more embodiments, the model 205 starts with an initial minimum resolution in stage 1. When training proceeds to the next stage s When +1, a randomly initialized layer can be added at the bottom to expand the resolution and obtain a new energy function E （s+1） For example, in Figure 2 In the example, when the training stage switches from stage 2 to stage 3, the energy function of the resolution 16×16 (215) is converted to E （2） Add the bottom block with a resolution of 32×32 (220) to get E （3） After that, in one or more embodiments, only the training phase 3 E（3） .

[0060] Compared to PGAN, which proposes a progressive growth strategy to train GAN, the CF-EBM implementation merges learning (Eq. (2)) and sampling (Eq. (3)) into a single neural network. E θ (x), the neural network is essentially and functionally different from the PGAN that applies the discriminator to guide learning and the generator to generate samples. Therefore, training EBM via a multi-stage growth strategy can be considered more challenging and significant. Using the newly expanded energy function E （s+1） and the expanded training examples x （s+1） , attention should be paid to phase transition to avoid instability issues. In one or more embodiments, CF-EBM addresses this issue from the following two aspects:

[0061] a) Implementation of smoothed EBM learning

[0062] Let Expand(·) denote the expansion block. In one or more embodiments, to expand the EBM resolution at each stage, Expand(·) may be added to the bottom of the energy function to increase its resolution. The embodiments herein double its resolution, but other increments may also be used. The block may decay smoothly to prevent abrupt impacts on the trained coarse-level energy function. Specifically, in one or more embodiments, Expand(·) is the composite of the original block Fade(·) and the decay block Fade(·). In one or more embodiments, Expand(x) is defined as Expand(x) = β Primal (x) + (1 – β )Fade(x), where β is a decay factor that gradually increases from 0 to 1 as more data is consumed in training. Therefore, the model first relies on the well-trained coarse-level layers and gradually transitions attention to the newly added fine-level layers. In one or more embodiments, when β When ∈ R₁ reaches 1, the auxiliary attenuation block may be discarded. In one or more embodiments, Primal (·) includes two convolutional layers; and Fade (·) includes a convolutional layer followed by 2×2 average pooling to connect the previous coarse level layer.

[0063] b) Implementation of smoothed MCMC sampling

[0064] In one or more embodiments, smooth sampling may be implemented implicitly, explicitly, or both.

[0065] (i) Implementation of implicit smoothing

[0066] Consider the stage s – 1 to stage s The energy function can be obtained by E （s） (x （s） ) = E （s–1） (Expand(x （s） )) to expand. The gradient for Langevin sampling in equation (3) E （s） (x （s） ) can be expanded as follows:

[0067]

[0068] Initially, due to the attenuation factor β is small, so the newly added primitive block Primal(·) whose weights can be randomly initialized will not immediately have a big impact on the sampling. In one or more embodiments, only the coarse level function E （s–1） and the auxiliary attenuation block Fade(·) make significant contributions to image synthesis. β Increase, Primal (x （s） ) becomes more and more well-trained. At the same time, in the sampling Fade (x （s） ) decline and Primal (x （s） ) gradually leads, so the synthesized image will become clearer and clearer.

[0069] (ii) Implementation of explicit smoothing

[0070] In one or more embodiments, to mitigate the impact of sudden model expansion on sampling, a well-trained coarse model can be used. E （s–1） To initialize the newly extended model E （s） Specifically, in one or more embodiments, at stage s , the implementation method is to first run T （s–1） (1 – β ) Step from Langevin E （s–1） Generate low-resolution samples. Then you can Upsample ( ) upsample these samples (e.g., 2×) and mix them with uniform noise to initialize E （s） MCMC sampling. With the decay factorβ As , this MCMC initialization aids decay from the coarse model. Eventually, when β When it reaches 1, MCMC initialized with pure noise can be used to directly obtain E （s） Medium sampling. Figure 3 FIG. 3 shows the smooth sampling process of stage 3 according to an embodiment of the present disclosure.

[0071] In one or more embodiments, although the same number of Langevin steps may be used for two or more phases, different numbers of Langevin steps may be used in different phases. T （1） = 15 steps to run MCMC, and gradually increase the number of steps as the model grows. In the tested implementation, the maximum number of steps is set to 60, although a different maximum value can be used.

[0072] c) Implementation of the alternative method

[0073] Figure 4 A computer-implemented method for training an energy-based model according to embodiments of the present disclosure is described. Embodiments involve a multi-stage coarse-to-fine expansion and sampling strategy, where training begins with an energy-based model having a minimum resolution and continues to improve the model's resolution capability in stages until a final energy-based resolution capability is reached. In one or more embodiments, each stage includes updating the current stage energy-based model (405) based on the difference between: (1) expected values ​​obtained using a set of non-synthetic images at the current stage resolution and the current stage energy-based model, and (2) expected values ​​obtained using a set of synthetic images at the current stage resolution and the current stage energy-based model.

[0074] In response to the current stage energy-based model not having final resolution capability (410), one or more neural network layers are added to the updated current stage energy-based model to create a higher resolution energy-based model for the next stage (415). The higher resolution energy-based model is set as the current stage energy-based model for the next iteration, and the process returns to the step of updating the current stage energy-based model for the next iteration (405).

[0075] In one or more embodiments, in response to the current stage energy-based model having final resolution capability, the updated current stage energy-based model may then be output as a trained energy-based model with final resolution ( 420 ).

[0076] In one or more embodiments, if there is a previous stage energy-based model, a set of upsampled images generated by the previous stage energy-based model is used to obtain a set of synthetic images, where the set of upsampled images has been upsampled from the previous stage resolution to the current stage resolution.

[0077] If this is the first iteration, no previous-stage model exists. Therefore, in one or more embodiments, a set of non-synthesized images having the current-stage resolution can be obtained by sampling from a set of real images. Alternatively, a set of synthesized images having the current-stage resolution can be obtained by: generating a set of initial synthesized images having the current-stage resolution by sampling from a distribution (e.g., a uniform distribution); and generating the set of synthesized images via Markov Chain Monte Carlo (MCMC) sampling, given the set of initial synthesized images and the current-stage energy-based model having the current-stage resolution as input.

[0078] In one or more embodiments, a computer-implemented method may include obtaining various images. For example, in one or more embodiments, a set of non-synthetic images having a current-stage resolution may be obtained by sampling from a set of real images. Additionally, the set of synthetic images having the current-stage resolution may be obtained by performing the following steps, the steps comprising: generating a set of current-stage initial synthetic images having the current-stage resolution by sampling from a distribution; generating a set of previous-stage initial synthetic images having the previous-stage resolution by sampling from a distribution; generating a set of updated previous-stage synthetic images via Markov Chain Monte Carlo (MCMC) sampling given the set of previous-stage initial synthetic images and a previous-stage energy-based model having the previous-stage resolution as input; generating a set of smoothed synthetic images having the current-stage resolution by combining the set of current-stage initial synthetic images with a set of updated previous-stage synthetic images that have been upsampled to the current-stage resolution; and generating the set of synthetic images via Markov Chain Monte Carlo (MCMC) sampling given the set of smoothed synthetic images and the current-stage energy-based model having the current-stage resolution as input.

[0079] In one or more embodiments, a set of non-synthesized images at the current-stage resolution is obtained by sampling from a set of training images at the final resolution. If the current-stage resolution is not the final resolution, training images from the set of training images may be downsampled to the current-stage resolution, and the downsampled training images may be sampled to obtain a set of real images at the current-stage resolution. Note that this order need not be followed; that is, images may be selected first and then downsampled to the appropriate resolution. If the current-stage resolution is the final resolution, since the images are at the correct resolution, sampling may be performed directly from the set of training images to obtain the set of real images.

[0080] In one or more embodiments, the step of adding one or more neural network layers to the updated current-stage energy-based model to create a higher-resolution energy-based model for the next stage may include: removing a red, green, and blue (RGB) block that transforms a three-channel RGB image into a z-channel feature map, where z is the number of channels per kernel in the bottom convolutional layer of the updated current-stage energy-based model and having the current-stage resolution; and adding an expansion block and a new RGB block, each having a higher resolution.

[0081] In one or more embodiments, the expansion block may include: an original block including two convolutional layers, wherein the first convolutional layer does not change the spatial size and the second convolutional layer has a downsampling stride; and an attenuation block including a convolutional layer followed by average pooling to connect the previous coarse-level layer.

[0082] Given a trained energy-based model at a final resolution, it can be used to generate a synthetic image at the final resolution. For example, the image can be generated using a methodology such as that described below with respect to Methodology 2. Note that, as explained in more detail below, the trained energy-based model can be trained for a domain different from the input image, and the step of using the trained energy-based model at the final resolution to generate a synthetic image at the final resolution can involve generating a synthetic image at the final resolution that represents an image-to-image transformation.

[0083] 3. Implementation Methods of the Exemplary Architecture Design

[0084] By way of illustration, the neural network architecture for the energy function of the model at 256×256 resolution is given below in Table 1. This neural network architecture is used for the experiments discussed in more detail in Section D.

[0085] Table 1: Neural network architectures with energy functions at resolution 256×256. c is the channel multiplier. In the experiments of this paper, for all experiments, cSet to c = 32, but other values ​​can be used.

[0086]

[0087] In this embodiment, each expansion block includes a primitive block Primal (·) and a decay block Fade (·). Primal (·) includes two convolutional layers, where the first layer does not change the spatial size and the second layer has a downsampling stride of 2. The kernel size is 3×3. The upper Primal (·) (no downsampling) uses a kernel size of 3×3 for the first layer and a kernel size of 4×4 for the second layer. This embodiment also has a residual connection in Primal (·), which is a single convolutional layer followed by an average pooling of size 2×2. In this embodiment, the residual connection is not applied in the last Primal (·) block. FromRGB (·) is to transform a three-channel RGB (red, green, and blue) image into z A special block of channel feature maps, where z is the number of channels per kernel in the bottom convolutional layer, in order to connect the input image to the bottom layer of the energy function. Figure 5 A description of an implementation of the process of scaling the resolution from 16×16 to 32×32 is provided in .

[0088] Figure 5 FIG2 illustrates the expansion of the resolution from 16×16 to 32×32 according to an embodiment of the present disclosure. The FromRGB(·) block 515 at the bottom of the energy function 505 with a resolution of 16×16 is removed, and then an expansion block 520 and a new FromRGB(·) block 535 are added. The expansion block 520 includes an original block 525 and an attenuation block 530. The new FromRGB(·) block 535 is used to connect the input RGB image 540 to the newly added original block 525.

[0089] 4. Implementation Methods Using Trained EBM

[0090] After training, the target EBM is obtained. E （S） An example sampling process through the learned model is presented below in Method 2.

[0091] Methodology 2: Implementation of CF-EBM Sampling

[0092]

[0093] In one or more embodiments, the MCMC starts with a uniform noise distribution and then runs Step Langevin update to generate samples. It should be noted that in one or more embodiments, the initial distribution and the number of Langevin steps are the same in both training and sampling. If MCMC is run with more steps than used in the training phase, oversaturation may occur.

[0094] Figure 6 A method for generating an image using a trained EBM according to an embodiment of the present disclosure is depicted. As shown, a trained EBM is obtained (605) using a multi-stage training process, such as described herein, and one or more input samples are also obtained (610). In one or more embodiments, the input samples may be sampled from a distribution. Alternatively, the input samples may be samples based on the intended use of the EBM. For example, for image-to-image translation, the input samples may be from a source domain, and the output image will be in a target domain.

[0095] Given a trained EBM and input samples, a synthetic image can be generated by performing multiple steps of Langevin updates (615). Once the number of steps has been completed, the synthetic image can be output (620).

[0096] As will be described below, the trained EBM can be used for a wide range of applications including but not limited to image generation, image restoration (painting and denoising), and out-of-distribution (OOD), and as explained below, it can be used for unpaired image-to-image translation.

[0097] 5. Implementation of Energy-Based Unsupervised Image-to-Image Translation

[0098] In one or more embodiments, the embodiments generalize to unpaired image-to-image translation and extend it to high-resolution image datasets. 705 and 710, given two real distributions p (x): x∈ and p (y): y∈ , the goal is to learn two energy functions and , and image-to-image translation using its short-term Langevin dynamics. Here, since the other side is straightforward, we explain how to learn For use from arrive In one or more embodiments, the image is transformed from the source domain The example starts with the target distribution of the following short-term Langevin dynamics :

[0099] (4)

[0100] This is also a conversion process. The basic assumption of applying EBM to image conversion is that there are two domains ( ) share the same environment space, where each data can be decomposed into content code c 720, and domain 705 Domain-Specific Style Code 715 or domain 710 725. This assumption can be considered somewhat similar to other partially shared latent space assumptions; however, it is defined directly in the ambient space and is more efficient without relying on supplementary models. Figure 7 As shown above, the goal of the dynamic is to style code Orientation evolution, where gradient descent steps into lower energy regions and the content c As unchanged as possible.

[0101] Generate saliency maps. Using embodiments of the present methodology, image transformation dynamics are performed. Since each dataset contains different types of generative saliency, such as color, shape, and texture, three datasets are selected for experiments. The main benefit of applying embodiments of the EBM generative model is interpretability. In one or more embodiments, in order to make the transformation process interpretable, a gradient-based saliency map is used to highlight the essential aspects that cause the transformation, which is referred to herein as Generate Display Drawing . It can be calculated as the aggregate gradient in equation (4), that is, . It quantifies the magnitude of the change in each pixel of the image transformation. It should be noted that different types of generative saliency maps can be created. For example, according to an embodiment of the present disclosure, experiments on unsupervised image-to-image translation were conducted using gradient-based generative saliency maps. Examples include translation from cat images to dog images, from photographs to Van Gogh-like photographic images, and from oranges to apples.

[0102] 6. Data Perturbation and Activation Function Implementation

[0103] Most well-established EBMs add Gaussian noise to the training data to stabilize training. Similarly, score-based models introduce attenuated noise to improve score estimates. However, additive noise will at least slightly alter the data noise, resulting in blurry synthetic examples. See Appendix E.4 for further discussion. By implementing a coarse-to-fine expansion and sampling strategy, EBM implementations do not rely on data noise for stable training.

[0104] This paper also studies the effect of the activation function used in the energy function. When the data x is continuous, the smoothness of the derivative of the activation function will significantly affect the Langevin sampling process (since the derivative is in Equation (3) ), applying activation functions with continuous gradients everywhere stabilizes sampling. To this end, we analyze and compare different activation functions, including ReLU, LeakyReLU, CELU, Swish, and GELU. We observe that non-smooth activations such as ReLU and LeakyReLU can lead to learning divergence. See Appendix E.5 for details.

[0105] D. Experiment

[0106] In this section, we discuss results on a variety of tasks, including image generation, image restoration (painting and denoising), out-of-distribution (OOD), and unpaired image-to-image translation. Most of these tasks are implemented using the same network architecture and run on a TITAN X (12GB) platform. Detailed architecture and experimental setup examples are provided in Sections C.3 and E.1, respectively. Quantitative evaluation is performed using the Fréchet Inception Distance (FID) and the Kernel Inception Distance (KID), and human perceptual evaluation is performed using the Amazon Turker platform (AMT) (see Appendix E.2 for details). The KID score is more reliable when fewer test images are available (e.g., for image translation). Additional synthetic results can be found in Appendix E.3. A detailed ablation study of activation functions, normalization, and other factors is performed in Appendix E.5. Finally, the model implementation is scaled to 512×512 resolution for the datasets in Appendix E.

[0107] It should be noted that these experiments and results are provided by way of illustration only and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of this patent document.

[0108] 1. Image Generation Experiment

[0109] Dataset. i ) Dataset 1 is a dataset of 60k images (32×32) containing 10 categories; ( ii (iii) Dataset 2 is a face dataset containing over 200k images. To enable fair comparison with previous EBM work, 64×64 resolution is used for quantitative evaluation; (iv) Dataset 3 is a dataset containing 30k high-resolution (512×512) face images.

[0110] Qualitative Results. Quantitative unconditional image generation results on Datasets 1 and 2 are presented in Tables 2 and 3. In Table 3, some works experiment on a center-cropped (140×140) and resized version of Dataset 2, while others learn on the resized version. In all cases, the tested CF-EBM implementations significantly outperform the other methods. In Table 4, the tested implementations are compared with several representative open-source EBM and score-based models in terms of parameter count and relative computation time. All methods rely on Langevin dynamics for sampling. The tested model implementations are sampled using 50 Langevin steps, while all other three methods require 60, 100, or even more steps, respectively. Observe that the tested model implementations consistently excel in terms of memory and computational efficiency, while maintaining the lowest FID on Dataset 1.

[0111] Table 2: FID on Dataset 1

[0112]

[0113] Table 3: FID on Dataset 2

[0114]

[0115] Table 4: Efficiency comparison

[0116]

[0117] Model Analysis: Once the EBM is learned, it can be directly applied to image restoration tasks. Two experiments are conducted on image restoration to demonstrate the pattern coverage and generalization ability of the model implementation on unseen data. (i) Image denoising : Randomly select images from the test dataset of dataset 1, and each image x is added with additive Gaussian noise Interference. For all categories of dataset 1, by using x+ Denoising results obtained by initializing Langevin dynamics This suggests good generalization capabilities. (ii) Image Inpainting : 25% of the area of ​​each image is masked, and the lost areas are restored using the Langevin dynamics of the learned EBM, which can be viewed as an associative memory. For the experimental results, each restored image exhibits a meaningful but different restored pattern from the ground truth. Multi-inpainting is further discussed in Appendix E.3, and the results show good pattern coverage. The nearest neighbors retrieved from the training data of the generated images are also obtained as part of the experimental results, which shows that the tested model implementation does not memorize the training images (i.e., the tested implementation does not suffer from overfitting) but synthesizes new images.

[0118] 2. Experiments for Out-of-Distribution Detection

[0119] Out-of-distribution (OOD) detection is a binary classification problem. It has been shown that the likelihood of an EBM can be used for OOD detection. It is expected that the likelihood output by the EBM will be higher for in-distribution examples and lower for out-of-distribution examples. Based on this principle, OOD detection experiments were conducted and the model implementation was compared with three other generative models: Glow (Kingma and Dhariwal, 2018), Conditional EBM (Du and Mordatch, 2019), and JEM (Grathwohl et al., 2020). All models were trained on Dataset 1, which is considered an in-distribution dataset. Various datasets, including digits, uniform distributions (Uniform), constant distributions (Constant), and interpolated images (Interp), were used as OOD datasets. The area under the receiver operating characteristic curve (AUROC) was used as the evaluation metric. Unlike JEM (Grathwohl et al., 2020), the test method implementation of this patent document does not incorporate any label information during training, yet still demonstrates superior or competitive results, as shown in Table 5.

[0120] Table 5: Comparison of AUROC scores in OOD detection, where higher scores indicate better performance. Except for the present CF-EBM model implementation, the other three models are conditional generative models.

[0121]

[0122] 3. Experiments on Unsupervised Image-to-Image Translation

[0123] Evaluation was conducted using four unpaired image translation datasets, including translating one animal into a different type of animal, translating an outdoor landscape from one season to a different season, translating a photograph into a fine art-style appearance, and translating one fruit or vegetable into a different fruit or vegetable. All images were resized to 256×256 pixels. Further details are provided in Appendix E.8. The method implementations of this patent document were compared with CycleGAN (Zhu et al., 2017), CUT (Park et al., 2020), and two new state-of-the-art methods: U-GAT-IT (Kim et al., 2020) and FQ-GAN (Zhao et al., 2020). CycleGAN is a popular GAN-based image translation framework and is used to demonstrate a basic comparison between GAN-based models with cycle consistency constraints and tested energy-based model implementations without cycle consistency constraints. CUT (Park et al., 2020) is a state-of-the-art single-sided unpaired image translation framework that applies a contrastive loss to preserve content. U-GAT-IT and FQ-GAN are currently the leading models for image translation. Benchmark results are from published papers or viewpoints.

[0124] Quantitative Results: The results are shown in Table 6. In most cases, the tested model implementations achieved better KID scores and human perception scores. At the same time, both training time and model size were significantly reduced. This efficiency stems primarily from two aspects: model design and optimization process.

[0125] (i) Model design EBM implementations use an image-to-scalar energy function, rather than an image-to-image mapping as in GAN-based methods. Without cycle consistency constraints, style is implicitly transferred and content is preserved via Langevin dynamics guided by the energy function. As a result, the model size is significantly smaller than GAN-based models that include a pair of GANs.

[0126] (ii) Optimization process : GAN uses an adversarial loss to match style and a cyclic loss to preserve content. In practice, the two losses are partially separated and optimized alternately. In contrast, embodiments herein apply MLE to a neural network (Equation (2)) and, in practice, find that training requires fewer iterations to converge. Although, in one or more embodiments, an additional Langevin step for sampling may be performed, the overall optimization cost remains low.

[0127] Table 6: Quantitative results of different methods on different image conversion datasets

[0128]

[0129] Qualitative Results: Experimental results show that the tested model implementation generates sharper images with better visual quality than the baseline. It better preserves the source content while evolving the style from the source domain to the target domain.

[0130] E. Appendix

[0131] 1. Experimental Setup

[0132] The implementation uses the Adam optimizer for training, where and The learning rate schedule is set to , and the data feedback plan is set to As a sampling hyperparameter, in most experiments the Langevin step schedule is set to , the Langevin step size is set to 1.0, and the variance of the Langevin noise term is set to .

[0133] Dataset A-64 is generated by preprocessing dataset A using two different settings. (1) Each image is first center-cropped to Then resize it to In Table 3, NCSN (Song and Ermon, 2019), mr-Langevin (Block et al., 2020), and WGAN-GP (Lee et al., 2018b) use this setting. The FID calculation protocol is based on Song and Ermon (2019), where the FID score is calculated between 10k generated images and all test images. (2) Each image is directly resized to The FID is calculated on 40k generated samples. In Table 3, VAE, DCGAN, and short-term EBM follow this setting.

[0134] 2. Evaluation Metrics

[0135] Consider three commonly used metrics to evaluate the quality of synthesized images.

[0136] (i) The Fréchet-Inception distance (FID) (Heusel et al., 2017) compares the distribution of generated images to the distribution of training images. Rather than comparing images directly, FID compares the mean and standard deviation of one of the deeper layers in Inception v3 (Szegedy et al., 2016), a convolutional neural network trained for classification. FID is consistent with human perceptual judgments. Lower FID scores indicate that the model generates higher-quality images. FID is calculated using a dataset splitting protocol similar to that of Zhang et al. (2020).

[0137] (ii) Kernel Inception Distance (KID) (Bi KID (Kowski et al., 2018) is an unbiased metric used to calculate the squared maximum mean difference (MMD) between initial representations, making it more reliable than FID, especially when the number of test images is small. To evaluate image translation performance, KID is calculated between a translated image from a test image in the source domain and a test image in the target domain. Lower KID scores indicate better translation results. KID is calculated following the method proposed by Kim et al. (2020).

[0138] (iii) For human perceptual evaluation using the Amazon Turker platform, we used a setup identical or similar to that in Zhao et al. (2020). Each test image was judged by six participants, who were asked to select the best converted image by considering both visual quality and preservation of source content. Participants were informed of the target domain; for example, they were shown six sample images from the target domain for reference.

[0139] 3. More image synthesis results

[0140] From using Dataset 1 and Dataset 4 ( Table 7 presents the quantitative results of image generation on Dataset 4. The CF-EBM implementation achieves competitive results in terms of FID with fewer parameters.

[0141] Table 7: Dataset 4 ( The channel multiplier is ch = 48.

[0142]

[0143] The short-term Langevin dynamics initialized with a uniform noise distribution used to generate the dataset 2 images are also visualized. Results for image inpainting of dataset 2 test images at 64×64 resolution are also generated. Running the Langevin dynamics on occluded images reveals the diversity of image completion. The results demonstrate that the implementations generalize well to unseen test data and that the MCMC of the learned model implementations exhibits good pattern coverage properties. The tested model implementations complete occluded regions with diverse meaningful facial expressions and varying occlusion positions.

[0144] The generated images are compared to the nearest neighbors retrieved from the training data. The synthesized images are found to be different from the training data, which means that the tested model implementation learns to generate new image patterns rather than reconstructing existing training images.

[0145] 4. Impact of Data Interference

[0146] Almost all energy-based correlation generative models add Gaussian noise ) to perturb the training data and stabilize the training. The effect of this data perturbation was explored. Nijkamp et al. (2019) reported that larger perturbations lead to lower FID scores, and the minimum standard deviation of the additive noise used was In addition, Song and Ermon (2019) applied the standard deviation from Decay to It is expected that an EBM learned from training data with additive noise will remember the noise information even if it is small. Therefore, the synthetic image will look noisy and blurry. The test results show the effect of data interference by showing synthetic images generated by the learning model with different levels of data interference. Note that even if the standard deviation of the additive noise is as small as , the effect reflected in the synthetic image is also visible. From Nijkamp et al. (2019) Figure 2 , Song and Ermon (2020) Figure 1 and Grathwohl et al. (2020) Figure 2 The same observation can be found in both of the synthesized images shown, which all use data perturbation during training. However, the embodiments herein do not rely on additive noise during data perturbation, resulting in better synthesis quality.

[0147] 5. Ablation studies

[0148] Ablation studies mainly examine activation functions, normalization, and layer connections.

[0149] a) Activation Function

[0150] Compare the following activation functions by presenting their formulas and derivatives:

[0151] Rectified Linear Unit (ReLU) (Nair and Hinton, 2010): , and its derivative:

[0152] Leaky Rectified Linear Function (LeakyReLU) (Maas et al., 2013): , and its derivative:

[0153] in It is a shared setting.

[0154] Continuously Differentiable Exponential Linear Unit (CELU) (Barron, 2017): ,in is the shape parameter; and the derivative:

[0155] Swish (Ramachandran et al., 2016): ,in is a sigmoid function, and is a learnable function. Derivative:

[0156] For experiments, Set to .

[0157] Gaussian Error Linear Unit (GELU) (Hendrycks and Gimpel, 2016b): ,in is the cumulative distribution function of the Gaussian distribution. It can be approximated as (1 + .

[0158] Figure 8 The above activation functions and their derivatives are visualized. Clearly, both ReLU and LeakyReLU are not smooth near 0. Table 8 shows the quantitative results of image synthesis using different activation functions on Dataset 1. It was found that models using LeakyReLU or ReLU activation functions tended to diverge earlier. This phenomenon was also observed in experiments on Dataset 2. Therefore, activation smoothness can improve EBM training stability and synthesis quality.

[0159] Since the object in Equation (2) is similar to the Wasserstein GAN, and the bottom-up energy function in EBM acts as the discriminator, it is natural to consider spectral normalization to improve performance and stability. Spectral normalization constraints learn the Lipschitz constant of the neural network parameters, which is widely used to stabilize the training of the GAN discriminator network.

[0160] Table 8: Effects of different activation functions

[0161]

[0162] b) Normalization

[0163] Since the object in Equation (2) can be considered similar to a Wasserstein GAN, and the bottom-up energy function in EBM acts as the discriminator, it is natural to consider spectral normalization (Miyato et al., 2018) to improve performance and stability. Spectral normalization constraints learn the Lipschitz constant of the neural network parameters, which is widely used to stabilize the training of the GAN discriminator network.

[0164] Batch normalization is not appropriate here, since each Langevin step depends on Even if the statistics are not updated in Equation (3), The distribution of is also always changing along the chain. Therefore, the pre-computed statistics of batch normalization are not suitable for asynchronous t of Other normalizations (e.g., instance normalization and layer normalization) are also not suitable in experiments.

[0165] c) Analysis of different components

[0166] We examined how each component affected the performance of the model implementation. The components included progressive growing, residual connections, and spectral normalization. The results are shown in Table 9 below.

[0167] Table 9: Ablation study on dataset 1

[0168]

[0169] From (a) to (c), it is found that both spectral normalization and residual connections can improve the quality of image synthesis. In (d), the performance degrades when coarse-to-fine training is disabled but residual connections and spectral normalization are still maintained. In addition, the time efficiency of training with and without the coarse-to-fine strategy is compared. It is observed that the model implementation without coarse-to-fine training takes approximately 3 times longer to converge than the corresponding implementation. In (e), multiple EBMs are trained sequentially at different resolutions using a coarse-to-fine approach. A fixed coarse EBM is tried when training the model sequentially at higher resolutions, and the coarse EBM is used to initialize the samples of the fine EBM. All EBMs are retained after training. The FID of (e) is found to be worse than learning a single fine EBM in (c).

[0170] 6. Likelihood Assessment

[0171] Experiments were conducted on a likelihood-based dataset, following the same or similar setup as in Du and Mordatch (2019). Annealed Importance Sampling (AIS) was applied to obtain bounds on the partitioning function. As can be seen, the model implementations produced realistic results. Table 10 shows a comparison of the log-likelihoods of the three likelihood-based models.

[0172] Table 10: Log-likelihood in Nats

[0173]

[0174] 7. Implementation of Extended CF-EBM for High-Resolution Synthesis

[0175] To extend the EBM to sample high-resolution images, the intermediate EBM learned in Methodology 1 is integrated. As training progresses to modeling higher resolutions, the coarse-level CF-EBM that generates the initial point for the fine-level CF-EBM sampler remains fixed. In this case, the number of Langevin steps can be reduced to 15 to speed up sampling at each level, meaning that generating a 256×256 image requires 90 steps. On a single TITAN X GPU (12GB), total training time (200k iterations) for Dataset 3 (128×128 and 256×256) is approximately 120 hours and 235 hours, respectively. On a single TITAN V100 GPU, these costs are reduced to 55 hours and 100 hours (tested).

[0176] In this experiment, during training, the initial variance of the noise term for the Langevin dynamics in Equation (3) is fixed to It is observed that, during the testing phase, choosing values ​​different from those used for training can significantly affect the synthesis results. Figure 8 As shown, the noise term is larger It tends to make the synthesized image more realistic and noisier. In the experiment, the channel multiplier is set to c =32 and trained on a single GPU for 5 days. In other examples, samples with a resolution of 512×512 are generated, where the channel multipliers are set to c = 48 and trained on a single GPU for 10 days. Table 11 compares the FID scores of different models on Dataset 3 at a resolution of 128×128. The model implementation achieves competitive performance without using any normalization term.

[0177] Table 11: FID for image generation (128×128). Results from the GAN-based method of Zhang et al. (2020).

[0178]

[0179] 8. Unpaired Image-to-Image Translation

[0180] a) Dataset

[0181] In this patent document, experiments were conducted on five image-to-image translation datasets: (i) Dataset A: This dataset contains selfie and anime domains, each with approximately 3,400 training images and approximately 100 test images. (ii) Dataset B: This dataset is used for collection style transfer. It has approximately 6,300 photographic images and 400 painting images of artists in the training set, and approximately 750 photographic images and approximately 400 painting images in the test set. (iii) Dataset C: This dataset is used for object deformation. It has approximately 1,000 training images and approximately 270 test images for the object 1 class, and approximately 1,019 training images and approximately 250 test images for the object 2 class. (iv) Dataset D: This dataset is used for season conversion. The summer class has approximately 1,300 training images and approximately 300 test images, and the winter class has approximately 900 training images and approximately 240 test images. (v) Dataset E: The number of images of Animal 1 and Animal 2 are approximately 875 and approximately 1,400, respectively. 100 images of Animal 1 and 100 images of Animal 2 are used as testing data.

[0182] b) Experimental setup

[0183] Compared to the setting in image generation, only two changes are made for the unpaired image-to-image translation task: ( i ) The channel multiplier is set to c =16; and (ii) only for Animals 2 → Animal 1 , the learning rate is multiplied by 10.

[0184] c) Comparison with the unpaired transformation (CUT)

[0185] Implement CUT based on open source code. Animals 2 → Animal 1 The most faithful conversion results were obtained on the CF-EBM implementation. Comparing these results with those of the CF-EBM implementation, it can be seen that CUT sometimes creates hallucinations in the tongue, resulting in unfaithful conversion results. This observation was also mentioned in the original paper (Park et al., 2020). In contrast, the CF-EBM implementation produces more faithful conversion results.

[0186] 9. Citations

[0187] Each reference / document mentioned in this patent document, including but not limited to those listed below, is hereby incorporated by reference in its entirety.

[0188] Miko aj Bi kowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton, Demystifying MMD GANs, ed. International Conference on Learning Representations ( ICLR ), 2018.

[0189] Adam Block, Youssef Mroueh, Alexander Rakhlin and Jerret Ross, Fast Mixing of Multi-Scale Langevin Dynamics Under the Manifold Hypothesis, arXiv preprints arXiv:2006.11166 , 2020.

[0190] Wei-Cheng Chang, Chun-Liang Li, Youssef Mroueh, and Yiming Yang, KernelStein Generative Modeling, arXiv preprint arXiv:2007 . 03074 , 2020.

[0191] Ricky TQ Chen, Jens Behrmann, David K ​​Duvenaud, and Jörn-Henrik Jacobsen, Residual Flows for Invertible Generative Modeling, see Advances in Neural Information Processing Systems , pp. 9916–9926, 2019a.

[0192] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby, Self-Supervised GANS via Auxiliary Rotation Loss, see Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ), pp. 12154–12163, 2019b.

[0193] Laurent Dinh, David Krueger, and Yoshua Bengio, NICE: Non-linearindependent components estimation, arXiv Preprint arXiv:1410.8516 , 2014.

[0194] Yilun Du and Igor Mordatch, Implicit Generation and Modeling With Energy-Based Models, see Advances in Neural Information Processing Systems ( NeurIPS ), pp. 3608–3618, 2019.

[0195] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu, Learning Generative Convnets Via Multi-Grid Modeling and Sampling, IEEE Trans. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 9155–9164, 2018.

[0196] Ruiqi Gao, Erik Nijkamp, ​​Diederik P. Kingma, Zhen Xu, Andrew M Dai, and YingNian Wu, Flow Contrastive Estimation of Energy-Based Models, see Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , pp. 7518–7528, 2020.

[0197] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky, Your Classifier Is Secretly An Energy-BasedModel And You Should Treat It Like One, see International Conference on Learning Representations ( ICLR ), 2020.

[0198] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville, Improved Training of Wasserstein GANs, see Advances In Neural Information > Processing Systems , pp. 5767–5777, 2017.

[0199] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler and Sepp Hochreiter, GANs trained by a two time-scale update rule converge to a local Nash Equilibrium, see Advances in Neural Information Processing Systems ( NIPS ), pp. 6626–6637, 2017.

[0200] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen and Timo Aila, Training generative adversarial networks with limited data, Advances in Neural Information Processing Systems ( NeurIPS ), 2020.

[0201] Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lee, U-GAT-IT: Unsupervised Generative Attentional Networks With Adaptive Layer-InstanceNormalization For Image-To-Image Translation, see International Conference on Learning Representations ( ICLR ), 2020.

[0202] Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes, see International Conference on Learning Representations ( ICLR ), 2014.

[0203] Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow withInvertible 1×1 Convolutions, see Advances In Neural Information Processing Systems ( NeurIPS ), pp. 10215–10224, 2018.

[0204] Zengyi Li, Yubei Chen, and Friedrich T. Sommer, Annealed Denoising ScoreMatching: Learning Energy-Based Model In High-Dimensional Spaces, arXiv preprints arXiv: 1910.07762 , 2020.

[0205] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida, Spectral Normalization for Generative Adversarial Networks, see International Conference on Learning Representations ( ICLR ), 2018.

[0206] Erik Nijkamp, ​​Mitch Hill, Song-Chun Zhu, and Ying Nian Wu, Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model, see Advances in Neural Information Processing Systems , pp. 5232–5242, 2019.

[0207] Georg Ostrovski, Will Dabney, and Remi Munos, Autoregressive QuantileNetworks for Generative Modeling, see International Conference on Machine Learning ( ICML ), pp. 3936–3945, 2018.

[0208] Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu, ContrastiveLearning For Unpaired Image-To-Image Translation, see Proceedings of the European Conference on Computer Vision ( ECCV ), 2020.

[0209] Alec Radford, Luke Metz, and Soumith Chintala, Unsupervised representationlearning with deep convolutional generative adversarial networks, see International Conference on Learning Representations ( ICLR ), 2016.

[0210] Yang Song and Stefano Ermon, Generative Modeling by Estimating Gradients of the Data Distribution, see Advances in Neural Information Processing Systems ( NeurIPS ), pp. 11918–11930, 2019.

[0211] Yang Song and Stefano Ermon, Improved Techniques for Training Score-BasedGenerative Models, arXiv preprint arXiv:2006.09011 , 2020.

[0212] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and ZbigniewWojna, Rethinking the inception architecture for computer vision, see Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition ( CVPR ), pp. 2818–2826, 2016.

[0213] Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, etc., Self-Supervised GAN: Analysis and Improvement with Multi-Class Minimax Game, see Advances in Neural Information Processing Systems ( NeurIPS ), pp. 13253–13264, 2019.

[0214] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, AlexGraves, etc., Conditional Image Generation with PixelCNN Decoders, see Advances in neural information processing [[ID= , pp. 4790–4798, 2016.

[0215] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu, Cooperative Training of Descriptor and Generator Networks, ​​ ​ ​ ​ ​ ​ ​ ( ​ ), 42(1):27–45, 2018b.

[0216] Jianwen Xie, Zilong Zheng and Ping Li, Learning energy-based model with variational auto-encoder as amortized sampler, see ​ ​ ​ ​ ​ ​ ​ ( ​ ), 2021d.

[0217] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee, Consistency Regularization for Generative Adversarial Networks, see ​ ​ ​ ​ ​ ( ​ ), 2020.

[0218] Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen, Feature Quantization Improves GAN Training, see ​ ​ ​ ​ ​ ( ​ ), pp. 11376–11386, 2020.

[0219] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros, UnpairedImage-To-Image Translation Using Cycle-Consistent Adversarial Networks, see​ ​ ​ ​ ​ ​ ​ ​ ​ ( ​ ), pp. 2223–2232, 2017.

[0220] F. Some Observations

[0221] To address the training instability and multimodal MCMC sampling difficulties of EBM, this paper proposes an implementation of a multi-stage coarse-to-fine expansion and sampling strategy, which for convenience may be generally referred to as CF-EBM. Compared with various generative models, extensive experiments have demonstrated that the proposed framework implementation has superior performance in terms of sample quality, computational and memory efficiency. The success of CF-EBM can be attributed at least in part to the proposed progressive expansion and sampling strategy, the architectural design of the energy function, and the selective smoothing activation. CF-EBM is believed to be the first pure EBM that can synthesize high-assurance images and also has unsupervised image-to-image translation capabilities. Those skilled in the art will recognize that different (such as larger) architectures can be used to achieve better performance, and that the implementation can be applied to or adapted for other energy-based applications.

[0222] G. Computing System Implementation

[0223] In embodiments, aspects of this patent document may relate to, include, or be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any means or combination of means operable to calculate, compute, determine, classify, process, transmit, receive, retrieve, originate, route, exchange, store, display, communicate, visualize, detect, record, reproduce, process, or utilize any form of information, intelligence, or data. For example, a computing system may be or include a personal computer (e.g., a laptop), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a phablet, a tablet, etc.), a smartwatch, a server (e.g., a blade server or a rack-mount server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources (such as a central processing unit (CPU) or hardware or software control logic), read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more drives (e.g., a hard drive, a solid-state drive, or both), one or more network ports for communicating with external devices, and various input and output (I / O) devices such as a keyboard, mouse, touch screen, stylus, microphone, camera, trackpad, display, etc. A computing system may also include one or more buses operable to transmit communications between the various hardware components.

[0224] ​ A simplified block diagram depicts an information processing system (or computing system) according to an embodiment of the present disclosure. Although it should be understood that computing systems can be configured differently and include different components, including ​ Although fewer or more components may be shown in , it should be understood that the functionality shown for system 900 is operable to support various implementations of computing systems.

[0225] like ​ As shown, computing system 900 includes one or more CPUs 901 that provide computing resources and control the computer. CPU 901 may be implemented as a microprocessor or the like and may also include one or more graphics processing units (GPUs) 902 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, one or more GPUs 902 may be incorporated into a display controller 909, such as part of one or more graphics cards. System 900 may also include system memory 919, which may include RAM, ROM, or both.

[0226] like ​As shown, multiple controllers and peripheral devices may also be provided. Input controller 903 represents an interface to various input devices 904. Computing system 900 may also include a storage controller 907 for interfacing with one or more storage devices 908. Each of these storage devices may include storage media (such as tape or disk) or optical media (which may be used to record programs of instructions for operating systems, utilities, and applications, which may include implementations of programs implementing aspects of the present disclosure). Storage devices 908 may also be used to store processed data or data to be processed in accordance with the present disclosure. System 900 may also include a display controller 909 for providing an interface to a display device 911. Display device 911 may be a cathode ray tube (CRT), a display, a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. Computing system 900 may also include one or more peripheral controllers or interfaces 905 for one or more peripheral devices 906. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, and the like. The communication controller 914 can interface with one or more communication devices 915, which enable the system 900 to connect to remote devices via any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area networks (LANs), wide area networks (WANs), storage area networks (SANs), or via any suitable electromagnetic carrier signals, including infrared signals. As shown in the depicted embodiment, the computing system 900 includes one or more fans or fan trays 918 and one or more cooling subsystem controllers 917, which monitor the thermal temperature of the system 900 (or components thereof) and operate the fans / fan trays 918 to help regulate the temperature.

[0227] In the illustrated system, all major system components may be connected to bus 916, which may represent more than one physical bus. However, various system components may or may not be physically close to one another. For example, input data and / or output data may be remotely transmitted from one physical location to another. Additionally, programs implementing aspects of the present disclosure may be accessed from a remote location (e.g., a server) via a network. Such data and / or programs may be transmitted via any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as compact discs (CDs) and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0228] Aspects of the present disclosure may utilize instructions for one or more processors or processing units to perform the steps encoded on one or more non-transitory computer-readable media. It should be noted that the one or more non-transitory computer-readable media should include volatile memory and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. Hardware-implemented functions can be implemented using ASICs, programmable arrays, digital signal processing circuits, etc. Therefore, the term "means" in any claim is intended to cover both software and hardware implementations. Similarly, the term "computer-readable medium or media" as used herein includes software and / or hardware having an instruction program implemented thereon, or a combination thereof. With these alternative implementations contemplated, it should be understood that the drawings and accompanying description provide those skilled in the art with the functional information required to write program code (i.e., software) and / or fabricate circuitry (i.e., hardware) to perform the desired processing.

[0229] It should be noted that embodiments of the present disclosure may also relate to computer products having non-transitory, tangible, computer-readable media embodied thereon with computer code for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be known or available to those skilled in the relevant art. Examples of tangible computer-readable media include: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CDs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as ASICs, PLDs, flash memory devices, other non-volatile memory devices (such as 3D XPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code (e.g., code produced by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. Embodiments of the present disclosure may be implemented in whole or in part as machine-executable instructions in program modules that are executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located locally, remotely, or both.

[0230] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. Such program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0231] Those skilled in the art will recognize that computing system or programming language is not important to the practice of the present disclosure. Those skilled in the art will also recognize that a plurality of the above elements can be physically and / or functionally divided into modules or / or submodules, or combined together.

[0232] Those skilled in the art will understand that the foregoing examples and implementations are illustrative and do not limit the scope of the present disclosure. It is intended that all permutations, enhancements, equivalents, combinations, or improvements of the present disclosure that would be apparent to those skilled in the art upon reading this specification and studying the accompanying drawings are included within the true spirit and scope of the present disclosure. It should also be noted that the elements of any claim may be arranged differently, including in multiple dependencies, configurations, and combinations.

Claims

1. A computer-implemented method for training an energy-based model, the method comprising: Obtain a set of real images with the current stage resolution; Obtaining a set of current-stage initial composite images having the current-stage resolution; Obtaining a set of previous stage synthesized images, wherein the previous stage synthesized images have a previous stage resolution lower than the current stage resolution; Given the set of previous-stage synthesized images and a previous-stage energy-based model having an input at the previous-stage resolution, generating a set of updated previous-stage synthesized images via Markov Chain Monte Carlo sampling; generating a set of updated current-stage initial composite images having the current-stage resolution by combining the set of current-stage initial composite images with the set of updated previous-stage composite images that have been upsampled to the current-stage resolution; generating a set of updated current-stage synthetic images via Markov Chain Monte Carlo sampling given the set of updated current-stage initial synthetic images and a current-stage energy-based model having an input at the current-stage resolution; updating the current-stage energy-based model based on a difference between expected values ​​obtained using the set of real images and the current-stage energy-based model and expected values ​​obtained using the set of updated current-stage synthetic images and the current-stage energy-based model; In response to the current stage resolution not being the final resolution: modifying the updated current-stage energy-based model to generate an extended energy-based model having a resolution higher than the current-stage resolution; Setting the updated current-stage energy-based model as the previous-stage energy-based model for a next iteration, and setting its resolution to the previous-stage resolution for the next iteration; Setting the extended energy-based model as the current-stage energy-based model for the next iteration and setting its resolution to the current-stage resolution for the next iteration; as well as Return to the step of obtaining a set of real images for iteration; as well as In response to the current stage resolution being the final resolution, outputting the updated current stage energy-based model as a trained energy-based model with the final resolution.

2. The computer-implemented method of claim 1 , further comprising: In response to the first iteration: obtaining a set of real images with a current stage resolution, where the current stage resolution is a lowest resolution; Obtaining a set of current-stage initial composite images having the current-stage resolution; generating a set of updated current-stage synthetic images via Markov Chain Monte Carlo sampling given the set of current-stage initial synthetic images and a current-stage energy-based model having an input of the current-stage resolution; updating the current-stage energy-based model based on a difference between expected values ​​obtained using the set of real images and the current-stage energy-based model and expected values ​​obtained using the set of updated current-stage synthetic images and the current-stage energy-based model; In response to the current stage resolution not being the final resolution: modifying the updated current-stage energy-based model to generate an extended energy-based model having a resolution higher than the current-stage resolution; Setting the updated current-stage energy-based model as the previous-stage energy-based model for a next iteration, and setting its resolution to the previous-stage resolution for the next iteration; Setting the extended energy-based model as the current-stage energy-based model for the next iteration and setting its resolution to the current-stage resolution for the next iteration; as well as Return to the step of obtaining a set of real images and iterate; as well as In response to the current stage resolution being the final resolution, outputting the updated current stage energy-based model as a trained energy-based model with the final resolution.

3. The computer-implemented method of claim 1 , wherein: The step of obtaining a set of current-stage initial composite images having the current-stage resolution comprises: The set of current-stage initial synthesized images is initialized by uniform noise distribution.

4. The computer-implemented method of claim 1 , wherein: The steps to obtain a set of real images with the current stage resolution include: Given a set of training images with the desired final resolution: In response to the current stage resolution not being the final resolution: downsampling a training image from the set of training images to the current stage resolution; and Sampling from the downsampled training images to obtain the set of real images having the current stage resolution; and In response to the current stage resolution being the final resolution, sampling is performed from the training images to obtain the set of real images having the current stage resolution.

5. The computer-implemented method of claim 1 , wherein: Given the set of updated current-stage initial synthesized images and the current-stage energy-based model having an input of the current-stage resolution, the step of generating a set of updated current-stage synthesized images via Markov Chain Monte Carlo sampling comprises: Uses short-term Markov chain Monte Carlo sampling with a fixed number of Langevin steps.

6. The computer-implemented method of claim 1 , wherein: The step of modifying the updated current-stage energy-based model to generate an extended energy-based model having a higher resolution than the current-stage resolution comprises: Remove the three-channel red, green and blue RGB image into z RGB block of channel feature map, where z is the number of channels per kernel at the bottom convolutional layer of the updated current-stage energy-based model and having the current-stage resolution; and Added extended chunks and new RGB chunks, each with higher resolution.

7. The computer-implemented method of claim 6, wherein: The extension block includes: An original block, comprising two convolutional layers, where the first convolutional layer does not change the spatial size, and the second convolutional layer has a downsampling stride; and An attenuation block, consisting of a convolutional layer followed by average pooling to connect the previous coarse-level layers.

8. The computer-implemented method of claim 2, further comprising: A composite image at the final resolution is generated using the trained energy-based model at the final resolution.

9. The computer-implemented method of claim 8, wherein: The trained energy-based model is trained for a different domain than the input image, and the steps of using the trained energy-based model at the final resolution to generate a synthetic image at the final resolution include: The trained energy-based model is used for image-to-image translation.

10. A computer-implemented method for training an energy-based model, the method comprising: Starting with the lowest resolution energy-based model, the model resolution capability is continuously improved in stages until the final resolution capability for the energy-based model is reached. Each stage consists of the following steps: The steps include: updating a current-stage energy-based model based on a difference between: (1) an expected value obtained using a set of non-synthesized images at the current-stage resolution and the current-stage energy-based model, and (2) an expected value obtained using a set of synthesized images at the current-stage resolution and the current-stage energy-based model; In response to the current stage energy-based model not having the final resolution capability: adding one or more neural network layers to the updated energy-based model of the current stage to create a higher resolution energy-based model for the next stage; setting the higher resolution energy-based model as the current-stage energy-based model for the next iteration; and Returning to the step of updating the current stage energy-based model for the next iteration; and In response to the current-stage energy-based model having the final resolution capability, outputting the updated current-stage energy-based model as a trained energy-based model with the final resolution.

11. The computer-implemented method of claim 10, wherein: If there is a previous stage energy-based model, the set of synthetic images is obtained using a set of upsampled images generated by using the previous stage energy-based model, where the set of upsampled images has been upsampled from the previous stage resolution to the current stage resolution.

12. The computer-implemented method of claim 11 , further comprising: In response to the first iteration: obtaining the set of non-synthetic images having the current stage resolution by sampling from a set of real images; as well as The set of composite images at the current stage resolution is obtained by performing the following steps, the steps comprising: generating a set of current-stage initial synthetic images having the current-stage resolution by sampling from a distribution; as well as Given the set of initial synthetic images and the current-stage energy-based model having input at the current-stage resolution, the set of synthetic images is generated via Markov Chain Monte Carlo sampling.

13. The computer-implemented method of claim 10 , further comprising: obtaining the set of non-synthetic images having the current stage resolution by sampling from a set of real images; as well as The set of composite images having the current stage resolution is obtained by performing the following steps, the steps comprising: generating a set of current-stage initial synthetic images having the current-stage resolution by sampling from a distribution; Generate a set of initial synthetic images of the previous stage with the resolution of the previous stage by sampling from the distribution; Given the set of previous-stage initial synthesized images and the previous-stage energy-based model having the input of the previous-stage resolution, generating a set of updated previous-stage synthesized images via Markov chain Monte Carlo sampling; generating a set of smoothed composite images having the current stage resolution by combining the set of current stage initial composite images with the set of updated previous stage composite images that have been upsampled to the current stage resolution; and Given the set of smoothed synthetic images and the current-stage energy-based model having input at the current-stage resolution, the set of synthetic images is generated via Markov Chain Monte Carlo sampling.

14. The computer-implemented method of claim 13, wherein: The step of obtaining the set of non-synthetic images having the current stage resolution by sampling from a set of real images comprises: Given a set of training images with the desired final resolution: In response to the current stage resolution not being the final resolution: downsampling a training image from the set of training images to the current stage resolution; and Sampling from the downsampled training images to obtain the set of real images having the current stage resolution; and In response to the current stage resolution being the final resolution, sampling is performed from the set of training images to obtain the set of real images having the current stage resolution.

15. The computer-implemented method of claim 10, wherein: The steps of adding one or more neural network layers to the updated current stage energy-based model to create a higher resolution energy-based model for the next stage include: Remove the three-channel red, green and blue RGB image into z RGB block of channel feature map, where z is the number of channels per kernel in the bottom convolutional layer of the updated current stage energy-based model at the current stage resolution; and Added extended chunks and new RGB chunks, each with higher resolution.

16. The computer-implemented method of claim 10, further comprising: A composite image at the final resolution is generated using the trained energy-based model at the final resolution.

17. A non-transitory computer-readable medium comprising one or more sequences of instructions that, when executed by at least one processor, cause steps for training an energy-based model to be performed, the steps comprising: Starting from a minimum resolution energy-based model, the model resolution capability is continuously improved in stages until a final resolution capability for the energy-based model is reached, each stage comprising the following steps, The steps include: updating a current-stage energy-based model based on a difference between: (1) an expected value obtained using a set of non-synthesized images at the current-stage resolution and the current-stage energy-based model, and (2) an expected value obtained using a set of synthesized images at the current-stage resolution and the current-stage energy-based model; In response to the current stage energy-based model not having the final resolution capability: adding one or more neural network layers to the updated energy-based model of the current stage to create a higher resolution energy-based model for the next stage; setting the higher resolution energy-based model as the current-stage energy-based model for the next iteration; and Returning to the step of updating the current stage energy-based model for the next iteration; and In response to the current-stage energy-based model having the final resolution capability, outputting the updated current-stage energy-based model as a trained energy-based model with the final resolution.

18. The non-transitory computer-readable medium of claim 17, wherein: If there is a previous stage energy-based model, the set of synthetic images is obtained using a set of upsampled images generated by using the previous stage energy-based model, where the set of upsampled images has been upsampled from the previous stage resolution to the current stage resolution.

19. The non-transitory computer-readable medium of claim 18, further comprising one or more sequences of instructions that, when executed by one or more processors, further cause the following steps to be performed, the steps comprising: In response to the first iteration: obtaining the set of non-synthetic images having the current stage resolution by sampling from a set of real images; as well as The set of composite images at the current stage resolution is obtained by performing the following steps, the steps comprising: generating a set of current-stage initial synthetic images having the current-stage resolution by sampling from a distribution; Given the set of initial synthetic images and the current-stage energy-based model having input at the current-stage resolution, the set of synthetic images is generated via Markov Chain Monte Carlo sampling.

20. The non-transitory computer-readable medium of claim 17, further comprising one or more sequences of instructions that, when executed by one or more processors, further cause the following steps to be performed, the steps comprising: obtaining the set of non-synthetic images having the current stage resolution by sampling from a set of real images; as well as The set of composite images having the current stage resolution is obtained by performing the following steps, the steps comprising: generating a set of current-stage initial synthetic images having the current-stage resolution by sampling from a distribution; Generate a set of initial synthetic images of the previous stage with the resolution of the previous stage by sampling from the distribution; Given the set of previous-stage initial synthesized images and the previous-stage energy-based model having the input of the previous-stage resolution, generating a set of updated previous-stage synthesized images via Markov chain Monte Carlo sampling; generating a set of smoothed composite images having the current stage resolution by combining the set of current stage initial composite images with the set of updated previous stage composite images that have been upsampled to the current stage resolution; and Given the set of smoothed synthetic images and the current-stage energy-based model having input at the current-stage resolution, the set of synthetic images is generated via Markov Chain Monte Carlo sampling.

21. The non-transitory computer-readable medium of claim 17, wherein: The steps of adding one or more neural network layers to the updated current stage energy-based model to create a higher resolution energy-based model for the next stage include: Remove the three-channel red, green and blue RGB image into z RGB block of channel feature map, where z is the number of channels per kernel in the bottom convolutional layer of the current stage energy-based model being updated and having the current stage resolution; and Added extended chunks and new RGB chunks, each with higher resolution.

22. The non-transitory computer readable medium of claim 17, further comprising one or more sequences of instructions that, when executed by one or more processors, further cause the following steps to be performed: A composite image at the final resolution is generated using the trained energy-based model at the final resolution.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images