Bridging the gap between diffusion models and uniform quantization for image compression
The quantization-based forward diffusion process aligns quantization errors with diffusion models to address noise and discretization gaps, ensuring realistic and detailed image reconstructions across varying bitrates.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-07-03
- Publication Date
- 2026-04-08
AI Technical Summary
Conventional neural image compression methods optimized for rate distortion often produce blurry and unrealistic images due to mismatches in noise type, noise level, and discretization gaps between quantization errors and diffusion models, leading to visual disturbances and loss of detail.
A quantization-based forward diffusion process that uses universal quantization and a finely tuned uniform noise diffusion model to align quantization errors with diffusion processing, addressing noise type, noise level, and discretization gaps, ensuring realistic and detailed reconstructions across various bitrates.
The system produces highly realistic and detailed image reconstructions even at low bitrates by resolving noise type, noise level, and discretization gaps, improving image quality and fidelity through denoising capabilities.
Smart Images
Figure 2026060879000001_ABST
Abstract
Description
[Technical Field]
[0001] Cross-reference of related applications
[0001] In accordance with Section 119(e) of the United States Patent Act, this application is entitled to and claims the benefit of the filing date of U.S. Provisional Application No. 63 / 700,489, entitled “LINKING OF DIFFUSION MODELS AND UNIFORM QUANTIZATION FOR IMAGE COMPRESSION,” filed on 27 September 2024, the contents of which said Provisional Application are incorporated herein by reference in their entirety for all purposes. [Background technology]
[0002]
[0002] Multimedia content is distributed worldwide via networks and accounts for the majority of traffic. Developing efficient compression algorithms is important for efficiently distributing multimedia content across the entire network.
[0003]
[0003] Conventional codecs (CODECS) that use user-created conversions may not perform as well as data-driven neural image compression (NIC) methods that optimize for both rate and distortion. Nevertheless, neural image compression methods can still produce blurry and unrealistic images, such as at low bitrate settings. This is because the method may be optimized for rate distortion, in which case the distortion is measured in pixel-level metrics such as mean squared error. Optimization for low distortion, such as pixel-level error, can result in unrealistic images. This may be because emphasizing pixel-level accuracy or similarity to the original image can result in an overly smoothed or blurred output. [Overview of the Initiative]
[0004]
[0004] The included drawings are for illustrative purposes only and serve only to provide examples of possible configurations and operations of the disclosed systems, apparatus, methods, and computer program products of the present invention. These drawings do not in any way limit any modifications of shape and detail that can be made by those skilled in the art without departing from the spirit and scope of the disclosed embodiments. [Brief explanation of the drawing]
[0005] [Figure 1]
[0005] A diagram showing a simplified system for performing compression according to several embodiments. [Figure 2]
[0006] A diagram showing a simplified flowchart of a method for determining a quantization schedule according to several embodiments. [Figure 3]
[0007] A diagram showing a simplified flowchart of a method for carrying out a diffusion treatment according to several embodiments. [Figure 4]
[0008] A diagram showing a simplified flowchart of a method for performing a training process according to several embodiments. [Figure 5]
[0009] A figure showing an example of a computing device according to several embodiments. [Modes for carrying out the invention]
[0006]
[0010] This specification describes techniques for content processing systems. The following description includes numerous examples and specific details for illustrative purposes to provide a thorough understanding of several embodiments. Some embodiments defined by the claims may include some or all of the features in these examples, either individually or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.
[0007]
[0011] System Overview
[0012] Generative neural image compression supports data compression at extremely low bitrates, enabling receivers (e.g., client devices) to synthesize detail and consistently produce highly realistic images. By leveraging the similarity between quantization error and additional noise, spread-based generative image compression codecs use latent spread models to denoise artifacts introduced by quantization. Image compression pipelines can use spread models that can synthesize detail lost during the compression process. For example, spread models may be used to compensate for quantization errors that may occur when using quantization. Errors introduced during quantization can be analogous to adding uniform noise. In fact, adding uniform noise is often used during training of neural codecs as a differentiable substitute for quantization operations. Since spread models are essentially denoising models, they can further be used to offset quantization errors that occur during encoding. By leveraging the similarity between quantization error and noise, spread models can perform a subset of denoising steps corresponding to the noise level (e.g., quantization error) of the quantized latent representation. The output of the resulting diffusion model can be corrected for quantization errors caused by the quantization process. This allows for improvement of the resulting decoded image, particularly at low bitrates, resulting in a more realistic image, although improvement can be achieved at all bitrates.
[0008]
[0013] Previous methods following this paradigm may have three gaps (i.e., noise type gap, discretization gap, and noise level gap) where the quantized data deviates from the data distribution known by the diffusion model. When this occurs, the diffusion model cannot optimally denoise the artifacts introduced by quantization. However, this system uses a quantization-based forward diffusion process that overcomes all three of the aforementioned gaps.
[0009]
[0014] This system addresses three gaps: the noise type gap, the noise level gap, and the discretization gap. The noise type gap refers to the difference in distribution between quantization errors (e.g., uniform noise) and the Gaussian diffusion model. The noise level gap refers to the possible discrepancy between the predicted signal-to-noise ratio and the actual ratio for partially noisy data. The discretization gap arises from passing discrete data to a continuous diffusion model. Leaving these gaps unresolved can cause the data to deviate from the diffusion model's distribution, negatively impacting the final reconstruction quality. This system incorporates quantization-based forward diffusion processing to fill the discretization and noise level gaps, and uses a uniform noise diffusion model to fill the noise type gap. This system consistently produces realistic and detailed reconstructions, even at very low bitrates.
[0010]
[0015] In some embodiments, the system uses a quantization-based forward diffusion process that places quantized data along a diffusion trajectory. The forward process employs universal quantization to fill discretization gaps and introduces a quantization schedule that dictates the signal-to-noise ratio of the quantized data. Finally, the system matches the distribution of quantization errors to eliminate noise type gaps using a diffusion model trained with uniform noise. In some embodiments, the uniform noise diffusion model can also be efficiently obtained by fine-tuning an existing Gaussian diffusion model. This system provides an image codec that can operate over a wider range of target bitrates while producing more realistic and detailed reconstructions than conventional methods.
[0011]
[0016] This system includes a pipeline that combines non-integer universal quantization with a finely tuned uniform noise diffusion model. The system employs universal quantization while selecting quantization bin widths to ensure that the signal-to-noise ratio (SNR) of the quantized variables matches the predicted signal-to-noise ratio at each time step, thereby eliminating the discretization gap and mitigating the discretization gap between the difference between the predicted Gaussian noise and the actual uniform noise in the quantization error.
[0012]
[0017] Assuming the noise-like properties of quantization, the system develops a codec that leverages the denoising capabilities of the diffusion model, which uses these models to explicitly remove errors that occur during quantization. The system uses a link between quantization errors and diffusion processing. Assuming the iterative nature of the diffusion model, instead of sampling new images from a generative model, the system takes an existing image and can obtain partially noisy samples in several arbitrary iterations, which can then be reconstructed into the original image by performing a subset of diffusion steps. The system uses universal quantization with a selected "quantization schedule" so that the quantized variables follow a diffusion trajectory, thereby enabling detailed and realistic reconstruction through denoising diffusion processing.
[0013]
[0018] This system can resolve the distribution mismatch between Gaussian noise and uniform noise. Instead of warping the uniform noise to a normal distribution, the system can instead replace the Gaussian diffusion model with one that operates on the uniform noise to achieve the same effect. When the uniform distribution satisfies this requirement, a diffusion model that denoises the uniform noise can be used.
[0014]
[0019] system
[0020] Figure 1 shows a simplified system 100 for performing compression according to some embodiments. System 100 includes a server system 102 and a receiver 104. The server system 102 can encode content, and the receiver 104 can decode content. In some embodiments, the server system 102 can transmit the encoded content to the receiver 104 via a network. However, encoding and decoding may also be performed on a single system. In some embodiments, the server system 102 may be encoding video, which is transmitted via a network to a client device such as the receiver 104. The client device can decode the video and display the video using a media player of the interface. The client device may in this case be a smartphone, a living room device, a television, a personal computer, a laptop, a tablet device, etc. Other system configurations may also be considered.
[0015]
[0021] The diffusion model may be a class of generative models that define an iterative process (e.g., a forward diffusion process) that gradually destroys an input signal by adding noise as the time step t increases, and then attempts to model the reverse process by removing noise from the noisy image (e.g., a reverse diffusion process). Experimentally, the forward process is performed by adding noise, such as uniform noise, to the signal. Therefore, the reverse process becomes a noise removal process that removes noise from the input. The diffusion model approximates the reverse process by estimating the noise level of the image and using it to predict the previous step of the forward process. Thereby, a certain amount of noise can be removed from the image. This removal can be performed in several time steps. To completely remove the noise from the image, the diffusion model can iteratively perform the entire set of time steps in the reverse process.
[0016]
[0022] In latent diffusion models, improved memory and computational efficiency can be achieved by moving the diffusion process to a spatially lower-dimensional latent space compared to image space (e.g., pixel space). The latent space can provide performance similar to the corresponding image-space diffusion model while requiring fewer parameters and less memory. Here, the latent diffusion model may be trained in the latent space, in which case the encoder can encode the image into a latent representation in the latent space. Next, this latent representation is processed by the latent diffusion model to denoise the latent representation over a time step t. The denoised latent representation can then be decoded into a decoded image in image space.
[0017]
[0023] This system resolves the noise type gap. Quantization errors in many regions (such as the latent region) can be approximated by uniform noise. However, some diffusion models assume a Gaussian noise structure because it aligns with the natural data distribution assumption and facilitates more manageable modeling. As a result, a noise type gap arises, i.e., a mismatch between the quantization error (which is adequately approximated by uniform noise) and the Gaussian noise used for diffusion processing. This mismatch means that when uniform quantization noise interacts with a Gaussian denoising diffusion model, the model fails to correctly predict the actual noise characteristics for denoising, resulting in generated artifacts. In particular, the mismatch can lead to visual disturbances such as unnatural color shifts, texture inconsistencies, and artificial patterns, which degrade the realism and fidelity of the generated image.
[0018]
[0024] Furthermore, this system also solves the discretization gap. Despite being a continuous model, the neural decoder operates on discrete representations extracted from the transmitted bitstream, and most methods construct a robust decoder that minimizes the resulting adverse effects. However, constructing a similar robust diffusion model in this situation may not be possible because these diffusion models model transitions between continuous states and cannot inherently handle discrete inputs. As a result, a discretization gap occurs, i.e., an incompatibility between using discrete input data with a continuous diffusion model. Under the discretization gap, small changes in the input data are removed, resulting in a flat texture and loss of detail. Also, using a large quantization bin size causes blocking artifacts and color shifts due to the low resolution of the color palette.
[0019]
[0025] Finally, this system solves the noise level gap. In diffusion image generation, a fixed progression is assumed through a variance schedule that indicates the noise level at each time step t. Therefore, it is important to ensure a noise level match between the forward diffusion process and the reverse diffusion process (e.g., the noise at all corresponding time steps t should be the same in the forward and reverse processes; otherwise, it goes against the theoretical basis of the diffusion model). However, when different forward processes are used (e.g., when quantization is used instead of the forward process), the noise levels in the forward and reverse processes may not match. This is the noise level gap, i.e., the difference between the actual noise level of the diffusion variable and what is predicted at any time step. Intuitively, the diffusion model may overestimate or underestimate the noise of the variable at the time step throughout the diffusion process, resulting in either noisy image restoration or overly smoothed image restoration. [[ID=⑥]] [[ID=⑦]]
[0020] [[ID=⑧]] [[ID=⑨]]
[0026] The following pipeline addresses the three gaps mentioned above: the noise type gap, the noise level gap, and the discretization gap. The pipeline allows the server system 102 to receive an image x. For example, image x may be an image from a video being encoded. The following processing may be performed for each image in the video. The encoder 106 can encode the image into a latent representation y in a latent space. The latent space may be a lower-dimensional space compared to the image space. That is, the latent space can represent a compressed version of the input that captures key features. In some embodiments, the encoder 106 may be a variational autoencoder (VAE), which may be a neural network or machine learning model trained to represent the image in the latent space. The encoder 106 may be considered part of the diffusion model 116, or it may be separate. In some embodiments, the latent representation y may be a latent vector that captures key features of the input image in the latent space. The latent representation y may be mapped from the input image to a distribution in the latent space that can be parameterized by mean and variance. Although a variational autoencoder has been described, other encodings that can map the input image to the latent space may also be used.
[0021]
[0027] In quantization process 108, a quantization schedule based on time step t is used to quantize the latent representation y into a latent representation
[0022]
number
[0023] It can be quantized. This quantization process reduces the precision of an image by representing it using a finite number of discrete values. Quantization can convert a continuous-value signal into a digital signal with a limited range of values. This conversion is done by mapping the continuous signal to a set of discrete values called quantization levels or quantization bins. Quantization can be an affine transformation T to a latent representation y before integer quantization is applied. An affine transformation can be a linear mapping used to convert floating-point values to fixed-point representations such as integers. Channels can be latent coding or image channels such as color and brightness.
[0024]
[0028] Quantized latent representation
[0025]
number
[0026] The entropy model can be entropy coded by entropy coding 110.
[0027]
number
[0028] It can be used to encode into a bitstream containing the quantized latent representation. Different entropy models can be used to entropically encode a quantized latent representation into a bitstream. Entropy coding can reduce the average number of bits required to represent a quantized latent representation using entropy coding methods, including Huffman coding and arithmetic coding. Entropy coding can reduce the average length of a quantized latent representation by assigning shorter codes to more frequent symbols and longer codes to less frequent symbols.
[0029]
[0029] The server system 102 can transmit the bitstream to the receiver 104. Entropy decoding 112 is performed on the quantized latent representation
[0030]
number
[0031] To reconstruct the bitstream, it can be entropy-decoded. Entropy decoding quantizes the bitstream into a latent representation.
[0032]
number
[0033] This is the reverse process of entropy coding to restore the original data.
[0034]
[0030] The renoising process 114 restores the latent expression
[0035]
number
[0036] To generate, a portion of the quantization can be performed using a quantization schedule based on a time step t. In the renoising process, discrete representations are used as input.
[0037]
number
[0038] The latent representation is taken and output in a continuous domain.
[0039]
number
[0040] This may have quantization errors due to information loss. These quantization errors can resemble noise. For example, quantization errors can cause random changes in the latent representation during the process of converting continuous (high-precision) data, such as floating-point numbers, to discrete (low-precision) values, such as integers. Accordingly, the quantization process is divided into two stages: one is performed by the server system 102 (transmitter), and the other is performed by the receiver 104.
[0041]
[0031] The reconstructed latent representation is input to the diffusion model 116 to perform inverse quantization. The diffusion model 116 can denoise the reconstructed latent representation to remove noise, and in this denoising, the denoised latent representation
[0042]
number
[0043] To obtain the output, the resulting quantization errors may be removed. That is, quantization errors can be analogous to adding noise to an image, and the diffusion model 116 may be used to denoise the reconstructed latent representation. This process is described in more detail below.
[0044]
[0032] The decoder 118 is a denoised latent representation
[0045]
number
[0046] decode image
[0047]
number
[0048] It can be decoded into an image. In some embodiments, the decoder 118 can be a variational autodecoder, but other decoders may also be used. The decoder 118 denoises the latent representation from the latent space and decodes the image.
[0049]
number
[0050] The image can be reconstructed in image space. The decoded image may be improved in that at least some of the quantization errors that occurred may have been removed by the diffusion model 116. This removal of errors may result in a more realistic reconstructed image.
[0051]
[0033] Diffusion treatment
[0034] The diffusion model defines a process that models the transition between random noise and structured data. When the forward process (e.g., data to noise) and the reverse process (e.g., noise to data) are divided into small steps, the transition between each step is the addition or removal of Gaussian noise samples. Thus, the complete diffusion process is a traversal between a series of time steps t ∈ [N,0]. This process is iterative, but the partially noisy diffusion variable y at any given time step t is... t This can be expressed in terms of the original data y0 and the noise sample ε. That is,
[0052]
number
[0053] Here α t (known as a "distributed schedule") is y every time step t. t The signal-to-noise ratio is defined and increases as t→0. The noise sample ε is the forward path of the diffusion model. The dispersion schedule is defined by the diffusion variable y at each time step t. tControls the trade-off between the signal component and the noise component. Reverse diffusion processing is extremely difficult, and therefore, it is parameterized by a diffusion model, and this diffusion model iteratively denoises y t by learning to execute one step at a time through t = {N,..., 1, 0}. The partially denoised data y t-1 can be calculated from the noisy data y t by the following equation.
[0054]
Equation
[0055] Here, ε θ [[]](y t , t) is the output of the diffusion model that takes y t and the current time step t as inputs, and
[0056]
Equation
[0057] is an estimate of the completely denoised data calculated from the output of the diffusion model and the current time step. The completely denoised data is generated by continuously applying Equation (2) to generate data with less noise until the completely denoised data is output.
[0058]
[0035] To use the diffusion model as an image compression codec, the forward noising process can be replaced with quantization (similar to adding uniform noise to the original signal) and the diffusion model used to remove the quantization error. As discussed above, three obstacles can result: the noise type gap, the noise level gap, and the discretization gap.
[0059]
[0036] System 100 performs improved forward processing. System 100 uses quantization for forward processing, replacing the spread model forward processing. Forward processing using quantization allows discrete variables to be encoded into a bitstream while maintaining the noise characteristics of the original spread-dispersive schedule. Standard forward processing (a slight reconstruction of Equation 1) is given by:
[0060]
number
[0061]
[0037] To address the discretization gap, the quantization process 108 may use universal quantization as a forward process that adds noise to obtain discrete variables for entropy coding. Universal quantization can be hard quantization dithered by a uniform random variable. This has the inherent property that its distribution is equivalent to simply adding another sample (from the same random variable) to the original unquantized variable.
[0062]
number
[0063] Here,
[0064]
number
[0065] This represents rounding to a bin of width Δ. A bin of width Δ can be the quantization step size or interval, referring to a range of values that map to a single quantization value. Hard quantization refers to the process of rounding a continuous signal to the nearest quantization level.
[0066]
number
[0067] This represents a hard quantization operation where the signal is rounded to bins of width Δ, and then dithered by a uniform random variable u to randomize the quantization error. The uniform random variable is a uniform noise sample u,u' distributed uniformly between -Δ / 2 and Δ / 2. The values of u,u' can take any value within this range with equal probability. The quantization process 108 and the renoising process 114 introduce errors to the input signal y, but preserve the statistical characteristics of the signal degradation predicted by the diffusion process.
[0068]
[0038] Combining equations (3) and (4), the forward noising process performed by the quantization process 108 and the renoising process 114 is given by the following equation.
[0069]
number
[0070]
number
[0071]
number
[0072] Compared to hard quantization, which passes discrete data directly to the decoder, the reconstructed latent representation
[0073]
number
[0074] is again a continuous variable, and the uniform noise sample u in equation (7) is quantized into a latent representation
[0075]
number
[0076] In addition, the restored latent expression
[0077]
number
[0078] It returns to continuous space. For example, after rounding in quantization process 108 (equation (6)), renoising process 114 restores the latent representation
[0079]
number
[0080] The output is in continuous space (equation (7)). Equation (6) for quantization is performed in the server system 102 by the quantization process 108, and equation (7) is performed in the receiver 104 by the renoising process 114.
[0081]
[0039] To address the noise level gap, system 100 calculates the latent expression y for every time step t. t and the restored latent expression
[0082]
number
[0083] Ensure the signal-to-noise ratio. Restored latent representation
[0084]
number
[0085] The signal-to-noise ratio can be controlled by adjusting the quantization bin width defined with respect to Δ and the uniform noise support. That is, to fill the noise level gap, system 100 uses latent representation y t and the restored latent expression
[0086]
number
[0087] The noise levels of the following will be matched as follows:
[0088]
number
[0089]
[0040] The quantization process 108 uses a quantization schedule that varies the quantization bin width as a function of the time step t. The uniform variable support is an interval or range of values in which the random variable u is uniformly distributed. This means that all values within this range have an equal probability of being selected. The additional noise u is uniformly distributed between -Δ / 2 and Δ / 2. In some embodiments, the quantization process 108 uses the width in equations 5 and 6.
[0090]
number
[0091] The bins are set. The bins of width Δ can be determined by substituting equations 1 and 5 into equation 7 and solving for the bins of width Δ. As the quantization schedule is followed and the bin size is varied, the reconstructed latent representation is obtained.
[0092]
number
[0093] The diffusion variable and latent representation y t The signal-to-noise ratio gap is eliminated by maintaining a consistent signal-to-noise ratio with respect to the quantized variables.
[0094]
[0041] In forward processing based on quantization, the discretization gap and noise level gap are simultaneously removed via universal quantization and quantization schedule, respectively. A further advantage of the quantization schedule is that it also becomes a rate-distortion trade-off parameter, as the quantization bin width directly affects the final size of the compressed bitstream. In addition, since the diffusion model can denoise the data at any time step, system 100 supports compression to multiple bit rates with a single model by accepting time step t as input during inference and performing compression.
[0095]
[0042] In some embodiments, the system 100 can train a uniform diffusion model by starting with a pre-trained Gaussian diffusion model. In some embodiments, the system 100 fine-tunes the underlying diffusion model and replaces the Gaussian noise with uniform noise. Since the diffusion model is susceptible to changes in the dispersion schedule, the system 100 may leave the dispersion schedule unchanged despite the changes in the distribution. To adapt the Gaussian diffusion model to the uniform noise, this is done by, during training, replacing the noise of the forward processing with equation (1)
[0096]
number
[0097] This is done by guiding the distribution to a uniform distribution between the values.
[0098]
[0043] Quantization schedule setting
[0044] Figure 2 shows a simplified flowchart 200 of a method for determining the quantization schedule according to several embodiments. In 202, the system 100 receives a time step t. The time step may be received from user input, automatically generated, or dynamically determined based on the input image. The value of the time step t becomes a rate-distortion trade-off parameter, as the quantization bin width directly affects the final size of the compressed bitstream. The rate-distortion trade-off can improve the balance between compression ratio and distortion. Compression ratio is the number of bits used to encode the data, and distortion is the difference between the restored image and the original image. Input settings can trade off high bitrates for low distortion or low bitrates for high distortion. When quantization results in a higher compression ratio, a higher bitrate is used, which can lead to a more accurate restoration and less distortion. When fewer bits are used for quantization, a lower bitrate is used, which can lead to a less accurate restoration and greater distortion. Reducing the number of bits achieves a lower rate, but can result in larger quantization errors and greater distortion in the restored image. However, in this pipeline, the diffusion model 116 can be used to offset higher quantization errors and greater distortions. The time step t can be the optimal number of denoising steps that the diffusion model 116 should perform to generate a realistic image. The quantization process 108 may introduce a certain amount of quantization error (e.g., noise). There may be several time steps to remove this certain amount of noise. When this number of time steps are performed by the diffusion model 116, a realistic image is obtained.
[0099]
[0045] In 204, the system 100 determines a quantization schedule that varies the quantization bin width as a function of the time step t. As described above, this bin width is
[0100]
number
[0101] It can be done this way.
[0102]
[0046] In step 206, the system 100 sets the quantization schedule for the quantization process 108 and the renoising process 114. In some embodiments, the server system 102 can transmit the quantization schedule to the receiver 104 over the network.
[0103]
[0047] In step 208, the system 100 transmits a time step t to the receiver 104. The spreading model 116 can use the time step t to perform several iterations of the denoising step on the reconstructed latent representation based on the value of the time step t. For example, if a value of 50 is received, the spreading model 116 can perform 50 iterations of denoising on the reconstructed latent representation.
[0104]
[0048] Diffusion Model
[0049] Figure 3 shows a simplified flowchart 300 of a method for carrying out the diffusion process according to several embodiments. In 302, the diffusion model 116 is shown with a time step t and the recovered latent representation
[0105]
number
[0106] The input is received. The time step t can be specified by the input.
[0107]
[0050] In 304, the diffusion model 116 is the recovered latent representation
[0108]
number
[0109] A noise reduction operation is performed on the input. As discussed above, the diffusion model 116 uses the reconstructed latent representation as input.
[0110]
number
[0111] The time step t can be received. Next, the diffusion model 116 recovers the latent representation
[0112]
number
[0113] The noise level is estimated and the latent representation is reconstructed.
[0114]
number
[0115] To remove some noise from it, the previous step of forward processing can be predicted. In 306, the diffusion model 116 is denoised and restored latent representation
[0116]
number
[0117] Outputs.
[0118]
[0051] In 308, it is determined whether time step t is satisfied. For example, if there are 50 time steps, the diffusion model 116 can compare 50 time steps with the current number of time steps. If time step t is not satisfied, the process returns to 304, in which case another denoising process is performed on the output of the diffusion model 116. Here, the denoised restored latent representation
[0119]
number
[0120] The model is denoised again using the same process described above. This process continues until time step t is satisfied. When time step t is satisfied, at 310, the diffusion model 116 is the final denoised restored latent representation.
[0121]
number
[0122] The output is as follows: Here, the diffusion model 116 may remove some of the noise that is generated as quantization error during the quantization process.
[0123]
[0052] Training
[0053] In some embodiments, the diffusion model 116 may be used as the base model. The system may use stable diffusion as the base model, but other models may be used. A base model (e.g., a diffusion model) trained on a large amount of data, etc., therefore has good generative power for denoising the latent representation of an image. In some embodiments, the system 100 fine-tunes a pre-trained Gaussian diffusion model to work with uniform noise.
[0124]
[0054] Figure 4 shows a simplified flowchart 400 of a method for carrying out the training process according to several embodiments. In 402, a training dataset of images may be input to the pipeline. The training dataset of images may be ground truth. In 404, the pipeline outputs a decoded image. The pipeline may process the image as described above. In 406, the decoded image may be compared to the ground truth of the original image to determine the difference between the decoded image and the original image. In 408, based on that difference, the parameters of the denoising model 116 may be adjusted to minimize the difference using a loss function, etc. For example, the noise ε is sampled from a uniform distribution with unit variance rather than a standard normal Gaussian distribution (commonly used in diffusion models). Such noise is applied to a source tensor (e.g., a latent image or image). The denoising model is then trained to predict the noise applied to the source tensor, assuming the noise is present.
[0125]
[0055] In the second stage, the processes 402-406 may be performed again. Next, in 410, in the second stage, the parameters of the encoder 106, quantization process 108, renoising process 114, diffusion model 116, and decoder 118 may be frozen. In 412, the system 100 quantized latent representation
[0126]
number
[0127] To efficiently encode and re-encode the data into a bitstream, entropy coding 110 and entropy decoding 112 are trained. In particular, since all image transformation modules are frozen and the entropy coding stage is reversible, system 100 can be optimized only for rate purposes. In addition, system 100 samples time step t during training. That is, the function of the entropy model is an accurate probabilistic model of the quantized data and the recovered quantized latent representation z ^Since the distribution of depends on the input parameter t, system 100 changes it to reflect the operating conditions during inference. The range of t can be changed.
[0128]
[0056] Conclusion
[0057] Thus, a lossy image compression codec based on a latent spread model can be provided to generate realistic image reconstructions from low bitrates to very low bitrates. By combining the denoising capability of the spread model with the intrinsic characteristics of quantization noise, the system generates perceptually pleasing reconstructions over a range of bitrates. Lower bitrates can be achieved by using fewer bits to tolerate quantization errors in the quantization process. Errors can be corrected by denoising using the spread model 116. The system 100 minimizes the noise level gap, the noise type gap, and the discretization gap. Furthermore, the spread model can be trained to denoise uniform noise.
[0129]
[0058] System
[0059] Figure 5 shows an example of a computing device according to several embodiments. Depending on the various embodiments, a system 500 suitable for implementing the embodiments described herein may include a processor 501, memory 503, storage device 505, interface 511, and bus 515 (e.g., PCI bus or other interconnect fabric). The system 500 can operate as various devices or as any other device or service described herein. Although specific configurations are described, various alternative configurations are possible. The processor 501 can perform operations such as those described herein. Instructions for performing such operations may be embodied in memory 503, one or more non-temporary computer-readable media, or any other storage device. Various specially configured devices can also be used instead of or in addition to the processor 501. Memory 503 may be random access memory (RAM) or other dynamic storage devices. The storage device 505 may include a non-temporary computer-readable storage medium that holds information, instructions, or any combination thereof (for example, instructions that, when executed by the processor 501, set or enable the processor 501 to perform one or more operations of the methods described herein). Bus 515 or other communication components may support the transmission of information within the system 500. Interface 511 may be connected to bus 515 and configured to send and receive data packets over a network. Examples of supported interfaces include, but are not limited to, Ethernet®, High Speed Ethernet, Gigabit Ethernet, Frame Relay, Cable, Digital Subscriber Line (DSL), Token Ring, Asynchronous Transfer Mode (ATM), High Speed Serial Interface (HSSI), and Fiber Distributed Data Interface (FDDI). These interfaces may include ports suitable for communication with appropriate media. They may also include a separate processor and / or volatile RAM.A computer system or computing device may include, or communicate with, a monitor, printer, or other suitable display for providing a user with any of the results referred to herein.
[0130]
[0060] Any of the disclosed embodiments can be embodied in various types of hardware, software, firmware, computer-readable media, and combinations thereof. For example, some of the techniques disclosed herein can be at least partially implemented by non-temporary computer-readable media, including program instructions, state information, etc., for configuring a computing system to perform the various services and operations described herein. Examples of program instructions include both machine code, such as that generated by a compiler, and higher-level code, which can be executed via an interpreter. Instructions can be embodied in any suitable language, such as Java®, Python, C++, C, HTML, or any other markup language, JavaScript®, ActiveX, VBScript, or Perl. Examples of non-temporary computer-readable media include, but are not limited to, magnetic media such as hard disks and magnetic tapes, optical media such as flash memory, compact discs (CDs) or digital versatile discs (DVDs), magneto-optical media, and other hardware devices such as read-only memory ("ROM") devices and random access memory ("RAM") devices. Non-temporary computer-readable media can be any combination of such storage devices.
[0131]
[0061] In the foregoing specification, various techniques and mechanisms are sometimes described in the singular for clarity. However, it should be noted that some embodiments involve multiple iterations of a technique or multiple instantiations of a mechanism, unless otherwise specified. For example, a system uses a processor in various situations, and unless otherwise specified, multiple processors may be used while remaining within the scope of this disclosure. Similarly, various techniques and mechanisms are sometimes described as involving a connection between two entities. However, since various other entities (e.g., bridges, controllers, gateways, etc.) may exist between the two entities, the connection does not necessarily mean a direct, unhindered connection.
[0132]
[0062] Some embodiments may be implemented on a non-temporary computer-readable storage medium for use in or in connection with an instruction execution system, device, system, or machine. This computer-readable storage medium contains instructions for controlling a computer system to carry out the methods described in some embodiments. This computer system may include one or more computing devices. The instructions may be configured or operable to carry out what is described in some embodiments when executed by one or more computer processors.
[0133]
[0063] As used herein and throughout the claims thereafter, “a,” “an,” and “the” include the plural form unless otherwise specified by the context. Also, as used herein and throughout the claims thereafter, the meaning of “in” includes both “in” and “on” unless otherwise specified by the context.
[0134]
[0064] The above description illustrates various embodiments, along with examples of how aspects of several embodiments may be carried out. The above examples and embodiments should not be considered as the only embodiments, but are presented to illustrate the applicability and advantages of several embodiments as defined by the appended claims. Based on the above disclosure and the appended claims, other configurations, embodiments, and equivalents may be used without departing from the scope of this disclosure as defined by the claims.
Claims
1. Receiving images and, The aforementioned image is encoded into a latent representation in the latent space, To generate a quantized latent representation, a quantization process is performed on the latent representation, A method comprising transmitting the quantized latent representation to a receiver, wherein a renoising process is performed to add uniform noise to generate a reconstructed latent representation, a diffusion model performs several iterations of denoising based on a time step t to remove noise from the reconstructed latent representation to generate a denoised reconstructed latent representation, and wherein the diffusion model is trained to perform denoising using the uniform noise.
2. The method according to claim 1, further comprising performing entropy coding on the quantized latent representation, wherein the receiver entropy decodes the entropy coded quantized latent representation.
3. Performing quantization on the aforementioned latent representation The method according to claim 1, comprising selecting a quantization schedule that varies the quantization bin width as a function of the time step t used by the diffusion model to denoise the recovered latent representation.
4. Performing quantization on the aforementioned latent representation The method according to claim 3, comprising selecting bin widths to match the signal-to-noise ratio at time step t, which is used by the diffusion model to denoise the recovered latent representation.
5. The aforementioned bin width is, [Math 1] Equivalent to, here, α t The method according to claim 4, wherein is a distributed schedule that defines the signal-to-noise ratio at time step t for the noise reduction process.
6. Receiving the aforementioned time step t, The time step t is used to determine the quantization schedule. The method according to claim 3, further comprising:
7. Performing quantization on the aforementioned latent representation Dithering the aforementioned latent expression with a uniformly distributed random variable. The method according to claim 1, comprising:
8. The method according to claim 1, wherein the noise reduction process is carried out using a distributed schedule that defines a signal-to-noise ratio at time step t and increases as the time step approaches zero.
9. The method according to claim 1, wherein the renoising process is performed on the latent representation, and uniform noise is added to the quantized latent representation by the renoising process.
10. The method according to claim 9, wherein the restored latent representation after the renoising process is a continuous variable.
11. The diffusion model is trained to remove noise of the first type, The method according to claim 1, wherein the diffusion model is adjusted to remove the uniform noise.
12. The method according to claim 1, wherein the diffusion model recovers information lost in the quantization of the latent representation.
13. The quantization error resulting from generating the quantized latent representation is added to the reconstructed latent representation by the addition of the quantization noise. The method according to claim 1, wherein the diffusion model denoises the reconstructed latent representation in order to remove the quantization noise from the reconstructed latent representation.
14. A non-temporary computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a computing device, the computing device... To receive images, The aforementioned image is encoded into a latent representation in the latent space, To generate a quantized latent representation, a quantization process is performed on the latent representation, and A non-temporary computer-readable storage medium that is operable to transmit the quantized latent representation to a receiver, wherein a renoising process is performed to add uniform noise to generate a reconstructed latent representation, and a diffusion model performs several iterations of denoising based on a time step t to remove noise from the reconstructed latent representation to generate a denoised reconstructed latent representation, and wherein the diffusion model is trained to perform denoising using the uniform noise.
15. The quantized latent representation of an image is received in a latent space, and therein, the image is encoded into a latent representation in the latent space and quantized in order to generate the quantized latent representation. This involves performing a renoising process that adds uniform noise to generate the restored latent representation, Using a diffusion model, denoising is performed in several time steps based on time step t to remove noise from the reconstructed latent representation in order to generate a denoised reconstructed latent representation, wherein the diffusion model is trained to perform denoising using the uniform noise. Decode the denoised restored latent representation into a restored image. A method that includes [a certain feature].
16. The quantized latent representation is entropy coded, The method according to claim 15, wherein the quantized latent representation is entropy-decoded before the renoising process is performed.
17. The aforementioned quantized latent representation is To denoise the quantized latent representation, select a quantization schedule that changes the bin width as a function of the time step t used by the diffusion model. The method according to claim 15, which is produced by
18. The method according to claim 15, wherein the noise reduction process is carried out using a distributed schedule that defines a signal-to-noise ratio at time step t and increases as the time step approaches zero.
19. The method according to claim 18, wherein the bin width is selected to match the signal-to-noise ratio at the time step t used by the diffusion model to denoise the quantized latent representation.
20. The diffusion model is trained to remove noise of the first type, The method according to claim 15, wherein the diffusion model is adjusted to remove the uniform noise.