Diffusion-Based Data Compression

Abstract

Systems and techniques are described for processing image data using a residual model that can be configured with an adjustable number of sampling steps. For example, the process can include obtaining a latent representation of the image and processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image. The process can further include processing the initial reconstructed image and noise data using the residual model to predict multiple predictions of a residual over several sampling steps. The residual represents a difference between the image and the initial reconstructed image. The process can include obtaining a final residual that represents a difference between the image and the initial reconstructed image from the multiple predictions of the residual. The process can further include combining the initial reconstructed image and the residual to generate the final reconstructed image.

Description

[Technical Field]

[0001] For example, aspects of the present disclosure include using machine learning systems to compress image and / or video content. [Background technology]

[0002] Many devices and systems allow data, such as media data (e.g., image data, video data, audio data, etc.), to be processed and output for consumption. Media data includes large amounts of data to meet increasing demands for image / video / audio quality, performance, and functionality. For example, consumers of video data typically desire high-quality video with high fidelity, resolution, frame rate, etc. The large amounts of video data frequently required to meet these demands place significant strain on communication networks and devices that process and store the video data. Video coding techniques may be used to compress video data. One exemplary goal of video coding is to compress video data into a form that uses a low bitrate while avoiding or minimizing degradation of video quality. As ever-evolving video services become available and the demand for large amounts of video data increases, coding techniques with better performance and efficiency are needed. Summary of the Invention

[0003] In some aspects, systems and techniques for data compression and / or decompression using one or more machine learning systems are described. In some cases, machine learning systems (e.g., using one or more neural network systems) are provided that include residual models (e.g., diffusion models such as denoising diffusion probability models), recurrent neural networks (RNNs), or other types of machine learning models) to perform data compression using an adjustable number of sampling steps.

[0004] According to at least one illustrative example, a method for processing image data is provided, the method including obtaining a latent representation of an image, processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image, processing the initial reconstructed image and noise data using a residual model to predict, over a number of sampling steps, a plurality of predictions of a residual representing a difference between the image and the initial reconstructed image, obtaining, from the plurality of predictions of the residual, a final residual representing a difference between the image and the initial reconstructed image, and combining the initial reconstructed image and the residual to generate the final reconstructed image.

[0005] In another example, an apparatus for processing media data is provided, the apparatus including at least one memory and at least one processor communicatively coupled to the at least one memory (e.g., configured in a circuit), wherein the at least one processor is configured to obtain a latent representation of the image, process the latent representation of the image using a decoder of the machine learning model to generate an initial reconstructed image, process the initial reconstructed image and noise data using the residual model to predict, over a number of sampling steps, a plurality of predictions of a residual representing a difference between the image and the initial reconstructed image, obtain from the plurality of predictions of the residual a final residual representing a difference between the image and the initial reconstructed image, and combine the initial reconstructed image and the residual to generate the final reconstructed image.

[0006] In another example, a non-transitory computer-readable medium having stored thereon at least one instruction that, when executed by one or more processors, causes the one or more processors to obtain a latent representation of an image; process the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; process the initial reconstructed image and noise data using a residual model to predict, over a number of sampling steps, multiple predictions of residuals representing differences between the image and the initial reconstructed image; obtain, from the multiple predictions of the residuals, a final residual representing differences between the image and the initial reconstructed image; and combine the initial reconstructed image and the residual to generate the final reconstructed image.

[0007] In another example, an apparatus for processing image data is provided, the apparatus may include means for obtaining a latent representation of the image, means for processing the latent representation of the image using a decoder of the machine learning model to generate an initial reconstructed image, means for processing the initial reconstructed image and noise data using the residual model to predict multiple predictions of a residual representing a difference between the image and the initial reconstructed image over a number of sampling steps, means for obtaining a final residual representing a difference between the image and the initial reconstructed image from the multiple predictions of the residual, and means for combining the initial reconstructed image and the residual to generate the final reconstructed image.

[0008] In some aspects, the device may be or may be part of a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a 3D scanner, a multi-camera system, or other device. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors.

[0009] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0010] The foregoing, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings: [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of an image processing system, according to some examples of the present disclosure. [Figure 2A] FIG. 1 illustrates an example of a fully connected neural network, in accordance with some examples of the present disclosure. [Figure 2B] FIG. 1 illustrates an example of a locally connected neural network, according to some examples of the present disclosure. [Figure 2C] FIG. 1 illustrates an example of a convolutional neural network, in accordance with some examples of the present disclosure. [Figure 2D] FIG. 1 illustrates an example of a deep convolutional network (DCN) for recognizing visual features from images, in accordance with some examples of the present disclosure. [Figure 3] FIG. 1 is a block diagram illustrating an example deep convolutional network (DCN), in accordance with some examples of the present disclosure. [Figure 4A] FIG. 1 is a diagram showing an original image. [Figure 4B] 1A-1C illustrate examples of reconstructed images using different machine learning systems, according to some examples of the present disclosure. [Figure 4C] 1A-1C illustrate examples of reconstructed images using different machine learning systems, according to some examples of the present disclosure. [Figure 4D] 1A-1C illustrate examples of reconstructed images using different machine learning systems, according to some examples of the present disclosure. [Figure 5] FIG. 1 illustrates an example of a machine learning system including a diffusion model for reconstructing an image from an input image, in accordance with some examples of the present disclosure. [Figure 6] FIG. 1 illustrates another example of a machine learning system including a diffusion model for reconstructing an image from an input image, in accordance with some examples of the present disclosure. [Figure 7] FIG. 1 illustrates an example of a system including a transmitting device for compressing video content and a receiving device for decompressing a received bitstream into video content, according to some examples of the present disclosure. [Figure 8] FIG. 1 illustrates an example of a rate-distortion autoencoder system, in accordance with some examples of this disclosure. [Figure 9]FIG. 1 illustrates an example of a machine learning system including a diffusion model conditioned on a reconstructed image and an auxiliary latent code, according to some examples of the present disclosure. [Figure 10] 1A-1C illustrate an example operation of a diffusion model for predicting a residual representing the difference between an input image and a reconstructed image, in accordance with some examples of the present disclosure. [Figure 11] FIG. 1 illustrates an example of a generative adversarial network (GAN) for reconstructing an image from an input image, in accordance with some examples of the present disclosure. [Figure 12] 1 is a flowchart illustrating an example of a process for processing image data using a machine learning system including a diffusion model, according to some examples of the present disclosure. [Figure 13] FIG. 1 illustrates an exemplary computing system, according to some examples of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to one skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and description are not intended to be limiting.

[0014] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for practicing the exemplary embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0015] As discussed above, media data (e.g., image data, video data, audio data, etc.) can contain large amounts of data, especially as the demand for high-quality video data continues to increase. For example, consumers of image, audio, and video data typically desire increasingly higher video quality, such as high fidelity, resolution, and frame rates. However, the large amounts of data required to meet such demands can place a significant strain on communication networks, such as high bandwidth and network resource requirements, as well as devices that process and store the video data. Therefore, compression algorithms (also called coding algorithms or coding tools) for reducing the amount of data required to store and / or transmit image and video data are advantageous.

[0016] Various techniques can be used to compress media data. For example, compression of image data has been achieved using algorithms such as Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG), among others. In some cases, video coding can be performed according to a particular video coding standard. Exemplary video coding standards include high-efficiency video coding (HEVC), essential video coding (EVC), advanced video coding (AVC), moving picture experts group (MPEG) coding, and versatile video coding (VVC). However, such conventional image and video coding techniques may cause artifacts in the reconstructed image after decoding is performed.

[0017] In some cases, machine learning systems can be used to compress and / or decompress data. Machine learning (ML) is a subset of artificial intelligence (AI). ML systems include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inference without explicit instructions. One example of an ML system is a neural network (also called an artificial neural network), which may be composed of an interconnected group of artificial neurons (e.g., neuron models). Neural networks can be used for a variety of applications and / or devices, such as image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, and service robots, among others. For example, neural network-based compression methods have shown considerable promise in compressing image data.

[0018] Individual nodes in a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Each vector and node in the network is associated with a weight value, which constrains how the input data relates to the output data. For example, each node's input data may be multiplied by the corresponding weight value and the products may be summed. The sum of the products may be adjusted by an optional bias, and an activation function may be applied to the result to generate the node's output signal or "output activation" (sometimes called an activation map or feature map). The weight values ​​may initially be determined by an iterative flow of training data through the network (e.g., the weight values ​​are established during a training phase in which the network learns how to identify particular classes according to their typical input data characteristics).

[0019] There are different types of neural networks and training mechanisms, such as deep generative neural network models (e.g., generative adversarial networks (GANs)), recurrent neural network (RNN) models, variational autoencoders (VAEs), multilayer perceptron (MLP) neural network models, convolutional neural network (CNN) models, and autoencoders (AEs), among others. For example, a GAN is a form of generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that would reasonably be derived from the original dataset. A GAN can include two neural networks working together, including a generative neural network (or generator) and a discriminative neural network (or discriminator).

[0020] RNNs operate on the principle of storing the output of a layer and feeding this output back to the input to aid in predicting the layer's outcome. In an MLP neural network, data is fed into the input layer, and one or more hidden layers may provide a level of abstraction to the data. Predictions may then be made in the output layer based on the abstracted data. MLPs may be particularly well-suited for classification and prediction problems, where inputs are assigned classes or labels. Convolutional neural networks (CNNs) are a type of feedforward artificial neural network. A CNN may include a collection of artificial neurons, each with a receptive field (e.g., a spatially localized region of input space) that collectively tile the input space. CNNs have many applications, including pattern recognition and classification.

[0021] In a hierarchical neural network architecture (called a deep neural network when there are multiple hidden layers), the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to a third layer of artificial neurons, and so on. Convolutional neural networks may be trained to recognize a hierarchy of features. Computation in convolutional neural network architectures may be distributed across a population of processing nodes, which may be configured in one or more computational chains. These multi-layer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.

[0022] An autoencoder (AE) can learn efficient data coding in an unsupervised manner. In some examples, the AE can learn a representation of a dataset (e.g., data coding) by training the network to ignore signal noise. The AE can include an encoder and a decoder. The encoder can map input data to codes, and the decoder can map the codes to a reconstruction of the input data. In some examples, a rate-distortion autoencoder (RD-AE), which is an instantiation of a VAE, can be trained to minimize the average rate-distortion loss over a dataset of data points, such as image and / or video data points. In some cases, the RD-AE can perform a forward pass during inference to encode new data points.

[0023] The goal of neural network-based compression is not to solve only the rate-distortion problem, but also the rate-distortion-perceptual tradeoff. Distortion (and therefore fidelity, which is the inverse of distortion) indicates the difference between the original image and the reconstructed version of the image output by AE or other machine learning systems. Rate refers to the amount of data required to represent the original image using a latent code. Perception refers to the perceptual quality, which is the quality a user perceives in the output of the machine learning system (e.g., a reconstructed image, a super-resolution image, etc.). For example, a reconstructed image of grass with high perceptual quality but low fidelity (and therefore high distortion) may not perfectly match the grass structure in the original image, but will still look very pleasing to the eye.

[0024] Generative models (e.g., GANs, VAEs, etc.) can be trained with perceptually driven losses, which can result in high-quality details that are pleasing to the human eye. Such models can significantly improve perceptual quality in low bitrate regimes, at the expense of distortion. Existing neural data compression methods that optimize in the perceptual domain are typically based on GANs.

[0025] Another variation of generative approaches to machine learning includes diffusion probabilistic models (DPMs), such as denoising diffusion probabilistic models (DDPMs). DPMs and DDPMs can provide good perceptual quality in image synthesis and super-resolution applications. DPMs have been shown to outperform GANs in terms of both perceptual quality and sample diversity. However, diffusion models have not been effectively used in the field of data compression, where improving the perceptual quality of reconstruction can have significant practical impact.

[0026] It can be difficult to generate high perceptual quality images at any data rate (or bit rate). Generative compression introduces a rate-distortion-perception (RDP) tradeoff, but outputting high perceptual quality images at any rate can be problematic. Traditional codecs (e.g., JPEG, MPEG, HEVC, VVC, etc.) only optimize for distortion and may output blurry images at low rates. Generative codecs (e.g., HiFiC) output high perceptual quality images at low rates, but are not optimized for distortion (and therefore fidelity), so they may differ too much from the original content (and therefore have low fidelity). At low rates, navigating the tradeoff between distortion and perceptual quality can be difficult.

[0027] Described herein are systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to herein as “systems and techniques”) for performing compression and decompression (also referred to as encoding and decoding, collectively referred to as coding) of data (e.g., image, video, audio) to reconstruct an image from an input image (also referred to as a reconstructed image) using a residual-based machine learning system (or codec). The machine learning system may include a codec machine learning model and a residual model configured to predict multiple residual predictions over several sampling steps based on an initial reconstruction of the image and an original input image (which the initial reconstruction represents). In some cases, the multiple residual predictions may include a sequence of predictions in a particular sequence. The residual model may include a diffusion model (e.g., DPM, DDPM, etc.), an RNN, or other model trained to predict multiple residual predictions based on an initial reconstruction of the image and the original image. As described in more detail herein, the number of sampling steps may be tuned or adjusted to meet a desired perceptual quality-fidelity tradeoff (e.g., at a particular bitrate).

[0028] A codec machine learning model can include an encoder (e.g., an encoder neural network architecture) and / or a decoder (e.g., a decoder neural network architecture). In some cases, the encoder neural network can be part of the same architecture as the decoder neural network. The encoder can process an input image to generate a latent representation (e.g., a latent code) of the input image. In some cases, the encoder can generate the latent representation at least in part by performing quantization and entropy coding (or using neural network operations to model quantization and entropy coding). The system can store the latent representation or transmit the latent representation to another system or device.

[0029] The decoder can obtain a latent representation of the image (e.g., a latent representation generated by an encoder or another latent representation of the image generated by an encoder in another system and transmitted to a system including the decoder) and process the latent representation to generate an initial reconstructed image that represents the input image. A residual model (e.g., a diffusion model, an RNN, etc.) of the machine learning system can be trained to predict a residual that represents a difference (e.g., a perceptual difference) between the original input image and the initial reconstructed image. The machine learning system can then combine the predicted residual with the initial reconstructed image to generate a final reconstructed image.

[0030] In some aspects, the residual model can be conditioned based on an initial reconstructed image and trained to optimize perceptual quality. For example, to predict the residual, the residual model can process an initial reconstruction and noise data (e.g., a noisy image containing random noise) to generate multiple predictions of the residual over several sampling steps (e.g., defined by a sampling schedule). The number of sampling steps can be modified or tuned (e.g., by modifying the sampling schedule) to improve or decrease perceptual quality. For example, the residual between the initial reconstruction and the target image is often sparse and easy to model. As the number of sampling steps decreases, the complexity and latency of the residual model become smaller (e.g., resulting in a reduced number of floating-point operations per pixel, floating-point operations per second, or FLOPS, or other metrics), but at the expense of decreased perceptual quality. For example, the residual model can be tuned over a small number of steps (e.g., 100 steps) to achieve a desired perceptual quality and fidelity with low complexity (a relatively small DDPM with a small number of sampling steps). The perceptual quality and fidelity can be determined by the number of steps used to determine the prediction residual at a particular bitrate (e.g., the bitrate can be set by the encoder of the codec machine learning model).

[0031] In some aspects, the system can determine or modify the number of sampling steps based on inputs, such as user inputs, inputs based on system or device constraints, inputs based on environmental conditions, and / or inputs based on other factors. In some examples, the inputs can include user inputs received via a user interface, which can be used to configure the machine learning system. In one illustrative example, the user inputs can indicate a particular number of steps that corresponds to a particular perceptual quality-fidelity tradeoff. In another illustrative example, the user inputs can indicate a desired perceptual quality, a desired fidelity, or a desired perceptual quality-fidelity tradeoff, and based on the inputs, the system can determine the number of steps necessary to satisfy the perceptual quality, fidelity, or perceptual quality-fidelity tradeoff.

[0032] In some examples, the input may be based on system or device constraints. In one illustrative example, the constraints may include computing constraints of a device implementing the residual-based machine learning system. For example, the device may have limited computing power. The residual-based machine learning system may receive input from a processor or other component of the device indicating that the number of sampling steps should be set to a particular number to reduce computational load. In another illustrative example, the constraints may include power constraints of a device implementing the residual-based machine learning system. The residual-based machine learning system may receive input from a processor or other component of the device indicating that the number of sampling steps should be set to a particular number relative to the amount of power used by the residual-based machine learning system.

[0033] Thus, modeling the residual allows the system to trade off fidelity for perceptual quality or perceptual quality for fidelity by modifying the sampling schedule (e.g., starting late, stopping early, or anything in between). For example, the resulting sampling procedure can be a viable method in which sampling can be stopped at any sampling step at which the desired tradeoff is achieved. In one example, the residual model can be tuned to stop sampling early (e.g., after 120 sampling steps), in which case the final reconstructed image will have high fidelity at the expense of visual quality. In another example, the residual model can be tuned to stop late (e.g., after 250 sampling steps), in which case the final reconstructed image will have high visual quality at the expense of some fidelity. By using a variable rate-based codec, the codec can smoothly traverse the rate-distortion-perceptual tradeoff when tested. The variable sampling schedule of the residual model enables a codec with dynamic computing capabilities so that performance can be tuned to user or device constraints.

[0034] The initial reconstructed image has good fidelity / distortion properties (based on a codec machine learning model that has been trained according to a rate-distortion tradeoff). By training a residual model to optimize perceptual quality, the number of sampling steps can be adjusted to obtain a residual with a desired perceptual quality. The residual can then be added to the initial reconstructed image to generate a final reconstructed image with an optimized fidelity / distortion, rate, and perceptual quality tradeoff (e.g., an image that looks perceptually good and closely resembles the original input image compressed at a low bitrate).

[0035] In some aspects, the residual model can be conditioned based on the initial reconstructed image and on auxiliary latent codes generated by the encoder of the codec machine learning model. The auxiliary latent codes provide additional information that can be useful for the residual model to predict high-quality residuals that indicate perceptual differences between the original image and the initial reconstructed image. The auxiliary latent codes are not used by the decoder of the codec machine learning model.

[0036] Residual model-based machine learning systems can provide coder-decoders (codecs) that perform lossy compression of images, including high-resolution images. Residual model-based machine learning systems can achieve performance comparable to or better than that of GAN-based methods and enable smooth traversal of the rate-distortion-perceptual tradeoff during testing. However, GAN-based methods require retraining with different alternatives and therefore cannot flexibly trade off distortion against perceptual quality. For example, residual model-based machine learning systems can dynamically navigate the rate-distortion-perceptual tradeoff during testing (or inference). Furthermore, while sampling from a diffusion probability model is computationally expensive, the cost of sampling can be significantly reduced by adapting the number of sampling steps (e.g., by modifying the sampling schedule), as described above.

[0037] Furthermore, residual model-based machine learning systems solve the challenges of GAN-based generative compression, including the difficulty and complexity of the training pipeline and the lack of a fidelity (low distortion)-perceptual quality tradeoff parameter. Residual-based codecs also offer control over the fidelity (distortion)-perceptual quality tradeoff (e.g., based on input, device constraints, etc.), which is not provided by current compression solutions (e.g., GAN-based compression solutions).

[0038] One or more of the described machine learning systems can be trained and used to perform data compression and / or decompression, such as compressing and decompressing data such as images, video, and / or audio. The machine learning systems described herein can be trained to perform compression / decompression techniques that generate high-quality data output. The systems and techniques described herein can perform compression and / or decompression of any type of data. For example, in some cases, the systems and techniques described herein can perform compression and / or decompression of image data. As another example, in some cases, the systems and techniques described herein can perform compression and / or decompression of video data. As used herein, the terms “image” and “frame” are used interchangeably and refer to a standalone image or frame (e.g., a photograph) or a group or sequence of images or frames (e.g., comprising a video or other sequence of images / frames). As another example, in some cases, the systems and techniques described herein can perform compression and / or decompression of audio data. For simplicity, illustrative, and explanatory purposes, the systems and techniques described herein are described with reference to compression and / or decompression of image data (e.g., an image or frame, video, etc.). However, as mentioned above, the concepts described herein can also be applied to other modalities, such as audio data and any other type of data.

[0039] Further background and aspects related to the systems and techniques are described with respect to the figures.

[0040] 1 illustrates an example of an image processing system 100 according to some examples of the present disclosure. In some cases, the image processing system 100 may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, among other information, may be stored in a memory block associated with the neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with the graphics processing unit (GPU) 104, a memory block associated with the digital signal processor (DSP) 106, or a memory block 118, or may be distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 and / or the memory block 118.

[0041] Image processing system 100 may include GPU 104, DSP 106, connectivity block 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and / or additional processing blocks tailored to specific functions, such as a multimedia processor 112, which may detect and recognize features. In one implementation, NPU 108 is implemented in CPU 102, DSP 106, and / or GPU 104. Image processing system 100 may also include a sensor processor 114, one or more image signal processors (ISPs) 116, and / or storage device 120. In some examples, image processing system 100 may be based on the ARM instruction set.

[0042] Image processing system 100 may be part of a computing device or multiple computing devices. In some examples, image processing system 100 may be part of an electronic device (or devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a telephone system (e.g., a smartphone, a cellular phone, a conferencing system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smart watch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a game console, a video streaming device, a drone, an in-vehicle computer, a system-on-chip (SOC), an Internet of Things (IoT) device, or any other suitable electronic device(s).

[0043] Although image processing system 100 is shown as including several components, those skilled in the art will understand that image processing system 100 can include more or fewer components than those shown in Figure 1. For example, image processing system 100 can also, in some cases, include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 1. Illustrative examples of computing devices and hardware components that can be implemented with image processing system 100 are described below with respect to Figure 13.

[0044] Image processing system 100 and / or its components may be configured to perform compression and / or decompression (also referred to as encoding and / or decoding, collectively referred to as image coding) using the machine learning systems and techniques described herein. In some cases, image processing system 100 and / or its components may be configured to perform image or video compression and / or decompression using the techniques described herein. In some examples, the machine learning system may utilize a deep learning neural network architecture to perform compression and / or decompression of image, video, and / or audio data. By using a deep learning neural network architecture, the machine learning system may increase the efficiency and speed of content compression and / or decompression on a device. For example, a device using the described compression and / or decompression techniques may efficiently compress one or more images using the machine learning-based techniques and transmit the compressed one or more images to a receiving device, which may efficiently decompress the one or more compressed images using the machine learning-based techniques described herein. As used herein, an image may refer to a still image and / or a video frame associated with a sequence of frames (e.g., a video).

[0045] As mentioned above, a neural network is an example of a machine learning system. A neural network can include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is generated through output nodes in the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network can include a feature map or activation map, which can include artificial neurons (or nodes). The feature map can include filters, kernels, etc. The nodes can include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, a deep learning network can have a series of many hidden layers, where early layers are used to determine simple, low-level characteristics of the input and later layers build a hierarchy of more complex and abstract characteristics.

[0046] Deep learning architectures may learn a hierarchy of features. When presented with visual data, for example, a first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0047] Deep learning architectures can work particularly well when applied to problems that have a natural hierarchical structure. For example, electric vehicle classification may benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0048] Neural networks may be designed with various connectivity patterns. In feedforward networks, each neuron in a given layer communicates with neurons in a higher layer, and information is passed from lower layers to higher layers. As described above, hierarchical representations may be constructed within successive layers of a feedforward network. Neural networks may also have recurrent or feedback (also called top-down) connections. In recurrent connections, outputs from neurons in a given layer may be communicated to other neurons in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks delivered to the neural network in sequence. Connections from neurons in a given layer to neurons in a lower layer are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognizing high-level concepts can help identify specific low-level features of the input.

[0049] The connections between layers of a neural network may be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, a neuron in a first layer can communicate its output to every neuron in a second layer, resulting in each neuron in the second layer receiving input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layer of a locally connected neural network 204 can be configured so that each neuron in the layer has the same or similar connectivity pattern but different connection strength values ​​(e.g., 210, 212, 214, and 216). Because higher layer neurons in a given region can receive inputs that are tuned through training to the properties of a limited subset of all inputs to the network, the connectivity patterns of local connections can give rise to spatially distinct receptive fields within the higher layers.

[0050] An example of a locally connected neural network is a convolutional neural network. FIG. 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths (e.g., 208) associated with inputs for each neuron in the second layer are shared. Convolutional neural networks may be suitable for problems in which the spatial location of the inputs is meaningful. According to aspects of the present disclosure, the convolutional neural network 206 may be used to perform one or more aspects of video compression and / or decompression.

[0051] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN 200 designed to recognize visual features from images 226 input from an image capture device 230, such as an in-car camera. The DCN 200 in this example may be trained to identify traffic signs and numbers printed on traffic signs. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or traffic signals.

[0052] DCN 200 may be trained using supervised learning. During training, DCN 200 may be presented with an image, such as speed limit sign image 226, and then a forward pass may be computed to generate output 222. DCN 200 may include a feature extraction section and a classification section. Upon receiving image 226, convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate first set of feature maps 218. As an example, the convolutional kernel for convolutional layer 232 may be a 5×5 kernel that generates 28×28 feature maps. In this example, four different feature maps are generated in first set of feature maps 218, so four different convolutional kernels were applied to image 226 in convolutional layer 232. A convolutional kernel may also be referred to as a filter or convolutional filter.

[0053] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0054] 2D , a second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign," "60," and "100." A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 contains one or more features.

[0055] In this example, the probability in output 222 for "sign" and "60" is higher than the probability for other outputs 222, such as "30," "40," "50," "70," "80," "90," and "100." Prior to training, the output 222 produced by DCN 200 may be inaccurate. Therefore, an error can be calculated between output 222 and a target output. The target output is the ground truth (e.g., "sign" and "60") of image 226. The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 more closely matches the target output.

[0056] To adjust the weights, the learning algorithm can calculate a gradient vector for the weights. The gradient can indicate the amount by which the error would increase or decrease if the weights were adjusted. In the top layer, the gradient can correspond directly to the values ​​of the weights connecting activated neurons in the penultimate layer to neurons in the output layer. In lower layers, the gradient can depend on the values ​​of the weights and the calculated error gradients of the upper layers. The weights can then be adjusted so that the error decreases. This method of adjusting the weights is sometimes called "backpropagation" because it involves a "backward pass" through the neural network.

[0057] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the overall system stops decreasing or reaches a target level. After training, the DCN can be presented with new images, and a forward pass through the network can produce output 222, which can be considered the DCN's inference or prediction.

[0058] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of a training dataset. DBNs can be obtained by stacking layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions without information about which class each input should be classified into, RBMs are frequently used in unsupervised learning. Using a hybrid unsupervised-supervised paradigm, the lower RBM of a DBN can be trained in an unsupervised manner and function as a feature extractor, while the upper RBM can be trained in a supervised manner (on the joint distribution of inputs from previous layers and the target class) and function as a classifier.

[0059] Deep convolutional networks (DCNs) are networks of convolutional networks constructed with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for a large number of examples and are used to modify the network weights using gradient descent.

[0060] A DCN may be a feedforward network. Additionally, as described above, connections from neurons in a first layer of a DCN to groups of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be exploited for high-speed processing. The computational burden of a DCN may be much less than that of a similarly sized neural network that includes, for example, recurrent or feedback connections.

[0061] The processing in each layer of a convolutional network may be viewed as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input may be viewed as three-dimensional, with two spatial dimensions along the image axes and a third dimension capturing color information. The outputs of the convolutional connections may be viewed as forming feature maps in subsequent layers, with each element of the feature map (e.g., 220) receiving input from a range of neurons in the previous layer (e.g., feature map 218) and from each of multiple channels. Values ​​in the feature map may be further processed using nonlinearities such as rectification, max(0,x), etc. Values ​​from neighboring neurons may also be pooled, which corresponds to downsampling and may provide additional local invariance and dimensionality reduction.

[0062] 3 is a block diagram illustrating an example of a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may be configured using a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0063] The convolutional layer 356 may include one or more convolutional filters that may be applied to the input data 352 to generate feature maps. While only two convolutional blocks 354A, 354B are shown, the present disclosure is not so limited; instead, any number of convolutional blocks (e.g., blocks 354A, 354B) may be included in the deep convolutional network 350 according to design preference. The normalization layer 358 may normalize the outputs of the convolutional filters. For example, the normalization layer 358 may provide whitening or lateral inhibition. The max-pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0064] For example, a parallel filter bank of a deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the image processing system 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the image processing system 100. Furthermore, the deep convolutional network 350 may have access to other processing blocks that may be present on the image processing system 100, such as the sensor processor 114.

[0065] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 are weights (not shown) that are to be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as the input of a subsequent one of the layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from input data 352 (e.g., image, audio, video, sensor data, and / or other input data) initially provided at convolution block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, each probability being the probability that the input data contains a feature from the set of features.

[0066] Image, audio, and video content may be stored and / or shared among devices. For example, image, audio, and video content can be uploaded to media hosting services and sharing platforms and transmitted to various devices. Recording uncompressed image, audio, and video content generally results in large file sizes that increase significantly as the resolution of the image, audio, and video content increases. For example, uncompressed 16-bit-per-channel video recorded at 1080p / 24 (e.g., a resolution of 1920 pixels wide by 1080 pixels high, captured at 24 frames per second) may occupy 12.4 megabytes per frame, or 297.6 megabytes per second. Uncompressed 16-bit-per-channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame, or 1195.2 megabytes per second.

[0067] Because uncompressed image, audio, and video content can result in large files that can require significant memory for physical storage and significant bandwidth for transmission, techniques may be utilized to compress such video content. For example, various compression algorithms may be applied to image, audio, and video content to reduce the size of the image content and, therefore, the amount of storage required to store the image content and the amount of bandwidth required to deliver the video content.

[0068] In some cases, image content may be compressed using a priori-defined compression algorithms such as the Joint Photographic Experts Group (JPEG) and Better Portable Graphics (BPG), among others. For example, JPEG is a lossy compression format based on the discrete cosine transform (DCT). For example, a device performing JPEG compression of an image may convert the image to an optimal color space (e.g., the YCbCr color space, which includes luminance (Y), blue chrominance (Cb), and red chrominance (Cr)), downsample the chrominance components by averaging groups of pixels together, apply a DCT function to blocks of pixels to remove redundant image data, and compress the image data. Compression is based on identifying similar regions within an image and converting the regions to the same color code (based on the DCT function). Video content may be compressed using a priori-defined compression algorithms such as the Moving Picture Experts Group (MPEG) algorithm, H.264, or the High Efficiency Video Coding algorithm.

[0069] These a priori-defined compression algorithms may be able to preserve most of the information in raw image and video content and may be defined a priori based on ideas from signal processing and information theory. However, while these predefined compression algorithms may be generally applicable (e.g., to any type of image / video content), the compression algorithms may not take into account content similarity, new resolutions or frame rates for video capture and delivery, non-natural imagery (e.g., radar imagery or other imagery captured via various sensors), etc.

[0070] Compression algorithms defined a priori are considered lossy compression algorithms. In lossy compression of an input image (or video frame), the input image cannot be coded and then decoded / reconstructed to reconstruct the exact input image. Rather, in lossy compression, an approximate version of the input image is generated after decoding / reconstruction of the compressed input image. Lossy compression results in bit rate reduction at the expense of distortion, resulting in artifacts present in the reconstructed image. Thus, there is a rate-distortion tradeoff in lossy compression systems. In certain compression methods (e.g., JPEG, BPG, among others), distortion-based artifacts can take the form of blocking or other artifacts. In some cases, neural network-based compression can be used, which can result in high-quality compression of image and video data. In some cases, example artifacts include blurring and color shifts.

[0071] Whenever the bit rate falls below the true entropy of the input data, it may be difficult or impossible to reconstruct the exact input data. However, the fact that distortion / loss occurs from compressing / decompressing data does not mean that there will be no artifacts in the reconstructed image or frame. In fact, it may be possible to reconstruct the compressed image into another, similar but different image with high visual quality.

[0072] As mentioned above, compression and decompression can be performed using one or more machine learning (ML) systems. In some cases, such ML-based systems can provide image and / or video compression that produces high-quality visual output. In some examples, such systems can perform compression and decompression of content (e.g., image content, video content, audio content, etc.) using deep neural network(s), such as a rate-distortion autoencoder (RD-AE). The deep neural network can include an autoencoder (AE) that maps an image to a latent code space (e.g., including a set of codes z). The latent code space can include a code space used by the encoder and decoder in which content is encoded into codes z. A code (e.g., code z) can also be referred to as a latent, latent variable, or latent representation. The deep neural network can include a probabilistic model (also referred to as a prior model or code model) that can losslessly compress a code z from the latent code space. The probabilistic model can generate a probability distribution over a set of codes z that can represent the encoded data based on input data. In some cases, the probability distribution can be denoted as (P(z)).

[0073] In some examples, the deep neural network may include an arithmetic coder that generates a bitstream containing compressed data to be output based on a probability distribution P(z) (which may itself be a neural network) and / or a set of codes z. The bitstream containing compressed data may be stored and / or transmitted to a receiving device. The receiving device may perform an inverse process to decode or decompress the bitstream, for example, using an arithmetic decoder, a probability (or code) model, and a decoder of the AE. The device that generated the bitstream containing compressed data may perform a similar decoding / decompression process when retrieving the compressed data from storage.

[0074] As mentioned above, the goal of machine learning-based (e.g., neural network-based) data compression is to resolve the rate-distortion-perceptual tradeoff. Figure 4A shows an original image 402, and Figures 4B-4D show example reconstructed images 404, 406, and 408. Reconstructed images 404, 406, and 408 were generated by processing original image 402 using different machine learning systems optimized for different aspects of rate, distortion, and perception.

[0075] For example, reconstructed image 404 was generated using a machine learning system optimized for rate and distortion. As a result, reconstructed image 404 has low distortion (and therefore high fidelity) but low perceptual quality (e.g., blurry). Reconstructed image 406 was generated using a machine learning system optimized for rate and perceptual performance. Because distortion is not part of the optimization, reconstructed image 406 is very different from original image 402 but has high perceptual quality. Reconstructed image 408 was generated using a machine learning system optimized for rate, distortion, and perceptual performance (e.g., utilizing a diffusion model, an RNN, etc.), such as the residual-based machine learning system (or codec) described herein. As a result, reconstructed image 408 is similar in content to original image 402 and has high perceptual quality, while allowing the latent code representation of original image 408 to be compressed to a low bit rate.

[0076] As previously described, systems and techniques are described herein for providing a residual-based machine learning system (e.g., a machine learning codec) for performing compression and decompression of data (e.g., images, video, audio, etc.). The residual-based machine learning system may include a codec machine learning model for compressing an input image into a latent code and generating an initial reconstructed version of the input image based on the latent code. In some cases, the codec machine learning model may implement a rate-distortion neural image (e.g., still image or video frame) compression algorithm (e.g., a mean-scale hyperprior), which may provide good baseline performance in rate-distortion.

[0077] The residual-based machine learning system may further include a residual model for generating a residual that represents a difference (e.g., a perceptual difference) between the original input image and the initial reconstructed image. As previously described, the residual model may include a diffusion model (e.g., DPM, DDPM, etc.), a recurrent neural network (RNN), or other types of machine learning models. The residual-based machine learning system may then combine the prediction residual with the initial reconstructed image to generate a final reconstructed image.

[0078] Residual models can be trained to improve perceptual quality, which constrains modeling to the residual. Fewer sampling steps are required than in traditional unconditional DPM settings, which can be based on residuals that are sparser and more normally distributed than the original input image. In some cases, fast sampling schemes can be implemented for the residual model, which can provide further control over the fidelity-perceptual tradeoff through the number of sampling steps T and additional sampling time hyperparameters.

[0079] As mentioned above, the residual model can be a diffusion model. In some cases, the diffusion model can be a DPM or a DDPM. In general, a denoising diffusion probability model is a set of T steps of latents x1,...,xT are latent variable models where x is defined as a Markov chain with a non-stochastic process (e.g., Gaussian transitions). They define a forward process that gradually corrupts the original data x as follows:

[0080]

number

[0081] A generative model (e.g., a U-net) parameterized by θ can then learn to reverse this process, modeling the transition from the tractable distribution to the data distribution.

[0082]

number

[0083] where the starting noise distribution is

[0084]

number

[0085] In the context of variational inference, training involves solving for the latent variables p θ (x 1:T |x0) as a posterior value for the forward pass q(x 1:T |x0) and p θ This can be seen as deriving a lower bound on the marginal likelihood for the log-likelihood of (x0). Assuming that the latent is a Markov chain with Gaussian transitions in both the target distribution and the variational distribution, the parameterization by Ho et al. (2020)

[0086]

number

[0087]

number

[0088]

number

[0089] where ∈ θ is typically x t It is parameterized as a neural network that learns to predict noise from

[0090]

number

[0091] Neural network-based codecs are systems that learn to compress data from examples. Many existing neural codecs are compressive autoencoders, which use an autoencoder-like architecture to compress data. e can take a data point x as input and output a quantized latent variable y. Given this latent, the neural decoder G d is reconstructed

[0092]

number

[0093] A neural codec can be trained using a rate-distortion objective that includes the following two terms:

[0094]

number

[0095] The rate loss corresponds to the number of bits required to transmit the quantized latent variable z under the neural prior,

[0096]

number

[0097]

number

[0098]

number

[0099] DDPM can be conditioned on side information, e.g., the target class, by providing input to the generative model. One example of how to apply conditional DDPM in a data compression setting is to define an encoder that maps the original input data to a compressed (quantized) representation y, and to directly condition DDPM on this representation. Then, we can model p(x|y) using the DDPM loss function Equation 3, using the following objective:

[0100]

number

[0101] A fully trained DDPM samples any data point from this distribution, although p(x|y) has much lower entropy than p(x), making the modeling task easier. However, in a data compression setting, data compression systems are primarily interested in data points with high fidelity.

[0102] As mentioned above, to improve perceptual quality, the residual-based machine learning systems and techniques described herein first generate a decoder G d An initial reconstruction can be generated using G, which can then be augmented using a residual model (e.g., DPM, DDPM, RNN, etc.). Figures 5, 6, and 9 show examples of residual-based machine learning systems that use a diffusion model (e.g., DDPM) as an illustrative example of a residual model, and are described in more detail below. A decoder in a residual-based machine learning system uses G c :={G d ,G ∈}, where:

[0103]

number

[0104] Initial predictions

[0105]

number

[0106]

number

[0107]

number

[0108]

number

[0109]

number

[0110]

number

[0111]

number

[0112] The machine learning system can be trained in two stages to provide a rate-distortion-perceptual quality tradeoff. For example, in the first stage, a codec machine learning model (which may be called the base codec) can be optimized using the rate-distortion loss from Equation 6. Initial Reconstruction

[0113]

number

[0114]

number

[0115] From the residual model

[0116]

number

[0117]

number

[0118]

number

[0119]

number

[0120] An example of a loss function that may be used to train a residual model (e.g., the diffusion model 526 described below with respect to FIG. 5) without an auxiliary latent is as follows:

[0121]

number

[0122] An example of a loss function that can be used to train a residual model with auxiliary latents (e.g., the diffusion model with auxiliary latents 526 described with respect to FIG. 9) is as follows:

[0123]

number

[0124] In some cases, similar to those described above, a codec machine learning model (e.g., codec machine learning model 521) first calculates L distortion +β·L rate Then the residual model (e.g., diffusion model 526 in FIG. 5 ) and auxiliary encoder (e.g., of encoder 522 in FIG. 5 ) are trained using L (with the weights of the codec machine learning model remaining frozen during the training of the residual model). diffusion +δ L rate,aux can be trained using

[0125] By separating the two stages, the residual-based machine learning system can generate a high-fidelity initial reconstruction. Depending on the application, the residual-based machine learning system can then use the final residual output by the residual model to decide whether and how much to augment the initial reconstruction.

[0126] The sampling process for diffusion models (e.g., DDPM) typically involves many steps (e.g., thousands of steps), and the best performance can be obtained using a large number of sampling steps. The denoising diffusion implicit model (DDIM) provides a training-free sampling scheme based on a generalization of the forward process to a non-Markovian diffusion process. This choice of forward process results in a "shorter" generative Markov chain, which improves sample efficiency. However, using this non-Markovian generative process sacrifices quality, especially for high-resolution images, and involves a large number of sampling steps. In a data compression setting, fidelity may be a more important factor than sample diversity. In addition, it may be important to keep computational cost low, which can be achieved by using as few sampling steps as possible. The residual model described herein may, in some cases, use DDIM sampling.

[0127] For a small number of sampling steps, the system may predict r0 directly instead of ∈0. One explanation is:

[0128]

number

[0129] Using the residual method and direct r prediction, the residual model stops sampling at any time step,

[0130]

number

[0131] Next, various illustrative examples are described using a diffusion model (e.g., DDPM) as an illustrative example of a residual model, in which case the residual-based machine learning systems and techniques may be referred to as diffusion-based machine learning systems and techniques. Although the examples are described using a diffusion model as an illustrative example of a residual model, other types of machine learning models or training mechanisms may be used as the residual model, such as an RNN, an RD-AE, or other types of VAE, any combination thereof, and / or other types of machine learning models or training mechanisms.

[0132] Figure 5 illustrates an example of a diffusion-based machine learning system 500 for reconstructing an image from an input image. The diffusion-based machine learning system 500 includes a base codec machine learning model 521 (or base codec) and a diffusion model 526. Figure 6 illustrates another example of a diffusion-based machine learning system (similar to system 500) that includes a diffusion model 926 implemented with a U-Net architecture.

[0133] In some cases, the codec machine learning model 521 can be implemented as a mean-scale hyperprior. As shown in FIG. 5, the codec machine learning model 521 includes an encoder 522 and a decoder 524. The encoder 522 receives an input image 520 (denoted as input image x) and processes the input image 520 to generate a latent code 523 (denoted as latent code z). The latent code 523 provides a latent representation of the input image 520. In some cases, the decoder 524 can obtain the latent code 523 generated by the encoder 522. In some cases, the decoder 524 can obtain another latent representation of the image (e.g., a latent code) generated by an encoder of another machine learning system. For example, the other machine learning system (or a device or system on which the other machine learning system is deployed) can generate a bitstream representing the latent representation of the image, such as by quantizing and / or entropy coding (e.g., using the arithmetic coder 706 of FIG. 7) the output of the encoder of the other machine learning system. Other machine learning systems (or devices or systems) can send bitstreams to the machine learning system 500 for processing by the decoder 524. The decoder 524 generates (initial reconstructed images

[0134]

number

[0135] 7 and 8 are diagrams illustrating examples of systems 700 and 800, respectively, that may be used to implement the codec machine learning model 521. For example, FIG. 7 illustrates the system 700 including a transmitting device 710 and a receiving device 720. In some examples, the codec machine learning model 521 may be or may include the transmitting device 710. In some examples, the codec machine learning model 521 may be or may include the receiving device 720.

[0136] The transmitting device 710 and the receiving device 720 may each, in some cases, be referred to as an RD-AE. The transmitting device 710 may compress image content, store the compressed image content, and / or transmit the compressed image content to the receiving device 720 for decompression. The receiving device 720 may decompress the compressed image content and output the decompressed image content on the receiving device 720 (e.g., for display, editing, etc.) and / or output the decompressed image content to another device connected to the receiving device 720 (e.g., a television, a mobile device, or other device). In some cases, the receiving device 720 may become a transmitting device by compressing image content (using the encoder 722) and storing and / or transmitting the compressed image content to another device, such as the transmitting device 710 (in which case the transmitting device 710 becomes a receiving device). In some examples, although the system 700 is described herein with respect to compressing and decompressing images, those skilled in the art will understand that the system 700 may use the techniques described herein to compress and decompress video, audio, and / or other media content.

[0137] As shown in FIG. 7 , the transmitting device 710 includes an image compression pipeline, and the receiving device 720 includes an image bitstream decompression pipeline. According to aspects of the present disclosure, the image compression pipeline in the transmitting device 710 and the bitstream decompression pipeline in the receiving device 720 generally use one or more artificial neural networks to compress image content and / or decompress a received bitstream into image content. The image compression pipeline in the transmitting device 710 includes an autoencoder 701, a code model 707, and an arithmetic coder 706. In some implementations, the arithmetic coder 706 is optional and, in some cases, may be omitted. The image decompression pipeline in the receiving device 720 includes an autoencoder 721, a code model 724, and an arithmetic decoder 726. In some implementations, the arithmetic decoder 726 is optional and, in some cases, may be omitted. The autoencoder 701 and code model 704 of the sending device 710 are shown in Figure 7 as machine learning systems that have been pre-trained and are therefore configured to perform operations during inference or operation of the trained machine learning system. The autoencoder 721 and code model 724 are also shown as pre-trained machine learning systems.

[0138] The autoencoder 701 includes an encoder 702 and a decoder 703. The encoder 702 can perform lossy compression on received uncompressed image content by mapping pixels in one or more images of the uncompressed image content to a latent code space (including codes z). The encoder 702 can be configured such that the latent codes z representing the compressed (or encoded) images are discrete or binary. In some cases, these codes can be generated based on stochastic perturbation techniques, soft vector quantization, or other techniques capable of generating distinct codes. In some aspects, the autoencoder 701 can map uncompressed images to codes having compressible (low-entropy) distributions. These latent codes may have cross-entropy close to a predefined or learned prior distribution.

[0139] In some examples, the autoencoder 701 may be implemented using a convolutional architecture. For example, in some cases, the autoencoder 701 may be configured as a two-dimensional convolutional neural network (CNN) such that the autoencoder 701 learns spatial filters for mapping image content to a latent code space. In examples where the system 700 is used to code video data, the autoencoder 701 may be configured as a three-dimensional CNN such that the autoencoder 701 learns spatiotemporal filters for mapping video to a latent code space. In such a network, the autoencoder 701 may encode video in terms of keyframes (e.g., initial frames marking the start of a sequence of frames in which subsequent frames in the sequence are described as differences relative to the initial frame in the sequence), warping (or differences) between the keyframe and other frames in the video, and residual factors. In other aspects, the autoencoder 701 may be implemented as a two-dimensional neural network conditioned based on previous frames, residual coefficients between frames, and conditioning via stacked channels or including recurrent layers.

[0140] The encoder 702 of the autoencoder 701 can receive as input a first image (designated as image x in FIG. 7) and can map the first image x to a code z in a latent code space (which can be referred to as a latent code). As described above, the encoder 702 can be implemented as a two-dimensional convolutional network, such that the latent code space has a vector at each (x, y) location that describes a block of image x centered at that location. The x coordinate can represent a horizontal pixel location within the block of image x, and the y coordinate can represent a vertical pixel location within the block of image x. When coding video data, the latent code space can have t variables or locations, where the t variables represent a timestamp within the block of video data (in addition to the spatial x and y coordinates). Using the two dimensions of horizontal and vertical pixel location, a vector can describe an image patch within image x.

[0141] The decoder 703 of the autoencoder 701 then reconstructs the first image x

[0142]

number

[0143]

number

[0144]

number

[0145] The code model 704 receives a code z that represents an encoded image or a portion thereof and can generate a probability distribution P(z) over a set of compressed codewords that can be used to represent the code z. In some examples, the code model 704 can include a stochastic autoregressive generative model. In some cases, the codes for which the probability distribution can be generated include learned distributions that control bit allocation based on the arithmetic coder 706. For example, the arithmetic coder 706 can be used to predict a compression code for a first code z alone, a compression code for a second code z based on the compression code for the first code z, a compression code for a third code z based on the compression codes for the first code z and the second code z, etc. The compression codes generally represent different spatiotemporal chunks of a given image to be compressed.

[0146] In some embodiments, the latent code z may be represented as a three-dimensional tensor. The three dimensions of the tensor are (e.g., the code z c,w,h Each code z (representing a code indexed by channel and horizontal and vertical position) may include a feature channel dimension (denoted as z , ... c,w,h is fixed and can be predicted based on the previous code, which in theory can be any order of the code. In some examples, the code can be generated by analyzing a given image file from beginning to end, analyzing each block in the image in raster scan order.

[0147] The code model 704 can use a stochastic autoregressive model to learn the probability distribution of the input code z. The probability distribution can be conditioned based on its previous values ​​(as described above). In some examples, the probability distribution can be expressed by the following equation:

[0148]

number

[0149] In some examples, the probability distribution P(z) can be predicted by a causal fully convolutional neural network. In some embodiments, the convolutional network uses the previous value z when calculating the probability distribution. 0:c,0:w,0:h The kernels of each layer of the convolutional neural network can be masked so that they recognize values ​​of σ, but cannot recognize other values. In some aspects, the final layer of the convolutional network can include a softmax function that determines the probability that a code in the latent space is applicable across input values ​​(e.g., the likelihood that a given code can be used to compress a given input).

[0150] The arithmetic coder 706 uses the probability distribution P(z) generated by the code model 704 to generate a bitstream 715 corresponding to a prediction of code z (shown in FIG. 7 as "0010011..."). The prediction of code z may be represented as the code having the highest probability score in the probability distribution P(z) generated over a set of possible codes. In some aspects, the arithmetic coder 706 may output a variable-length bitstream based on the accuracy of the prediction of code z and the actual code z generated by the autoencoder 701. For example, the bitstream 715 may correspond to a short codeword if the prediction is accurate, while the bitstream 715 may correspond to a longer codeword as the magnitude of the difference between code z and the prediction of code z increases.

[0151] In some cases, bitstream 715 may be output by arithmetic coder 706 for storage in a compressed image file. Bitstream 715 may also be output for transmission to a requesting device (e.g., receiving device 720, as shown in FIG. 7). In general, bitstream 715 output by arithmetic coder 706 may losslessly encode z so that z can be exactly restored during a decompression process applied to the compressed image file.

[0152] The bitstream 715 generated by the arithmetic coder 706 and transmitted from the transmitting device 710 may be received by the receiving device 720. Transmission between the transmitting device 710 and the receiving device 720 may occur using any of a variety of suitable wired or wireless communication technologies. Communication between the transmitting device 710 and the receiving device 720 may occur directly or through one or more network infrastructure components (e.g., base stations, relay stations, mobile stations, network hubs, routers, and / or other network infrastructure components).

[0153] As shown, receiving device 720 may include an arithmetic decoder 726, a code model 724, and an autoencoder 721. Autoencoder 721 includes an encoder 722 and a decoder 723. Decoder 723 may generate the same or similar output as decoder 703 for a given input. Although autoencoder 721 is shown as including encoder 722, encoder 722 may reconstruct code z from the code z received from transmitting device 710.

[0154]

number

[0155]

number

[0156] The received bitstream 715 may be input to an arithmetic decoder 726 to obtain one or more codes z from the bitstream. The arithmetic decoder 726 may extract the decompressed code z based on a probability distribution P(z) generated by a code model 724 over a set of possible codes and information associating each generated code z with the bitstream. Given a received portion of the bitstream and a probabilistic prediction of the next code z, the arithmetic decoder 726 may generate a new code z that is encoded by the arithmetic coder 706 at the transmitting device 710. Using the new code z, the arithmetic decoder 726 may make a probabilistic prediction of successive codes z, read additional portions of the bitstream, and decode successive codes z until the entire received bitstream is decoded. The decompressed code z may be provided to a decoder 723 within the autoencoder 721. The decoder 723 decompresses the code z and generates an approximation (which may be referred to as a reconstruction or decoded image) of the image content x.

[0157]

number

[0158]

number

[0159]

number

[0160] As mentioned above, the autoencoder 701 and code model 704 of the transmitting device 710 are shown in FIG. 7 as pre-trained machine learning systems. In some aspects, the autoencoder 701 and code model 704 can be trained together using image data. For example, the encoder 702 of the autoencoder 701 can receive a first training image n as input and can map the first training image n to a code z in a latent code space. The code model 704 can learn a probability distribution P(z) for the code z using a stochastic autoregressive model (similar to the techniques described above). The arithmetic coder 706 can use the probability distribution P(z) generated by the code model 704 to generate an image bitstream. Using the bitstream from the code model 704 and the probability distribution P(z), the arithmetic coder 706 can generate a code z and output the code z to the decoder 703 of the autoencoder 701. The decoder 703 can then decompress the code z to reconstruct the first training image n.

[0161]

number

[0162]

number

[0163] In some cases, a backpropagation engine used during training of the transmitting device 710 may perform a backpropagation process based on one or more loss functions to tune the parameters (e.g., weights, biases, etc.) of the neural network of the autoencoder 701 and the code model 704. In some cases, the backpropagation process may be based on stochastic gradient descent. The backpropagation may include a forward pass, one or more loss functions, a backward pass, and weight (and / or other parameter(s)) updates. The forward pass, loss function, backward pass, and parameter updates may be performed in a single training iteration. This process may be repeated for a certain number of iterations for each set of training data until the weights and / or other parameters of the neural network are precisely tuned.

[0164] For example, the autoencoder 701 can be

[0165]

number

[0166]

number

[0167]

number

[0168]

number

[0169] In yet another example, the loss function is

[0170]

number

[0171]

number

[0172] Based on the determined loss (e.g., distance vector or other difference value), and using a backpropagation process, the parameters (e.g., weights, biases, etc.) of the neural network system of the autoencoder 701 and code model 704 can be adjusted (effectively adjusting the mapping between the received image content and the latent code space) to reduce the loss between the input uncompressed image and the compressed image content produced as output by the autoencoder 701.

[0173] For the first training images, the loss (or error) may be high because the actual output values ​​(reconstructed images) may differ significantly from the input image. The goal of training is to minimize the amount of loss in the predicted output. The neural network can perform a backward pass by determining which nodes of the neural network (with corresponding weights) contributed most to the neural network's loss, and the weights (and / or other parameters) can be adjusted so that the loss decreases and is eventually minimized. To determine the weights that contributed most to the neural network's loss, the derivative of the loss with respect to the weight (denoted as dL / dW, where W is the weight in a particular layer) can be calculated. For example, the weights can be updated so that they change in the opposite direction of the gradient. Weight updates can be

[0174]

number

[0175] The neural network system of autoencoder 701 and code model 704 can continue to be trained in this manner until a desired output is achieved. For example, autoencoder 701 and code model 704 can repeat the backpropagation process to generate the input image n and the reconstructed image resulting from decompression of the generated code z.

[0176]

number

[0177] The autoencoder 721 and code model 724 may be trained using techniques similar to those described above for training the autoencoder 701 and code model 704 of the transmitting device 710. In some cases, the autoencoder 721 and code model 724 may be trained using the same or a different training dataset than that used to train the autoencoder 701 and code model 704 of the transmitting device 710.

[0178] 7, the rate-distortion autoencoders (sending device 710 and receiving device 720) are trained according to bit rates and run at inference time. In some implementations, the rate-distortion autoencoders can be trained at multiple bit rates to enable the generation and output of high-quality reconstructed images or video frames (e.g., with no or limited distortion artifacts to the input image) when various amounts of information are provided in the latent code z.

[0179] In some implementations, a set of continuous latents (e.g., capable of conveying a large amount of information) and corresponding quantized discrete latents (e.g., containing less information) may be used. After training the RD-AE model, an auxiliary inverse quantization model may be trained. In some cases, when using RD-AE, only the discrete latents are transmitted, and at the decoder side, an auxiliary inverse quantization model is used to infer the continuous latents from the discrete latents.

[0180] While system 700 is shown as including several components, those skilled in the art will understand that system 700 can include more or fewer components than those shown in FIG. 7 . For example, transmitting device 710 and / or receiving device 720 of system 700 may also, in some cases, include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in FIG. 7 . The components shown in FIG. 7 and / or other components of system 700 may be implemented using one or more computing or processing components. The one or more computing components may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or an image signal processor (ISP). Illustrative examples of computing devices and hardware components that may be implemented by system 1600 are described below with respect to FIG. 13 .

[0181] System 700 may be part of or implemented by a single computing device or multiple computing devices. In some examples, transmitting device 710 may be part of a first device, and receiving device 720 may be part of a second computing device. In some examples, transmitting device 710 and / or receiving device 720 may be included as part of an electronic device (or devices), such as a telephone system (e.g., a smartphone, a cellular phone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, a video streaming device, a SOC, an IoT (Internet of Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), or any other suitable electronic device(s). In some cases, system 700 may be implemented by image processing system 100 shown in FIG. 1 . In other cases, system 700 may be implemented by one or more other systems or devices.

[0182] 8 illustrates another exemplary neural network compression system 800 that may be used to implement the codec machine learning model 521 of FIG. 5. In some examples, the neural network compression system 800 may include an RD-AE system. In FIG. 8, the neural network compression system 800 includes an encoder 802, an arithmetic encoder 808, an arithmetic decoder 812, and a decoder 814. In some cases, the encoder 802 and / or the decoder 814 may be the same as the encoder 702 and / or the decoder 703, respectively. In other cases, the encoder 802 and / or the decoder 814 may be different from the encoder 702 and / or the decoder 703, respectively.

[0183] The encoder 802 may receive an image 801 (image x) as input and may map and / or transform the image 801 (image x) to a latent code 804 (latent z) in a latent code space. The image 801 may represent a still image and / or a video frame associated with a sequence of frames (e.g., a video). In some cases, the encoder 802 may perform a forward pass to generate the latent code 804. In some examples, the encoder 802 may implement a learnable function. In some cases, the encoder 802 may implement a learnable function parameterized by φ. For example, the encoder 802 may map and / or transform the image 801 (image x) to a latent code 804 (latent z) in a latent code space. The image 801 may represent a still image and / or a video frame associated with a sequence of frames (e.g., a video). In some cases, the encoder 802 may implement a learnable function parameterized by φ. For example, the encoder 802 may implement a learnable function q φ (z|x). In some examples, the learnable function need not be shared with or known to the decoder 814.

[0184] The arithmetic encoder 808 can generate a bitstream 810 based on the latent code 804 (latent z) and the latent prior distribution 806. In some examples, the latent prior distribution 806 can implement a learnable function. In some cases, the latent prior distribution 806 can implement a learnable function parameterized by ψ. For example, the latent prior distribution 806 can be a function p ψ (z) can be implemented. The latent prior 806 can be used to convert the latent code 804 (latent z) into a bitstream 810 using lossless compression. The latent prior 806 can be shared and / or made available at both the transmitter side (e.g., encoder 802 and / or arithmetic encoder 808) and the receiver side (e.g., arithmetic decoder 812 and / or decoder 814).

[0185] The arithmetic decoder 812 receives the coded bitstream 810 from the arithmetic encoder 808 and can use the latent prior 806 to decode the latent code 804 (latent z) in the coded bitstream 810. The decoder 814 converts the latent code 804 (latent z) into an approximate reconstructed image 816 (reconstructed

[0186]

number

[0187] The neural network compression system 800 can be trained to minimize rate-distortion. In some examples, the rate reflects the length of the bitstream 810 (bitstream b), and the distortion reflects the difference between the length of the image 801 (image x) and the reconstructed image 816 (reconstructed

[0188]

number

[0189] Returning to FIG. 5, as described above, the encoder 522 of the codec machine learning model 521 takes as input an image 520 (image x) and compresses it into a latent code 523 (latent z), and the decoder 524 of the codec machine learning model 521 converts the latent code 523 into an initial reconstructed image 525 (prediction

[0190]

number

[0191]

number

[0192] The diffusion model 526 may receive an initial reconstructed image 525 from the decoder 524 of the codec machine learning model 521. As described above, the diffusion model 526 is conditioned based on the initial reconstructed image 525. For example, the diffusion model 526 may be trained to predict a residual 527 (referred to as a prediction residual 527) that represents a difference (e.g., a perceptual difference) between the original image 520 and the initial reconstructed image 525. In some aspects, the diffusion model may be conditioned based on the initial reconstructed image and an auxiliary latent code generated by an encoder of the codec machine learning model. FIG. 9 illustrates an example of a machine learning system including a diffusion model 926 that is conditioned based on the initial reconstructed image 525 and an auxiliary latent code 930 generated by the encoder 522 of the codec machine learning system 521. The auxiliary latent code 930 provides additional information that may be useful for the diffusion model to predict high-quality residuals that indicate perceptual differences between the original image 520 and the initial reconstructed image 525. For example, the auxiliary latent code 930 provides the diffusion model 526 with access to additional information about the input image 520 beyond that contained in the initial reconstructed image 525. The auxiliary latent code 930 may be used only by the diffusion model 526 (and not by the decoder 524 of the codec machine learning model 521). In the example of FIG. 9, as shown in FIG.

[0193]

number

[0194] To predict the residuals, the diffusion model 526 is shown in Figure 5 as T , r T-1 The initial reconstructed image 525 and noise data may be processed to generate multiple predictions of the residual over a number of sampling steps (e.g., defined by a sampling schedule), denoted as , to a final step r0, where a final predicted residual 527 is determined at the final step r0. The noise data may, in some cases, include a noise image containing random noise. In some cases, sampling may be performed according to a DDIM sampling strategy. Through iterative sampling with a relatively small number of sampling steps T (e.g., 100-500 steps), the diffusion model 526 generates a final reconstructed image 525 (

[0195]

number

[0196] 10 is a diagram 1000 illustrating an example operation of the diffusion model 526 to predict a residual representing the difference between an input image (e.g., input image 520) and an initial reconstructed image 1025 (similar to reconstructed image 525). In a first sampling step (denoted as 0) of the diffusion model 526, the initial reconstructed image 1025 is combined (e.g., concatenated) with initial noise data 1042. The initial noise data 1042 may include an image of random noise. In the first sampling step (step 0), the diffusion model 526 can process the combined reconstructed image 1025 and initial noise data 1042 to predict a residual (e.g., a latent representation of a predicted residual).

[0197] In a subsequent sampling step, the diffusion model 526 may combine (e.g., concatenate) the initial reconstructed image 525 with the prediction residual predicted in the previous sampling step and process the newly combined data to predict another residual. For example, as shown in FIG. 10 , in a subsequent sampling step (denoted as step 50), the diffusion model 526 may obtain a latent 1044, which is a latent representation of the prediction residual predicted in the sampling step 49 of the diffusion model 526. In a final sampling step (denoted as step 100), the diffusion model may generate a final prediction residual 1027. Similar to what was described above with respect to FIG. 5 , the final prediction residual 1027 may be combined with the initial reconstructed image 1025 to generate a final reconstructed image 1029.

[0198] In some aspects, the number of sampling steps of the diffusion model 526 may be modified to improve or decrease the perceptual quality of the final reconstructed image 529. As the number of sampling steps decreases, the complexity and latency of the diffusion model becomes lower (e.g., resulting in a reduced number of floating-point operations per pixel, floating-point operations or FLOPS per second, or other metrics), but at the expense of decreased fidelity. For example, the diffusion model can be tuned with a small number of steps (e.g., 100 steps) to achieve the desired perceptual quality and fidelity at low complexity (a relatively small DDPM with a small number of sampling steps).

[0199] In some cases, the machine learning system 500 can receive an input indicating the number of sampling steps to use for the diffusion model 526. For example, as shown in FIG. 5, the diffusion model 526 can receive a sampling step input 539 that indicates the number of sampling steps to use to predict the prediction residual 527. Based on the sampling step input 539, the diffusion model 526 can determine or modify the number of sampling steps. The sampling step input 539 can include user input, input based on system or device constraints, input based on environmental conditions, any combination thereof, and / or other input.

[0200] In some examples, the input may include user input received via a user interface, which may be used to configure the machine learning system. In some cases, a user may provide input to a graphical user interface (e.g., providing a user-friendly knob that controls the perceptual-distortion tradeoff) to modify the sampling schedule of the diffusion model 526, which may increase or decrease the number of sampling steps of the diffusion model 526 (e.g., from 250 to 100 sampling steps, from 50 to 100 sampling steps, etc.). In one illustrative example, the user input may indicate a particular number of steps corresponding to a particular perceptual quality-fidelity tradeoff. In another illustrative example, the user input may indicate a desired perceptual quality, a desired fidelity, or a desired perceptual quality-fidelity tradeoff, and based on the input, the system may determine the number of steps necessary to satisfy the perceptual quality, fidelity, or perceptual quality-fidelity tradeoff.

[0201] In some examples, the input may be based on system or device constraints. In one illustrative example, the constraints may include computing constraints of a device implementing the residual-based machine learning system. For example, the device may have limited computing power. The residual-based machine learning system may receive input from a processor or other component of the device indicating that the number of sampling steps should be set to a particular number to reduce computational load. In another illustrative example, the constraints may include power constraints of a device implementing the residual-based machine learning system. The residual-based machine learning system may receive input from a processor or other component of the device indicating that the number of sampling steps should be set to a particular number relative to the amount of power used by the residual-based machine learning system.

[0202] In some cases, sampling can be stopped at any point at which a desired trade-off is achieved. In one illustrative example, the diffusion model can be tuned to stop sampling early (e.g., after 120 sampling steps), in which case the final reconstructed image will have high fidelity at the expense of visual quality. In another illustrative example, the diffusion model can be tuned to stop late (e.g., after 250 sampling steps), in which case the final reconstructed image will have high visual quality at the expense of some fidelity. Using such a variable sampling rate for the diffusion model 526 can enable the machine learning system 500 to smoothly traverse the rate-distortion-perceptual trade-off during testing. Variable sampling also enables codecs with dynamic computing capabilities so that performance can be tuned to user or device constraints.

[0203] In some cases, the machine learning system 500 can be trained such that the codec machine learning model 521 is trained (e.g., as described above with respect to FIG. 7) and the parameters (e.g., weights, biases, etc.) are frozen (remain fixed) during the training of the diffusion model 526, in which case the diffusion model 526 may only perform reinforcement. In other cases, the entire machine learning system 500 can be trained end-to-end.

[0204] The residual-based machine learning systems and techniques described herein can be used to generate output sample reconstructions (e.g., reconstructed images) of very high perceptual quality with good fidelity and rate characteristics. Residual model-based machine learning systems can provide codecs that perform lossy compression of images, including high-resolution images. Residual model-based machine learning systems can achieve performance that is competitive with or better than that of GAN-based methods and enable smooth traversal of the rate-distortion-perceptual tradeoff during testing, while GAN-based methods require retraining with different alternatives and therefore cannot flexibly trade off distortion against perceptual quality.

[0205] FIG. 11 illustrates an example of a generative adversarial network (GAN) 1100 for reconstructing an image from an input image. The GAN 1100 is a form of generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that would reasonably be obtained from the original data set. A GAN can include two neural networks operating together. One of the neural networks is called the generative neural network or generator 1150 (sometimes written as G(z)). The generator 1150 can include an encoder 1153 and a decoder 1155. The encoder 1153 can extract features from the input image x and output a latent code 1157. The decoder 1155 can receive the latent code 1157 and generate a synthetic output 1151 (e.g., a reconstructed image

[0206]

number

[0207] As an illustrative example, the training input and training output can include images. The generator 1150 is trained to attempt to fool the classifier 1152 into determining that the synthetic images generated by the generator 1150 are real images from the dataset. The training process continues, and the generator 1150 becomes better at generating synthetic images that look like real images. The classifier 1152 continues to look for defects in the synthetic images, and the generator 1150 figures out what the classifier 1152 is looking at to determine defects in the images. Once the GAN 1100 is trained, the generator 1150 can generate realistic-looking images that the classifier 1152 cannot distinguish from real images.

[0208] Unlike GAN 1100, the residual-based machine learning systems described herein (e.g., the diffusion-based machine learning system 500 of FIG. 5) can optimize perceptual quality through a discriminator / adversarial loss (e.g., the distortion-perceptual tradeoff is fixed after training). The residual-based machine learning systems described herein can also operate in image space instead of residual space. GAN-based systems also use much larger decoders (e.g., 120 million (120M) parameters) compared to the residual-based machine learning systems described herein (e.g., 30 million (30M) or less parameters).

[0209] The residual model-based machine learning system described herein can also dynamically navigate the rate-distortion-perceptual tradeoff during testing (or inference). Furthermore, while sampling from a diffusion probability model is computationally expensive, the cost of sampling can be significantly reduced by adapting the number of sampling steps (e.g., by modifying the sampling schedule), as described above. For example, as described above, the variable sampling of the diffusion model 526 allows for coding under multiple bit rates using the same codec (machine learning system 500), unlike other systems (e.g., GAN-based systems and methods).

[0210] 12 is a flowchart illustrating an example of a process 1200 for processing media data. Process 1200 may include, at block 1202, obtaining a latent representation of an image. In one illustrative example, the image may include image 520 of FIG. 5, and the latent representation of image 520 may include latent code 523. In some aspects, process 1200 may include obtaining an image and processing the image using an encoder of a machine learning model to generate the latent representation of the image. In one illustrative example, the encoder may include encoder 524 of FIG. 5.

[0211] Process 1200 may include processing the latent representation of the image using a decoder of the machine learning model to generate an initial reconstructed image at block 1204. In one illustrative example, the decoder may include decoder 524 of Figure 5, and the initial reconstructed image may include initial reconstructed image 525. In another example, the decoder may include decoder 723 of Figure 7.

[0212] At block 1206, process 1200 may include processing the initial reconstructed image and noise data using a residual model to predict multiple predictions (e.g., a sequence of predictions) of a residual over several sampling steps. The residual represents the difference between the image and the initial reconstructed image. In one illustrative example, the residual model may include a diffusion model, such as diffusion model 526 of FIG. 5 or diffusion model 926 of FIG. 9. In another illustrative example, the residual model may include a recurrent neural network (RNN). In some cases, a prediction of the residual of the multiple predictions may be determined at each sampling step of several sampling steps. In one example, the number of sampling steps may be the step r shown in FIG. 5 and / or FIG. 9. T , r T-1 , and r0. In another example, the number of sampling steps may include sampling steps 0 to 100 shown in FIG.

[0213] Process 1200 may include obtaining a final residual representing the difference between the image and the initial reconstructed image from multiple predictions of the residual at block 1208. For example, the final residual may include the residual determined at sampling step r0 shown in FIG. 5 (or FIG. 9) or sampling step 100 shown in FIG.

[0214] In some aspects, as described herein, the number of sampling steps is adjustable to obtain a trade-off between the rate of the latent representation, the perceptual quality of the final reconstructed image, and distortion between the image and the final reconstructed image. For example, process 1200 may include receiving an input indicating the number of sampling steps, and may determine the number of sampling steps based on the input. In one example, the input includes a user input received via a user interface as described herein. In another example, the input is based on constraints associated with the computing device (e.g., a computing device including a decoder and a residual model), such as computing constraints of the computing device, power constraints of the computing device, latency constraints associated with the computing device, any combination thereof, and / or other constraints. In another example, process 1200 may include determining the number of sampling steps based on a trade-off between a target perceptual quality and fidelity for the final reconstructed image (e.g., as indicated by user input, determined by the computing device, etc.).

[0215] In some cases, the residual model is conditioned based on an initial reconstructed image. For example, process 1200 may include processing the initial reconstructed image and noise data using the residual model in a first sampling step to predict a first prediction of the residual. Process 1200 may further include processing the initial reconstructed image and the first prediction of the residual using the residual model in a second sampling step to predict a second prediction of the residual. Process 1200 may continue to generate predictions of the residual until a configured final sampling step is reached. Process 1200 may include processing the initial reconstructed image and a previous prediction of the residual (e.g., a residual prediction determined in a sampling step immediately before the final sampling step, a second prediction of the residual in the above example, or another residual prediction during a sampling step) using the residual model in a third sampling step (e.g., a final sampling step such as sampling step r0 shown in FIG. 5 (or FIG. 9) or sampling step 100 shown in FIG. 10, or another sampling step) to predict a final residual.

[0216] Process 1200 may include combining the initial reconstructed image and the residual to generate a final reconstructed image at block 1210. In one illustrative example, the final reconstructed image may include final reconstructed image 529 of FIG. 5 and / or FIG. 9.

[0217] In some cases, the residual model is further conditioned based on an auxiliary latent representation of the image. For example, process 1200 may include obtaining an auxiliary latent representation of the image. In one illustrative example, the auxiliary representation may include auxiliary latent code 930 of FIG. 9. Process 1200 may further include processing the initial reconstructed image, the noise data, and the auxiliary latent representation of the image using the residual model to predict multiple predictions of the residual.

[0218] In some examples, the processes described herein (e.g., process 1200 and / or other processes described herein) may be performed by a computing device or apparatus. In one example, process 1200 may be performed by computing system 1300 shown in FIG. 13.

[0219] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device with the resource capabilities to perform the processes described herein, including process 1100, process 1200, process 1300, process 1400, process 1500, and / or other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0220] Components of a computing device may be implemented in circuitry. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0221] Process 1200 is illustrated as a logical flow diagram, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0222] Additionally, process 1200 and / or other processes described herein can be executed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or a combination thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0223] Figure 13 illustrates an example of a system for implementing certain aspects of the present technology. In particular, Figure 13 illustrates an example of a computing system 1300, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device comprising any components whose components communicate with each other using a connection 1305. The connection 1305 may be a physical connection using a bus or a direct connection to a processor 1310, such as in a chipset architecture. The connection 1305 may also be a virtual, networked, or logical connection.

[0224] In some embodiments, computing system 1300 is a distributed system, allowing the functionality described in this disclosure to be distributed across one data center, multiple data centers, within a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each performing some or all of the functionality described with respect to that component. In some embodiments, a component may be a physical or virtual device.

[0225] The exemplary system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components to the processor 1310, including system memory 1315 such as read-only memory (ROM) 1320 and random access memory (RAM) 1325. The computing system 1300 may include a cache 1312 of high-speed memory that is directly connected to the processor 1310, connected in close proximity to the processor 1310, or integrated as part of the processor 1310.

[0226] Processor 1310 may include any general-purpose processor, hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, configured to control processor 1310, and special-purpose processors where software instructions are embedded in the actual processor design. Processor 1310 may essentially be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0227] To enable user interaction, computing system 1300 includes input devices 1345, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1300 may also include output devices 1335, which may be one or more of a number of output mechanisms. In some cases, a multimodal system may allow a user to provide multiple types of input / output to communicate with computing system 1300. Computing system 1300 may include a communication interface 1340, which may generally govern and manage user input and system output.

[0228] The communications interface may be an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, an optical fiber port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11b / g, and the like.11 Wired and / or wireless transceivers may be used to perform or facilitate the reception and / or transmission of wired or wireless communications, including those utilizing Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.

[0229] Communications interface 1340 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of computing system 1300 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S.-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operating on any particular hardware configuration, and therefore, the basic features herein can be easily replaced as improved hardware or firmware configurations are developed.

[0230] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memory storage device, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical media, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC). circuit (IC) chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM),The memory may be a hard disk or other type of computer-readable medium capable of storing data that is accessible by a computer, such as random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0231] The storage devices 1330 may include software services, servers, services, etc., where code defining such software, when executed by the processor 1310, causes the system to perform a function. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium in association with the necessary hardware components, such as the processor 1310, connections 1305, output devices 1335, etc., to perform that function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instruction(s) and / or data. Computer-readable media may also include non-transitory media that can store data and do not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memories, or memory devices. Code and / or machine-executable instructions can be stored on a computer-readable medium, which can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0232] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0233] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks, including devices, device components, and method steps or routines embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0234] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0235] The processes and methods according to the above-described embodiments can be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions, or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the execution of the methods according to the described embodiments include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0236] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program product) to perform the necessary tasks can be stored in a computer-readable or machine-readable medium. A processor or processors can perform the necessary tasks. Typical example form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein can also be embodied in peripheral devices or add-in cards. Such functionality can also be implemented among various chips on a circuit board or among various processes running within a single device, as further examples.

[0237] The instructions, media for carrying such instructions, computing resources for executing those instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0238] While aspects of the present application have been described above with reference to specific embodiments thereof, those skilled in the art will recognize that the present application is not limited thereto. Therefore, while exemplary embodiments of the present application have been described in detail herein, it should be understood that, except as limited by the prior art, the concepts of the present application may be variously embodied and employed in other ways, and the appended claims are intended to be construed to include such variations. The various features and aspects of the present application described above may be used individually or in combination. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present application. Accordingly, the specification and drawings should be regarded as illustrative and not restrictive. For illustrative purposes, methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0239] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein can be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0240] Where a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0241] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or that is in communication with another component, either directly or indirectly (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0242] Claim language or other language referring to "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, a claim language referring to "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, a claim language referring to "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C, or any other ordering, overlap, or combination of A, B, and C, or any overlapping information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or A, B, and C. The phrases "at least one of" a set and / or "one or more" of a set do not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0243] Claim language or other language reciting "at least one processor configured to," "at least one processor configured to," "one or more processors configured to," "one or more processors configured to," etc. indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language "at least one processor configured to perform X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z, or that multiple processors can each be tasked with some subset of operations X, Y, and Z such that multiple processors together perform X, Y, and Z, or that a group of multiple processors can cooperate to perform operations X, Y, and Z. As another example, claim language "at least one processor configured to perform X, Y, and Z" can mean that any single processor can perform only at least a subset of operations X, Y, and Z.

[0244] When referring to one or more elements that perform a function (e.g., a step of a method), one element may perform all of the functions, or two or more elements may collectively perform the function. When two or more elements collectively perform a function, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in its entirety by only one element (e.g., different elements may perform different sub-functions of the function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform a function, one element may be configured to cause the other element to perform all of the functions, or two or more elements may be collectively configured to cause the other element to perform the function.

[0245] When referring to an entity (e.g., any entity or device described herein) that performs a function or is configured to perform a function (e.g., a step of a method), the entity may be configured to cause one or more elements to perform the function (individually or collectively). One or more elements of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. When referring to an entity that performs a function, the entity may be configured to cause one component to perform all of the functions, or to cause two or more components to collectively perform the functions. When an entity is configured to cause multiple components to collectively perform functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in its entirety by only one component (e.g., different components may perform different sub-functions of the function).

[0246] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0247] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium having program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. These techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave that carries or communicates program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer.

[0248] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0249] Illustrative examples of the present disclosure include the following:

[0250] Aspect 1. A method for processing image data, comprising: obtaining a latent representation of an image; processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict, over several sampling steps, multiple predictions of a residual representing a difference between the image and the initial reconstructed image; obtaining, from the multiple predictions of the residual, a final residual representing a difference between the image and the initial reconstructed image; and combining the initial reconstructed image and the residual to generate the final reconstructed image.

[0251] Aspect 2. The method of aspect 1, wherein the number of sampling steps is adjustable to obtain a trade-off between the rate of the latent representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image.

[0252] Aspect 3. The method of aspect 1 or 2, further comprising determining the number of sampling steps.

[0253] Aspect 4. The method of aspect 3, further comprising determining the number of sampling steps based on the setting.

[0254] Aspect 5. The method of aspect 3, further comprising configuring settings based on the input.

[0255] Aspect 6. The method of aspect 5, further comprising receiving an input.

[0256] Aspect 7. The method of aspect 5 or 6, wherein the input comprises user input received via a user interface.

[0257] Aspect 8. The method of any one of aspects 5 to 7, wherein the input is based on constraints associated with the computing device, and the constraints include at least one of a computing constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device.

[0258] Aspect 9. The method of any one of aspects 1 to 8, further comprising determining the number of sampling steps based on a trade-off between perceptual quality and fidelity for a goal of the final reconstructed image.

[0259] Embodiment 10. The method of any one of embodiments 1 to 9, wherein the residual model is conditioned based on an initial reconstructed image.

[0260] Aspect 11. The method of aspect 10, wherein the residual model is further conditioned based on an auxiliary latent representation of the image.

[0261] Aspect 12. The method of any one of aspects 1 to 11, further comprising: processing the initial reconstructed image and noise data using a residual model in a first sampling step to predict a first prediction of the residual; and processing the initial reconstructed image and the first prediction of the residual using the residual model in a second sampling step to predict a second prediction of the residual.

[0262] Aspect 13. The method of aspect 12, further comprising processing the initial reconstructed image and a previous prediction of the residual using a residual model in a third sampling step to predict a final residual.

[0263] Aspect 14. The apparatus of aspect 13, wherein the previous prediction of the residual is a second prediction of the residual or the third sampling step is a final sampling step, and the previous prediction of the residual is a residual prediction determined in a sampling step immediately preceding the final sampling step.

[0264] Aspect 15. The method of any one of aspects 1 to 14, further comprising acquiring an image and processing the image using an encoder of a machine learning model to generate a latent representation of the image.

[0265] Aspect 16. The method of any one of aspects 1 to 15, further comprising obtaining an auxiliary latent representation of the image; and processing the initial reconstructed image, the noise data, and the auxiliary latent representation of the image using a residual model to predict multiple predictions of the residual.

[0266] Aspect 17. The method of any one of aspects 1 to 16, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

[0267] Embodiment 18. The method of any one of embodiments 1 to 17, wherein the plurality of predictions comprises a sequence of predictions.

[0268] Aspect 19. An apparatus for processing image data, comprising: at least one memory; and at least one processor coupled to the at least one memory, configured to obtain a latent representation of the image; process the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; process the initial reconstructed image and noise data using a residual model to predict, over several sampling steps, multiple predictions of a residual representing a difference between the image and the initial reconstructed image; obtain from the multiple predictions of the residual a final residual representing a difference between the image and the initial reconstructed image; and combine the initial reconstructed image and the residual to generate the final reconstructed image.

[0269] Aspect 20. The apparatus of aspect 19, wherein the number of sampling steps is adjustable to obtain a trade-off between the rate of the latent representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image.

[0270] Aspect 21. The apparatus of aspect 19 or 20, wherein the at least one processor is configured to determine the number of sampling steps.

[0271] Aspect 22. The apparatus of aspect 21, wherein the at least one processor is configured to determine the number of sampling steps based on the setting.

[0272] Aspect 23. The apparatus of aspect 22, wherein the at least one processor is configured to configure settings based on the input.

[0273] Aspect 24. The apparatus of aspect 23, wherein the at least one processor is configured to receive input.

[0274] Aspect 25. The device of aspect 23 or 24, wherein the input comprises user input received via a user interface.

[0275] Aspect 26. The apparatus of any one of aspects 23 to 25, wherein the input is based on constraints associated with the computing device, and the constraints include at least one of a computing constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device.

[0276] Aspect 27. The apparatus of any one of aspects 19 to 26, wherein at least one processor is configured to determine the number of sampling steps based on a trade-off between a target perceptual quality and fidelity for the final reconstructed image.

[0277] Embodiment 28. The apparatus of any one of embodiments 19 to 27, wherein the residual model is conditioned based on an initial reconstructed image.

[0278] Aspect 29. The apparatus of aspect 28, wherein the residual model is further conditioned based on an auxiliary latent representation of the image.

[0279] Aspect 30. The apparatus of any one of aspects 19 to 29, wherein at least one processor is configured to process the initial reconstructed image and noise data using a residual model in a first sampling step to predict a first prediction of the residual, and to process the initial reconstructed image and the first prediction of the residual using the residual model in a second sampling step to predict a second prediction of the residual.

[0280] Aspect 31. The apparatus of aspect 30, wherein at least one processor is configured to process the initial reconstructed image and a previous prediction of the residual using a residual model in the third sampling step to predict a final residual.

[0281] Aspect 32. The apparatus of aspect 13, wherein the previous prediction of the residual is a second prediction of the residual, or the third sampling step is a final sampling step, and the previous prediction of the residual is a residual prediction determined in a sampling step immediately preceding the final sampling step.

[0282] Aspect 33. The apparatus of any one of aspects 19 to 32, wherein at least one processor is configured to obtain an image and process the image using an encoder of the machine learning model to generate a latent representation of the image.

[0283] Aspect 34. The apparatus of any one of aspects 19 to 33, wherein at least one processor is configured to process the initial reconstructed image, the noise data, and the auxiliary latent representation of the image using a residual model to obtain an auxiliary latent representation of the image and predict multiple predictions of the residual.

[0284] Aspect 35. The apparatus of any one of aspects 19 to 34, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

[0285] Embodiment 36. The apparatus of any one of embodiments 19 to 35, wherein the plurality of predictions comprises a sequence of predictions.

[0286] Aspect 37. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the operations set forth in any one of aspects 1-18.

[0287] Aspect 38. An apparatus for processing image data, comprising one or more means for performing the operations recited in any one of aspects 1 to 18.

Claims

1. 1. An apparatus for processing image data, comprising: at least one memory; at least one processor coupled to the at least one memory, Obtain the latent representation of the image, processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict a plurality of predictions of residuals representing differences between the image and the initial reconstructed image over a number of sampling steps; obtaining a final residual from the multiple predictions of the residual, the final residual representing the difference between the image and the initial reconstructed image; at least one processor configured to combine the initial reconstructed image and the residual to generate a final reconstructed image.

2. 2. The apparatus of claim 1, wherein the number of sampling steps is adjustable to obtain a trade-off between the rate of the latent representation, the perceptual quality of the final reconstructed image, and distortion between the image and the final reconstructed image.

3. the at least one processor: The apparatus of claim 1 configured to determine the number of sampling steps.

4. the at least one processor: The apparatus of claim 3 , configured to determine the number of sampling steps based on a setting.

5. the at least one processor: The device of claim 4 , configured to configure the settings based on an input.

6. the at least one processor: The device of claim 5 configured to receive the input.

7. The apparatus of claim 5 , wherein the input comprises a user input received via a user interface.

8. 6. The apparatus of claim 5, wherein the input is based on constraints associated with a computing device, the constraints including at least one of a computing constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device.

9. the at least one processor: The apparatus of claim 1 , configured to determine the number of sampling steps based on a trade-off between a desired perceptual quality and fidelity for the final reconstructed image.

10. The apparatus of claim 1 , wherein the residual model is conditioned based on the initial reconstructed image.

11. The apparatus of claim 10 , wherein the residual model is further conditioned based on an auxiliary latent representation of the image.

12. the at least one processor: processing the initial reconstructed image and the noise data using the residual model in a first sampling step to predict a first prediction of the residual; 2. The apparatus of claim 1, configured to process the initial reconstructed image and the first prediction of the residual using the residual model in a second sampling step to predict a second prediction of the residual.

13. the at least one processor:

13. The apparatus of claim 12, configured to process the initial reconstructed image and a previous prediction of the residual using the residual model in a third sampling step to predict the final residual.

14. The apparatus of claim 13 , wherein the previous prediction of the residual is the second prediction of the residual.

15. 14. The apparatus of claim 13, wherein the third sampling step is a final sampling step, and the previous prediction of the residual is a residual prediction determined in a sampling step immediately preceding the final sampling step.

16. the at least one processor: acquiring the image; The apparatus of claim 1 , configured to process the image using an encoder of the machine learning model to generate the latent representation of the image.

17. the at least one processor: obtaining an auxiliary latent representation of the image; 2. The apparatus of claim 1, configured to process the initial reconstructed image, the noise data, and the auxiliary latent representation of the image using the residual model to predict the multiple predictions of the residual.

18. The apparatus of claim 1 , wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

19. The apparatus of claim 1 , wherein the plurality of predictions comprises a sequence of predictions.

20. 1. A method for processing image data, comprising: Obtaining a latent representation of an image; processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict a plurality of predictions of residuals representing differences between the image and the initial reconstructed image over a number of sampling steps; obtaining, from the multiple predictions of the residual, a final residual representing the difference between the image and the initial reconstructed image; combining the initial reconstructed image and the residual to generate a final reconstructed image.

21. 21. The method of claim 20, wherein the number of sampling steps is adjustable to obtain a trade-off between the rate of the latent representation, the perceptual quality of the final reconstructed image, and distortion between the image and the final reconstructed image.

22. configuring a setting based on an input, the input being based on constraints associated with a computing device, the constraints including at least one of a computing constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device; determining the number of sampling steps based on the setting.

21. The method of claim 20.

23. determining the number of sampling steps based on a tradeoff between a desired perceptual quality and fidelity for the final reconstructed image.

21. The method of claim 20.

24. The method of claim 20 , wherein the residual model is conditioned based on the initial reconstructed image.

25. The method of claim 24 , wherein the residual model is further conditioned on an auxiliary latent representation of the image.

26. processing the initial reconstructed image and the noise data using the residual model in a first sampling step to predict a first prediction of the residual; processing the initial reconstructed image and the first prediction of the residual using the residual model in a second sampling step to predict a second prediction of the residual.

21. The method of claim 20.

27. and processing the initial reconstructed image and a previous prediction of the residual using the residual model in a third sampling step to predict the final residual.

27. The method of claim 26.

28. acquiring the image; processing the image using an encoder of the machine learning model to generate the latent representation of an image.

21. The method of claim 20.

29. obtaining an auxiliary latent representation of the image; processing the initial reconstructed image, the noise data, and the auxiliary latent representation of the image using the residual model to predict the plurality of predictions of the residual.

21. The method of claim 20.

30. 21. The method of claim 20, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

Images