Diffusion-based data compression

Abstract

Systems and techniques are described for processing image data using a residual model that may be configured with an adjustable number of sampling steps. For example, a process may include obtaining a potential representation of an image and processing the potential representation of the image using a decoder of a machine learning model to generate an initial reconstructed image. The process may also include processing the initial reconstructed image and noise data using the residual model to predict multiple predictions of residuals over multiple sampling steps. The residual represents a difference between the image and the initial reconstructed image. The process may include obtaining a final residual representative of the difference between the image and the initial reconstructed image from the plurality of predictions of the residual. The process may also include combining the initial reconstructed image and the residual to generate a final reconstructed image.

Description

Technical Field

[0001] The present disclosure generally relates to data compression. For example, aspects of the present disclosure include using a machine learning system to compress image and / or video content. Background Art

[0002] Many devices and systems allow processing and output of data such as media data (e.g., image data, video data, audio data, etc.) for consumption. For example, media data includes a large amount of data to meet the growing demand for image / video / audio quality, performance and features. For example, consumers of video data generally expect high-quality video with high fidelity, high resolution, high frame rate, etc. The large amount of video data that is usually required to meet these demands places a heavy burden on communication networks and devices that process and store video data. Video decoding (coding) technology can be used to compress video data. An example goal of video decoding is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the decline in video quality. As evolving video services become available and the demand for large amounts of video data continues to increase, decoding technology with better performance and efficiency is needed. Summary of the invention

[0003] In some aspects, systems and techniques for performing data compression and / or decompression using one or more machine learning systems are described. In some cases, a machine learning system (e.g., using one or more neural network systems) is provided that includes a residual model (e.g., a diffusion model such as a denoised diffusion probability model), a recurrent neural network (RNN), or other type of machine learning model for performing data compression using an adjustable number of sampling steps.

[0004] According to at least one illustrative example, a method for processing image data is provided. The method includes: obtaining a potential representation of an image; processing the potential representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict multiple predictions of residuals over multiple sampling steps, wherein the residual represents the difference between the image and the initial reconstructed image; obtaining a final residual representing the difference between the image and the initial reconstructed image from the multiple predictions of the residual; and combining the initial reconstructed image and the residual to generate a final reconstructed image.

[0005] In another example, a device for processing media data is provided, the device comprising at least one memory and at least one processor (e.g., configured in a circuit), the at least one processor being communicatively coupled to the at least one memory. The at least one processor is configured to: obtain a potential representation of an image; process the potential representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; process the initial reconstructed image and noise data using a residual model to predict multiple predictions of residuals over multiple sampling steps, wherein the residuals represent the difference between the image and the initial reconstructed image; obtain a final residual representing the difference between the image and the initial reconstructed image from the multiple predictions of the residuals; and combine the initial reconstructed image and the residual to generate a final reconstructed image.

[0006] In another example, a non-transitory computer-readable medium is provided comprising at least one instruction stored thereon, which, when executed by one or more processors, causes the one or more processors to: obtain a latent representation of an image; process the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; process the initial reconstructed image and noise data using a residual model to predict multiple predictions of residuals over multiple sampling steps, wherein the residuals represent the difference between the image and the initial reconstructed image; obtain a final residual representing the difference between the image and the initial reconstructed image from the multiple predictions of the residuals; and combine the initial reconstructed image and the residual to generate a final reconstructed image.

[0007] In another example, an apparatus for processing image data is provided. The apparatus may include: a component for obtaining a latent representation of an image; a component for processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; a component for processing the initial reconstructed image and noise data using a residual model to predict a plurality of predictions of residuals over a plurality of sampling steps, wherein the residuals represent the difference between the image and the initial reconstructed image; a component for obtaining a final residual representing the difference between the image and the initial reconstructed image from the plurality of predictions of the residuals; and a component for combining the initial reconstructed image and the residual to generate a final reconstructed image.

[0008] In some aspects, the apparatus can be a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smart phone", or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop, a server computer, a 3D scanner, a multi-camera system, or other device, or can be a part of them. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings, in which:

[0012] Figure 1 is a diagram illustrating an example of an image processing system according to some examples of the present disclosure;

[0013] Figure 2A is a diagram illustrating an example of a fully connected neural network according to some examples of the present disclosure;

[0014] Figure 2B is a diagram illustrating an example of a locally connected neural network according to some examples of the present disclosure;

[0015] Figure 2C is a diagram illustrating an example of a convolutional neural network according to some examples of the present disclosure;

[0016] Figure 2D is a diagram illustrating an example of a deep convolutional network (DCN) for recognizing visual features from an image according to some examples of the present disclosure;

[0017] Figure 3 is a block diagram illustrating an example deep convolutional network (DCN) according to some examples of the present disclosure;

[0018] Figure 4Ais a diagram illustrating an original image according to some examples of the present disclosure, and FIG. 4B to FIG. 4D is a diagram illustrating examples of reconstructing images using different machine learning systems according to some examples of the present disclosure;

[0019] Figure 5 is a diagram illustrating an example of a machine learning system including a diffusion model for reconstructing an image from an input image according to some examples of the present disclosure;

[0020] Figure 6 is a diagram illustrating another example of a machine learning system including a diffusion model for reconstructing an image from an input image according to some examples of the present disclosure;

[0021] Figure 7 is a diagram illustrating an example of a system including a transmitting device for compressing video content and a receiving device for decompressing a received bitstream into the video content according to some examples of the present disclosure;

[0022] Figure 8 is a diagram illustrating an example of a rate-distortion autoencoder system according to some examples of the present disclosure;

[0023] Fig. 9 is a diagram illustrating an example of a machine learning system including a diffusion model conditioned on a reconstructed image and an auxiliary latent code according to some examples of the present disclosure;

[0024] Fig.10 is a diagram illustrating example operations of a diffusion model for predicting a residual representing a difference between an input image and a reconstructed image according to some examples of the present disclosure;

[0025] Fig.11 is a diagram illustrating an example of a generative adversarial network (GAN) for reconstructing an image from an input image according to some examples of the present disclosure;

[0026] Fig.12 is a flow chart illustrating an example of a process for processing image data using a machine learning system including a diffusion model according to some examples of the present disclosure;

[0027] Fig.13 An example computing system according to some examples of the present disclosure is illustrated. DETAILED DESCRIPTION

[0028] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0029] The following description provides only example embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example embodiments will provide an enabling description for implementing the example embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0030] As described above, media data (e.g., image data, video data, and / or audio data) may include large amounts of data, particularly as the demand for high-quality video data continues to grow. For example, consumers of image, audio, and video data generally desire increasingly higher levels of quality, such as high fidelity, resolution, frame rate, etc. However, the large amounts of data required to meet such demands may place a heavy burden on communication networks (e.g., high bandwidth and network resource requirements) as well as on devices that process and store video data. Therefore, compression algorithms (also referred to as decoding algorithms or tools) for reducing the amount of data required to store and / or transmit image and video data are advantageous.

[0031] Various techniques may be used to compress media data. For example, compression of image data has been accomplished using algorithms such as Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG), etc. In some cases, video coding may be performed according to a specific video coding standard. Example video coding standards include High Efficiency Video Coding (HEVC), Basic Video Coding (EVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding, and Versatile Video Coding (VVC). However, such conventional image and video coding techniques may cause artifacts in the reconstructed image after decoding is performed.

[0032] In some cases, machine learning systems can be used to compress and / or decompress data. Machine learning (ML) is a subset of artificial intelligence (AI). ML systems include algorithms and statistical models that a computer system can use to perform various tasks by relying on patterns and inferences without using explicit instructions. An example of an ML system is a neural network (also known as an artificial neural network), which can be composed of a group of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in various applications and / or devices, such as image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc. For example, compression methods based on neural networks have shown great promise in compressing image data.

[0033] Individual nodes in a neural network can simulate biological neurons by taking input data and performing simple operations on the data. The results of simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​restrict the way in which the input data is related to the output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be added. The sum of the products can be tuned by an optional bias, and an activation function can be applied to the result, producing an output signal or "output activation" (sometimes referred to as an activation map or feature map) of the node. The weight values ​​can initially be determined by an iterative flow of training data through the network (for example, the weight values ​​are established during the training phase when the network learns how to identify a specific class by its typical input data characteristics).

[0034] There are different types of neural networks and training mechanisms, such as deep generative neural network models (e.g., generative adversarial networks (GANs)), recurrent neural network (RNN) models, variational autoencoders (VAEs), multilayer perceptron (MLP) neural network models, convolutional neural network (CNN) models, autoencoders (AEs), etc. For example, a GAN is a generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that are reasonably likely to come from the original data set. A GAN may include two neural networks operating together, including a generative neural network (or generator) and a discriminative neural network (or discriminator).

[0035] RNN works on the following principle: save the output of a layer and feed it back to the input to help predict the result of that layer. In an MLP neural network, data can be fed to an input layer, and one or more hidden layers provide a level of abstraction for the data. The output layer can then be predicted based on the abstract data. MLP can be particularly suitable for classification prediction problems that assign categories or labels to inputs. Convolutional neural network (CNN) is a feedforward artificial neural network. CNN can include a collection of artificial neurons, each of which has a corresponding receptive field (e.g., a spatially local region of the input space) and together tiles the input space. CNN has many applications, including pattern recognition and classification.

[0036] In a layered neural network architecture (called a deep neural network when there are multiple hidden layers), the outputs of the first layer of artificial neurons become the inputs of the second layer of artificial neurons, the outputs of the second layer of artificial neurons become the inputs of the third layer of artificial neurons, and so on. Convolutional neural networks can be trained to recognize hierarchical structures of features. The computations in a convolutional neural network architecture can be distributed across a set of processing nodes, which can be configured in one or more computation chains. These multi-layer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.

[0037] An autoencoder (AE) can learn effective data decoding in an unsupervised manner. In some examples, an AE can learn a representation of a set of data (e.g., data decoding) by training a network to ignore signal noise. An AE may include an encoder and a decoder. The encoder may map input data into a code, and the decoder may map the code to a reconstruction of the input data. In some examples, a rate-distortion autoencoder (RD-AE), which is a specific example of a VAE, may be trained to minimize the average rate-distortion loss over a data set of data points (such as images and / or video data points). In some cases, an RD-AE may perform a forward pass to encode new data points when inferring.

[0038] The goal of neural network-based compression is to solve the rate-distortion-perception tradeoff, rather than rate-distortion alone. Distortion (and in turn fidelity, which is the inverse of distortion) indicates the difference between the original image and the reconstructed version of the image output by the AE or other machine learning system. Rate refers to the amount of data required to represent the original image using the latent code. Perception refers to the perceived quality, which is the quality perceived by the user in the output of the machine learning system (e.g., reconstructed image, image with super-resolution applied, etc.). For example, a reconstructed image of grass with high perceptual quality but low fidelity (and therefore high distortion) may not perfectly match the grass structure in the original image, but still look very pleasing to the eye.

[0039] Generative models (e.g., GANs, VAEs, etc.) can be trained with perceptually driven losses, which can produce pleasing, high-quality details. Such models can greatly improve perceptual quality at low bitrates, but at the expense of distortion. Existing neural data compression methods that optimize perceptual domains are usually based on GANs.

[0040] Another variation of generative methods for machine learning includes diffusion probability models (DPMs), such as the denoising diffusion probability model (DDPM). DPM and DDPM can provide good perceptual quality in applications of image synthesis and super-resolution. DPM has been shown to outperform GANs in both perceptual quality and sample diversity. However, diffusion models have not been effectively used in the field of data compression, where improvements in the perceptual quality of reconstruction could lead to significant practical impact.

[0041] Generating images of high perceptual quality at any data rate (or bitrate) can be difficult. Generative compression introduces a rate-distortion-perception (RDP) tradeoff, but there can be problems in outputting high perceptual quality images at any rate. Conventional codecs (e.g., JPEG, MPEG, HEVC, VVC, etc.) may output blurry images at low rates, optimizing only for distortion. Generative codecs (e.g., HiFiC) output high perceptual quality images at low rates, but may differ too much from the original content (and therefore have low fidelity) because they are not optimized for distortion (and therefore fidelity). At low rates, it can be difficult to compromise between distortion and perceptual quality.

[0042] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for performing data (e.g., image, video, audio, etc.) compression and decompression (also referred to as encoding and decoding, collectively referred to as decoding) using a residual-based machine learning system (or codec) for reconstructing an image from an input image (referred to as a reconstructed image). The machine learning system may include a codec machine learning model and a residual model configured to predict a plurality of residual predictions over a plurality of sampling steps based on an initial reconstruction of the image and an original input image (represented by the initial reconstruction). In some cases, the plurality of residual predictions may include a sequence of predictions in a particular sequence. The residual model may include a diffusion model (e.g., DPM, DDPM, etc.), an RNN, or other model trained to predict a plurality of residual predictions based on an initial reconstruction of the image and the original image. As described in more detail herein, the number of sampling steps may be tuned or adjusted to meet a desired perceptual quality-fidelity tradeoff (e.g., at a particular bitrate).

[0043] The codec machine learning model may include an encoder (e.g., an encoder neural network architecture) and / or a decoder (e.g., a decoder neural network architecture). In some cases, the encoder neural network may be part of the same architecture as the decoder neural network. The encoder may process an input image to generate a potential representation (e.g., a potential code) of the input image. In some cases, the encoder may generate the potential representation at least in part by performing quantization and entropy coding (or modeling quantization and entropy coding using neural network operations). The system may store the potential representation or send the potential representation to another system or device.

[0044] The decoder may obtain a latent representation of the image (e.g., a latent representation generated by the encoder or another latent representation of the image generated by an encoder of another system and sent to a system including the decoder), and may process the latent representation to generate an initial reconstructed image representing the input image. A residual model (e.g., a diffusion model, an RNN, etc.) of a machine learning system may be trained to predict a residual representing the difference (e.g., a perceptual difference) between the original input image and the initial reconstructed image. The machine learning system may then combine the predicted residual with the initial reconstructed image to generate a final reconstructed image.

[0045] In some aspects, the residual model may be conditioned on an initial reconstructed image and may be trained to optimize perceptual quality. For example, to predict the residual, the residual model may process the initial reconstruction and noisy data (e.g., a noisy image including random noise) to generate multiple predictions of the residual over multiple sampling steps (e.g., as defined by a sampling schedule). The number of sampling steps may be modified or tuned (e.g., by modifying the sampling schedule) to increase or decrease perceptual quality. For example, the residual between the initial reconstruction and the target image is typically sparse and easier to model. As the number of sampling steps decreases, the complexity and latency of the residual model become less (e.g., resulting in a reduction in floating point operations per pixel, floating point operations per second or FLOPS, or other metrics), at the expense of reduced perceptual quality. For example, the residual model may be tuned with a small number of steps (e.g., 100 steps) to achieve a desired perceptual quality and fidelity at low complexity (relatively small DDPM with a low number of sampling steps). Perceptual quality and fidelity may be determined by the number of steps used to determine the prediction residual at a particular bitrate (e.g., where the bitrate may be set by the encoder of the codec machine learning model).

[0046] In some aspects, the system may determine or modify the number of sampling steps based on input, such as user input, input based on system or device constraints, input based on environmental conditions, and / or input based on other factors. In some examples, the input may include user input received via a user interface that can be used to configure the machine learning system. In one illustrative example, the user input may indicate a specific number of steps that corresponds to a specific perceptual quality-fidelity tradeoff. In another illustrative example, the user input may indicate a desired perceptual quality, a desired fidelity, or a desired perceptual quality-fidelity tradeoff, and based on the input, the system may determine the number of steps required to satisfy the perceptual quality, fidelity, or perceptual quality-fidelity tradeoff.

[0047] In some examples, the inputs may be based on system or device constraints. In one illustrative example, the constraints may include computational constraints of a device implementing the residual-based machine learning system. For example, the device may have limited computational power. The residual-based machine learning system may receive inputs from a processor or other component of the device indicating that the number of sampling steps is to be set to a particular number to reduce computational load. In another illustrative example, the constraints may include power constraints of a device implementing the residual-based machine learning system. The residual-based machine learning system may receive inputs from a processor or other component of the device indicating that the number of sampling steps is to be set to a particular number to the amount of power used by the residual-based machine learning system.

[0048] Thus, modeling the residual allows the system to trade off fidelity and perceived quality, or perceived quality and fidelity, by modifying the sampling schedule (e.g., starting late, stopping early, or anything in between). For example, the resulting sampling process can become an anytime method where sampling can be stopped at any sampling step that achieves the desired tradeoff. In one example, the residual model can be tuned to stop sampling earlier (e.g., after 120 sampling steps), in which case the final reconstructed image will have high fidelity at the expense of visual quality. In another example, the residual model can be tuned to stop sampling later (e.g., after 250 sampling steps), in which case the final reconstructed image will have high visual quality at the expense of a certain fidelity. By using a variable rate base codec, the codec can smoothly traverse the rate-distortion-perception tradeoff at test time. The variable sampling schedule of the residual model enables the codec to have dynamic computational capabilities, so that performance can be tuned according to user or device constraints.

[0049] The initial reconstructed image has good fidelity / distortion properties (based on training the codec machine learning model according to the rate-distortion tradeoff). By training the residual model to optimize the perceptual quality, the number of sampling steps can be adjusted to obtain a residual with the desired perceptual quality. The residual can then be added to the initial reconstructed image to generate a final reconstructed image with an optimized fidelity / distortion, rate, and perceptual quality tradeoff (e.g., an image that looks perceptually good and is highly similar to the original input image compressed at a low bit rate).

[0050] In some aspects, the residual model can be conditioned on the initial reconstructed image and can also be conditioned on an auxiliary latent code generated by an encoder of the codec machine learning model. The auxiliary latent code provides additional information that can help the residual model predict a high-quality residual that indicates the perceptual difference between the original image and the initial reconstructed image. The decoder of the codec machine learning model does not use the auxiliary latent code.

[0051] Residual model-based machine learning systems can provide decoders-decoders (codecs) that perform lossy compression of images, including high-resolution images. Residual model-based machine learning systems can achieve performance that is competitive or better than GAN-based methods and allow for smooth traversal of rate-distortion-perception tradeoffs at test time, whereas GAN-based methods are unable to flexibly trade off distortion for perceptual quality because they require retraining with different adversarial factors. For example, residual model-based machine learning systems can dynamically navigate rate-distortion-perception tradeoffs at test time (or at inference time). Furthermore, while sampling from a diffusion probability model is expensive from a computational perspective, as described above, the sampling cost can be greatly reduced by adjusting the number of sampling steps (e.g., by modifying the sampling schedule).

[0052] In addition, the residual model-based machine learning system solves the challenges of GAN-based generation compression, including the difficulty and complexity of the training pipeline, and the lack of fidelity (low distortion)-perceptual quality trade-off parameters. The residual-based codec also provides control over the fidelity (distortion)-perceptual trade-off (e.g., based on input, device constraints, etc.), which is not provided by current compression solutions (e.g., GAN-based compression solutions).

[0053] One or more machine learning systems described may be trained and used to perform data compression and / or data decompression, such as image, video, and / or audio compression and decompression. The machine learning systems described herein may be trained to perform compression / decompression techniques that produce high-quality data output. The systems and techniques described herein may perform compression and / or decompression of any type of data. For example, in some cases, the systems and techniques described herein may perform compression and / or decompression of image data. As another example, in some cases, the systems and techniques described herein may perform compression and / or decompression of video data. As used herein, the terms "image" and "frame" are used interchangeably to refer to an independent image or frame (e.g., a photo) or a group or sequence of images or frames (e.g., constituting a video or other image / frame sequence). As another example, in some cases, the systems and techniques described herein may perform compression and / or decompression of audio data. For the purpose of simplicity, illustration, and explanation, the systems and techniques described herein are discussed with reference to compression and / or decompression of image data (e.g., images or frames, videos, etc.). However, as described above, the concepts described herein may also be applied to other modalities, such as audio data and any other type of data.

[0054] Additional background and aspects related to the systems and techniques will be described with respect to the accompanying drawings.

[0055] Figure 1 1 is a diagram illustrating an example of an image processing system 100 according to some examples of the present disclosure. In some cases, the image processing system 100 may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more functions described herein. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, or distributed in multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102 and / or the memory block 118.

[0056] The image processing system 100 may include additional processing blocks tailored for specific functions, such as a GPU 104; a DSP 106; a connection block 110, which may include a fifth generation (5G) connection, a fourth generation long term evolution (4G LTE) connection, a Wi-Fi connection, a USB connection, a Bluetooth connection, etc.; and / or a multimedia processor 112, which may, for example, detect and recognize features. In one specific implementation, the NPU 108 is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The image processing system 100 may also include a sensor processor 114, one or more image signal processors (ISPs) 116, and / or a storage device 120. In some examples, the image processing system 100 may be based on the ARM instruction set.

[0057] The image processing system 100 may be part of a computing device or a plurality of computing devices. In some examples, the image processing system 100 may be part of an electronic device (or devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a phone system (e.g., a smartphone, a cellular phone, a conference system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smart watch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a game console, a video streaming device, a drone, an in-vehicle computer, a system on a chip (SOC), an Internet of Things (IoT) device, or any other suitable electronic device.

[0058] Although the image processing system 100 is shown as including certain components, one of ordinary skill will recognize that the image processing system 100 may include more than Figure 1 For example, in some instances, the image processing system 100 may also include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or Figure 1 Other hardware or processing devices not shown in the figure. Fig.13 Illustrative examples of computing devices and hardware components that may be implemented utilizing image processing system 100 are described.

[0059] The image processing system 100 and / or its components may be configured to perform compression and / or decompression (also referred to as encoding and / or decoding, collectively referred to as image decoding) using the machine learning systems and techniques described herein. In some cases, the image processing system 100 and / or its components may be configured to perform image or video compression and / or decompression using the techniques described herein. In some examples, the machine learning system may utilize a deep learning neural network architecture to perform compression and / or decompression of image, video, and / or audio data. By using a deep learning neural network architecture, the machine learning system may improve the efficiency and speed of compression and / or decompression of content on the device. For example, a device using the described compression and / or decompression techniques may effectively compress one or more images using machine learning-based techniques, may send the compressed one or more images to a receiving device, and the receiving device may use the machine learning-based techniques described herein to more effectively decompress the one or more compressed images. As used herein, an image may refer to a still image and / or a video frame associated with a frame sequence (e.g., a video).

[0060] As described above, a neural network is an example of a machine learning system. A neural network may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. A node may include one or more weights for indicating the importance of a node in one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, in which early layers are used to determine simple and low-level features of the input, while later layers establish a hierarchy of more complex and abstract features.

[0061] Deep learning architectures can learn hierarchies of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0062] Deep learning architectures perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0063] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0064] The connections between neural network layers can be fully connected or partially connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In a fully connected neural network 202, a neuron in a first layer may communicate its output to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer will have the same or similar connection pattern, but the connection strengths may have different values ​​(e.g., 210, 212, 214, and 216). The connectivity patterns of the local connections may produce spatially different receptive fields in higher layers because higher layer neurons in a given area may receive inputs that are tuned through training to properties of a limited portion of the total input to the network.

[0065] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is shown. The convolutional neural network 206 can be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful. According to aspects of the present disclosure, the convolutional neural network 206 can be used to perform one or more aspects of video compression and / or decompression.

[0066] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 is shown, which is designed to recognize visual features from an image 226 input from an image capture device 230 (e.g., a vehicle-mounted camera). The DCN 200 of the current example can be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or identifying traffic lights.

[0067] The DCN 200 may be trained by supervised learning. During training, an image (such as an image 226 of a speed limit sign) may be presented to the DCN 200, and a forward pass may then be calculated to produce an output 222. The DCN 200 may include a feature extraction portion and a classification portion. Upon receiving the image 226, a convolutional layer 232 may apply a convolutional kernel (not shown) to the image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel of the convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to the image 226 at the convolutional layer 232. A convolutional kernel may also be referred to as a filter or a convolutional filter.

[0068] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (such as 14x14) is smaller than the size of the first set of feature maps 218 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0069] exist Figure 2D In the example of , the second set of feature maps 220 are convolved to generate a first feature vector 224. In addition, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of the DCN 200 is the probability that the image 226 includes one or more features.

[0070] In this example, the probability of "logo" and "60" in output 222 is higher than the probability of other numbers (such as "30", "40", "50", "70", "80", "90", and "100") in output 222. Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the true value of image 226 (e.g., "logo" and "60"). The weights of DCN 200 can then be adjusted so that the output 222 of DCN 200 is closer to the target output.

[0071] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0072] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient is close to the true error gradient. This approximation may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, new images may be presented to the DCN, and a forward pass through the network may produce an output 222 that may be considered an inference or prediction of the DCN.

[0073] Deep belief network (DBN) is a probabilistic model including multiple layers of hidden nodes. DBN can be used to extract hierarchical representations of training data sets. DBN can be obtained by stacking the layers of restricted Boltzmann machine (RBM). RBM is a type of artificial neural network that can learn probability distributions through a set of inputs. Because RBM can learn probability distributions without information about the categories to which each input should be classified, RBM is generally used for unsupervised learning. Using an unsupervised and supervised hybrid paradigm, the bottom RBM of DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of the input and target class from the previous layer) and can be used as a classifier.

[0074] A deep convolutional network (DCN) is a network that is a convolutional network configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0075] The DCN may be a feed-forward network. In addition, as described above, the connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN may be used for fast processing. For example, the computational burden of the DCN may be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0076] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then the convolutional network trained on the input can be thought of as three-dimensional, with two spatial dimensions along the axis of the image and the third dimension capturing color information. The output of the convolutional connection can be thought of as forming a feature map in a subsequent layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each channel in the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as corrections, max(0,x)). The values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0077] Figure 3 is a block diagram showing an example of a deep convolutional network 350. Based on connections and weight sharing, the deep convolutional network 350 may include multiple different types of layers. Figure 3 As shown, the deep convolution network 350 includes convolution blocks 354A and 354B. Each of the convolution blocks 354A and 354B may be configured with a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0078] The convolution layer 356 may include one or more convolution filters that can be applied to the input data 352 to generate a feature map. Although only two convolution blocks 354A, 354B are shown, the present disclosure is not limited thereto, and any number of convolution blocks (e.g., blocks 354A, 354B) may be included in the deep convolution network 350 according to design preferences. The normalization layer 358 may normalize the output of the convolution filter. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximum pooling layer 360 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0079] For example, the parallel filter bank of the deep convolutional network can be loaded onto the CPU 102 or GPU 104 of the image processing system 100 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the image processing system 100. In addition, the deep convolutional network 350 can access other processing blocks that may be present on the image processing system 100, such as the sensor processor 114.

[0080] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled as “FC1”) and layer 362B (labeled as “FC2”). The deep convolutional network 350 may also include a logistic regression (LR) layer 364. Each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 has weights (not shown) to be updated between them. The output of each of these layers (e.g., 356, 358, 360, 362, 364) may be used as an input to a subsequent layer of these layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) supplied at the first convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is a probability that the input data includes a feature from a set of features.

[0081] Image, audio and video content can be stored in the device and / or can be shared between devices. For example, image, audio and video content can be uploaded to media hosting services and sharing platforms and can be sent to various devices. Recording uncompressed image, audio and video content usually results in large file sizes, which greatly increase with the increase of the resolution of the image, audio and video content. For example, uncompressed 16-bit per channel video recorded in 1080p / 24 format (e.g., a resolution of 1920 pixels wide and 1080 pixels high, where 24 frames are captured per second) can take up 12.4 megabytes per frame, or 297.6 megabytes per second. Uncompressed 16-bit per channel video recorded at 4K resolution at 24 frames per second may take up 49.8 megabytes per frame, or 1195.2 megabytes per second.

[0082] Because uncompressed image, audio, and video content can result in large files that can involve considerable memory for physical storage and considerable bandwidth for transmission, techniques can be utilized to compress such video content. For example, various compression algorithms can be applied to image, audio, and video content in order to reduce the size of image content, and thereby reduce the amount of memory involved in storing the image content and the amount of bandwidth involved in delivering the video content.

[0083] In some cases, image content may be compressed using a priori defined compression algorithms, such as Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG), etc. For example, JPEG is a form of lossy compression based on discrete cosine transform (DCT). For example, a device performing JPEG compression of an image may convert the image to an optimal color space (e.g., a YCbCr color space, including brightness (Y), chrominance-blue (Cb), chrominance-red (Cr)), may downsample the chrominance components by averaging groups of pixels together, and may apply a DCT function to pixel blocks to remove redundant image data, and thereby compress the image data. Compression is based on identifying similar areas within the image and converting these areas to the same color code (based on the DCT function). Video content may also be compressed using a priori defined compression algorithms, such as the Motion Picture Experts Group (MPEG) algorithm, H.264, or a high efficiency video coding algorithm.

[0084] These a priori defined compression algorithms may be able to preserve most of the information in the original image and video content, and may be defined a priori based on ideas from signal processing and information theory. However, while these predefined compression algorithms may be generally applicable (e.g., applicable to any type of image / video content), the compression algorithms may not take into account similarity of content, new resolutions or frame rates of video capture and transmission, non-natural images (e.g., radar images or other images captured via various sensors), etc.

[0085] A priori defined compression algorithms are considered lossy compression algorithms. In lossy compression of an input image (or video frame), the input image cannot be encoded and then decoded / reconstructed in order to reconstruct the exact input image. In contrast, in lossy compression, an approximate version of the input image is generated after decoding / reconstructing the compressed input image. Lossy compression results in a reduction in bit rate, but at the expense of distortion, resulting in artifacts in the reconstructed image. Therefore, there is a rate-distortion tradeoff in lossy compression systems. For some compression methods (e.g., JPEG, BPG, etc.), distortion-based artifacts may take the form of blockiness or other artifacts. In some cases, neural network-based compression can be used, and image data and video data can be compressed with high quality. In some cases, blur and color shift are examples of artifacts.

[0086] Whenever the bitrate is lower than the true entropy of the input data, it may be difficult or impossible to reconstruct the exact input data. However, the fact that distortion / loss is achieved from the compression / decompression of the data does not mean that the reconstructed image or frame will not necessarily have artifacts. In fact, it is possible to reconstruct a compressed image into another similar but different image with high visual quality.

[0087] As previously described, compression and decompression can be performed using one or more machine learning (ML) systems. In some cases, such ML-based systems can provide image and / or video compression that produces high-quality visual output. In some examples, such systems can use deep neural networks such as rate-distortion autoencoders (RD-AE) to perform compression and decompression of content (e.g., image content, video content, audio content, etc.). The deep neural network can include an autoencoder (AE) that maps an image to a potential code space (e.g., including a set of codes z). The potential code space may include a code space used by an encoder and a decoder, and wherein the content has been encoded into code z. The code (e.g., code z) may also be referred to as a potential amount, a potential variable, or a potential representation. The deep neural network may include a probability model (also referred to as a model prior or a code model) that can perform lossless compression on code z from a potential code space. The probability model can generate a probability distribution on a code set z that can represent encoded data based on input data. In some cases, the probability distribution can be represented as (P(z)).

[0088] In some examples, the deep neural network may include an arithmetic decoder that generates a bitstream including compressed data to be output based on a probability distribution P(z) (which itself may be a neural network) and / or a code set z. The bitstream including compressed data may be stored and / or may be sent to a receiving device. The receiving device may use, for example, an arithmetic decoder, a probability (or code) model, and a decoder of the AE to perform an inverse process to decode or decompress the bitstream. A device that generates a bitstream including compressed data may also perform a similar decoding / decompression process when retrieving compressed data from storage.

[0089] As described above, a goal of machine learning-based (eg, neural network-based) data compression is to address the rate-distortion-perception tradeoff. Figure 4A is a diagram illustrating an original image 402, and FIG. 4B to FIG. 4D is a diagram illustrating examples of reconstructed images 404, 406, 408. The reconstructed images 404, 406, and 408 are generated by processing the original image 402 using different machine learning systems optimized for different aspects of rate, distortion, and perception.

[0090] For example, a machine learning system optimized for rate and distortion is used to generate a reconstructed image 404. As a result, the reconstructed image 404 has low distortion (and therefore high fidelity), but low perceptual quality (e.g., blurry). A machine learning system optimized for rate and perception is used to generate a reconstructed image 406. Since distortion is not part of the optimization, the reconstructed image 406 is very different from the original image 402, but has high perceptual quality. A machine learning system optimized for rate, distortion, and perception (e.g., using a diffusion model, RNN, etc.) is used to generate a reconstructed image 408, such as a residual-based machine learning system (or codec) described herein. As a result, the reconstructed image 408 is similar in content to the original image 402 and has high perceptual quality, while allowing the potential code representation of the original image 408 to be compressed to a low bit rate.

[0091] As previously described, systems and techniques are described herein that provide a residual-based machine learning system (e.g., a machine learning codec) for performing compression and decompression of data (e.g., images, video, audio, etc.). The residual-based machine learning system may include a codec machine learning model for compressing an input image into a latent code and generating an initial reconstructed version of the input image based on the latent code. In some cases, the codec machine learning model may perform a rate-distortion neural image (e.g., still image or video frame) compression algorithm (e.g., average scale hyperprior), which may provide good baseline performance in terms of rate-distortion.

[0092] The residual-based machine learning system may also include a residual model for generating a residual representing the difference (e.g., perceptual difference) between the original input image and the initial reconstructed image. As previously described, the residual model may include a diffusion model (e.g., DPM, DDPM, etc.), a recurrent neural network (RNN), or other types of machine learning models. The residual-based machine learning system may then combine the predicted residual with the initial reconstructed image to generate a final reconstructed image.

[0093] Residual models can be trained to improve perceptual quality, which constrains modeling to the residuals. Fewer sampling steps are required compared to the traditional unconditional DPM setting, which may be based on the fact that the residuals are sparser and more normally distributed than the original input images. In some cases, a fast sampling scheme can be implemented for the residual model, which can provide further control over the fidelity-perception tradeoff through multiple sampling steps T and additional sampling time hyperparameters.

[0094] As mentioned above, the residual model can be a diffusion model. In some cases, the diffusion model can be a DPM or a DDPM. In general, the denoised diffusion probability model is a latent variable model, where the T-step latent quantities x1,…,x Tare defined as Markov chains with non-random processes (e.g., Gaussian transitions). They define a forward process that gradually corrupts the original data x0 as follows:

[0095]

[0096] where β t t th The variance of the step can be defined empirically.

[0097] A generative model parameterized by θ (e.g., U-net) can then learn to reverse this process, modeling the transition from the tractable distribution to the data distribution:

[0098]

[0099] Here, the starting noise distribution can be defined as

[0100] In the context of variational inference, training can be viewed as passing q(x 1:T |x0) to approximate the latent variable p θ (x 1:T |x0), and thus p θ Given that the latent quantity is assumed to be a Markov chain with Gaussian transitions in both the target distribution and the variational distribution, the parameterization (Ho et al. (2020)) showed that the training objective can be simplified to the following form:

[0101]

[0102] in:

[0103]

[0104] Here, ∈ θ It is usually parameterized as a neural network that learns according to x t Predict noise. As a rule of thumb, this can be achieved by optimizing a simplified version of Equation 3, which sets ) to improve the sample quality.

[0105] Neural network-based encoders and decoders are systems that learn to compress data from examples. Many existing neural encoders and decoders are compressive autoencoders, which use an autoencoder-like architecture to compress data. e It takes as input a data point x and outputs a quantized latent variable y. Given this latent variable, the neural decoder G d Generate Reconstruction Neural prior or context model G pIt can be used to learn the distribution of the latent variable p(y) so that the latent quantity can be losslessly compressed to the desired -log p(y) bits using an entropy coding algorithm.

[0106] The neural codec can be trained using a rate-distortion objective that consists of two terms:

[0107]

[0108] The rate loss corresponds to the number of bits required to send the quantized latent variable z under the neural prior and is equal to Distortion loss It is reconstruction The distance between x and the true value x, where the mean squared error is a common choice. The tradeoff parameter β determines the desired compression ratio: for high β, few bits should be used, and vice versa. Since it is impractical to have a codec operate at only one bitrate, the codec can be conditioned on this tradeoff parameter, and β can be varied during training.

[0109] DDPM can be conditioned on side information (e.g., target class) by providing input to the generative model. An example of how conditional DDPM can be applied in a data compression setting is to define an encoder that maps the original input data to a compressed (quantized) representation y and condition DDPM directly on this representation. The DDPM loss function Equation 3 can then be used to model p(x|y) using the following objective:

[0110]

[0111] Although p(x|y) has much lower entropy than p(x), which makes the modeling task easier, a perfectly trained DDPM will sample any data point from this distribution. However, in a data compression setting, the data compression system is most interested in data points with high fidelity.

[0112] As mentioned above, to improve the perceptual quality, the residual-based machine learning systems and techniques described herein may first use a decoder G d An initial reconstruction is produced and then enhanced using a residual model (e.g., DPM, DDPM, RNN, etc.). Figure 5 , Figure 6 and Fig. 9 is a diagram illustrating an example of a residual-based machine learning system using a diffusion model (e.g., DDPM) as an illustrative example of a residual model, and is described in more detail below. The decoder of the residual-based machine learning system may include two components, which may be represented as G c ∶={G d ,G ∈},in represents the initial predictor and G∈ A residual model (e.g., a diffusion model) is learned to predict the residual.

[0113] Denote the initial prediction as And the residual is expressed as The residual model can be used to reconstruct the original (or true value) image and the initial (Initial reconstructed image ). For a perfect encoder / decoder pair, the data processing inequality indicates that the initial reconstruction should contain the same information as y, and the conditions for the latent reconstruction or initial reconstruction should be equivalent. In practice, if a suboptimal encoder / decoder is used, then choosing either may have practical advantages. The systems and techniques described in this article enable the residual model to reconstruct the image with the initial is conditioned on, since its spatial dimension is equal to the residual, which can simplify the implementation. In some cases, the residual model can use a U-Net architecture (e.g., Figure 6 As shown), and at each sampling step the image is reconstructed concatenated to its input (e.g., noise data or predictions or potential residuals).

[0114] The machine learning system can be trained in two stages to provide a rate-distortion-perceptual quality tradeoff. For example, in the first stage, the codec machine learning model (which can be referred to as the base codec) can be optimized using the rate-distortion loss in Equation 6. The mean square error or other loss can be used as the distortion loss to ensure that the initial reconstruction In the second stage, the generative model r can be learned θ To train the residual model:

[0115]

[0116] Then, sample from the residual model The system can be allowed to combine the final residuals With the initial reconstructed image To produce the final reconstruction (expressed as ).

[0117] can be used to train residual models without an auxiliary latent volume (e.g., below relative to Figure 5 An example of a loss function for the described diffusion model 526) is as follows:

[0118] L total =L distortion +β·L rate +Υ·L diffusion

[0119] Ldistortion =||x true -RDAE(x true )||

[0120] L rate = -log(z)

[0121]

[0122] Where x represents the image and r represents the residual. In some cases, as described herein, L distortion +βL rate to train the encoder-decoder machine learning model (for example, the following is relative to Figure 5 The codec machine learning model 521 described above can then be used to optimize the L diffusion To train a residual model (e.g., Figure 5 The diffusion model 526 of the residual model is used to freeze the weights of the codec machine learning model during the training of the residual model. Equation (8) provided above illustrates an example of the perceptual quality (P) part of the RDP loss (providing a trade-off between rate, distortion and perceptual quality).

[0123] Can be used to train residual models with auxiliary latent quantities (e.g., as compared to Fig. 9 An example of a loss function for the described diffusion model 526) with auxiliary potential is as follows:

[0124] L total =L distortion +β·L rate +δ·L rate,aux

[0125] L rate,aux = -log(z aux )

[0126] In some cases, similar to the above, L can be used first distortion +β·L rate To train a codec machine learning model (e.g., codec machine learning model 521), and then L diffusion +δ·L rate,aux To train a residual model (e.g., Figure 5 diffusion model 526) and auxiliary encoders (e.g., Figure 5 An auxiliary encoder of encoder 522) (freezing the weights of the encoder-decoder machine learning model during training of the residual model).

[0127] By separating these two stages, the residual-based machine learning system can produce a high-fidelity initial reconstruction. Depending on the application, the residual-based machine learning system can then use the final residual output by the residual model to determine whether to enhance the initial reconstruction and how much to enhance it.

[0128] The sampling process of a diffusion model (e.g., DDPM) typically includes many steps (e.g., thousands of steps), and a large number of sampling steps can be used to obtain optimal performance. The denoising diffusion implicit model (DDIM) provides a training-free sampling scheme based on generalizing the forward process to a non-Markov diffusion process. This choice of forward process results in a "shorter" generated Markov chain, thereby improving sample efficiency. However, the use of this non-Markov generation process comes at the expense of quality, especially for high-resolution images and a large number of sampling steps. In a data compression setting, fidelity may be a more important factor than sample diversity. Additionally, it is also important to keep the computational cost low, which can be achieved by using as few sampling steps as possible. In some cases, the residual model described in this article can use DDIM sampling.

[0129] For a low number of sampling steps, the system can directly predict r0 instead of ∈0. One explanation is that due to and α t →0,∈0 space is amplified in r space. Since it may be beneficial to keep the number of sampling steps low, as described herein, the residual-based machine learning systems and techniques described herein may use a residual model (e.g., DDPM) that directly predicts r.

[0130] Using the residual scheme and direct r prediction, the residual model can be configured or tuned to stop sampling at any time step and use As a final or enhanced reconstruction. For example, stopping sampling early (close to t=T) may result in prediction residuals close to zero, thereby achieving a reconstruction with high fidelity. When stopping sampling late (close to t=0), high perceptual quality can be obtained at the expense of fidelity. This early stopping process may be conceptually similar to using a different noise schedule at test time, in which case adjusting the noise instead of adjusting the time step using a fixed schedule can improve the reconstruction results.

[0131] Various illustrative examples are now described using a diffusion model (e.g., DDPM) as an illustrative example of a residual model, in which case residual-based machine learning systems and techniques may be referred to as diffusion-based machine learning systems and techniques. Although examples are described using a diffusion model as an illustrative example of a residual model, other types of machine learning models or training mechanisms may also be used as residual models, such as RNNs, RD-AEs or other types of VAEs, any combination thereof, and / or other types of machine learning models or training mechanisms.

[0132] Figure 5 is a diagram illustrating an example of a diffusion-based machine learning system 500 for reconstructing an image from an input image. The diffusion-based machine learning system 500 includes a base codec machine learning model 521 (or base codec) and a diffusion model 526. Figure 6 is a diagram illustrating another example of a diffusion-based machine learning system (similar to system 500) that includes a diffusion model 926 implemented using a U-Net architecture.

[0133] In some cases, the codec machine learning model 521 may be implemented as an average scale hyper-prior. Figure 5 As shown, the codec machine learning model 521 includes an encoder 522 and a decoder 524. The encoder 522 obtains an input image 520 (shown as input image x) and processes the input image 520 to generate a latent code 523 (shown as latent code z). The latent code 523 provides a latent representation of the input image 520. In some cases, the decoder 524 may obtain the latent code 523 generated by the encoder 522. In some cases, the decoder 524 may obtain another latent representation of the image (e.g., a latent code) generated by an encoder of another machine learning system. For example, another machine learning system (or a device or system in which another machine learning system is deployed) may be used, such as by (e.g., using Figure 7 The arithmetic decoder 706 of the machine learning system 500 quantizes and / or entropy decodes the output of the encoder of the other machine learning system to generate a bitstream representing the potential representation of the image. The other machine learning system (or device or system) can send the bitstream to the machine learning system 500 for processing by the decoder 524. The decoder 524 can process the potential code 523 (or the potential representation or code from the other machine learning system) to generate an initial reconstructed image 525 (shown as the initial reconstructed image ). In some cases, the machine learning system 500 may perform entropy decoding and / or dequantization on the latent code 523 prior to or as part of generating the initial reconstructed image 525.

[0134] Figure 7 and Figure 8are diagrams illustrating examples of systems 700 and 800 that can be used to implement the codec machine learning model 521, respectively. For example, Figure 7 is a diagram illustrating a system 700 including a transmitting device 710 and a receiving device 720. In some examples, the codec machine learning model 521 may be or may include the transmitting device 710. In some examples, the codec machine learning model 521 may be or may include the receiving device 720.

[0135] In some cases, the sending device 710 and the receiving device 720 may each be referred to as an RD-AE. The sending device 710 may compress image content, and may store the compressed image content and / or send the compressed image content to the receiving device 720 for decompression. The receiving device 720 may decompress the compressed image content, and may output the decompressed image content on the receiving device 720 (e.g., for display, editing, etc.) and / or may output the decompressed image content to other devices connected to the receiving device 720 (e.g., a television, mobile device, or other device). In some cases, the receiving device 720 may become a sending device by compressing the image content (using the encoder 722) and storing and / or sending the compressed image content to another device, such as the sending device 710 (in which case the sending device 710 will become the receiving device). Although the system 700 is described herein in some examples with respect to image compression and decompression, those skilled in the art will appreciate that the system 700 may use the techniques described herein to compress and decompress video, audio, and / or other media content.

[0136] like Figure 7 As illustrated in , the sending device 710 includes an image compression pipeline, and the receiving device 720 includes an image bitstream decompression pipeline. According to aspects of the present disclosure, the image compression pipeline in the sending device 710 and the bitstream decompression pipeline in the receiving device 720 typically use one or more artificial neural networks to compress image content and / or decompress the received bitstream into image content. The image compression pipeline in the sending device 710 includes an autoencoder 701, a code model 707, and an arithmetic decoder 706. In some specific implementations, the arithmetic decoder 706 is optional and may be omitted in some cases. The image decompression pipeline in the receiving device 720 includes an autoencoder 721, a code model 724, and an arithmetic decoder 726. In some specific implementations, the arithmetic decoder 726 is optional and may be omitted in some cases. The autoencoder 701 and the code model 704 of the sending device 710 are Figure 7exemplified as a machine learning system that has been previously trained and is therefore configured to perform operations during speculation or operation of the trained machine learning system. The autoencoder 721 and the code model 724 are also exemplified as a machine learning system that has been previously trained.

[0137] The automatic encoder 701 includes an encoder 702 and a decoder 703. The encoder 702 can perform lossy compression on the received uncompressed image content by mapping pixels in one or more images of the uncompressed image content to a potential code space (including code z). The encoder 702 can be configured to make the potential code z representing the compressed (or encoded) image discrete or binary. In some cases, these codes can be generated based on random perturbation techniques, soft vector quantization, or other techniques that can generate different codes. In some aspects, the automatic encoder 701 can map the uncompressed image to a code with a compressible (low entropy) distribution. The cross entropy of these potential codes can be close to a predefined or learned prior distribution.

[0138] In some examples, the autoencoder 701 may be implemented using a convolutional architecture. For example, in some cases, the autoencoder 701 may be configured as a two-dimensional convolutional neural network (CNN) such that the autoencoder 701 learns a spatial filter for mapping image content to a latent code space. In an example where the system 700 is used to decode video data, the autoencoder 701 may be configured as a three-dimensional CNN such that the autoencoder 701 learns a spatiotemporal filter for mapping a video to a latent code space. In such a network, the autoencoder 701 may encode a video based on a key frame (e.g., an initial frame marking the beginning of a sequence of frames, where subsequent frames in the sequence are described as differences relative to the initial frame in the sequence), a warp (or difference) between a key frame and other frames in the video, and a residual factor. In other aspects, the autoencoder 701 may be implemented as a two-dimensional neural network conditioned on a previous frame, a residual factor between frames, and conditioned by stacking channels or including a recurrent layer.

[0139] The encoder 702 of the autoencoder 701 may receive a first image (at Figure 7The encoder 702 may be implemented as a two-dimensional convolutional network such that the latent code space has a vector at each (x, y) position describing a block of image x centered at that position. The x coordinate may represent a horizontal pixel position in the block of image x, and the y coordinate may represent a vertical pixel position in the block of image x. When decoding video data, the latent code space may have t variables or positions, where the t variable represents a timestamp in the block of video data (in addition to the spatial x and y coordinates). By using two dimensions of horizontal pixel position and vertical pixel position, a vector may describe an image patch in image x.

[0140] The decoder 703 of the autoencoder 701 may then decompress the code z to obtain a reconstruction of the first image x Usually, reconstruction can be an approximation of the uncompressed first image x, but need not be an exact copy of the first image x. In some cases, the reconstructed image Can be output as a compressed image file for storage in the sending device.

[0141] The code model 704 receives a code z representing an encoded image or portion thereof and may generate a probability distribution P(z) over a set of compressed codewords that may be used to represent the code z. In some examples, the code model 704 may include a probabilistic autoregressive generative model. In some cases, the code for which the probability distribution may be generated includes a learned distribution that controls bit allocation based on the arithmetic decoder 706. For example, using the arithmetic decoder 706, a compression code for a first code z may be predicted separately; a compression code for a second code z may be predicted based on the compression code for the first code z; a compression code for a third code z may be predicted based on the compression codes for the first code z and the second code z, and so on. The compression codes typically represent different spatiotemporal blocks of a given image to be compressed.

[0142] In some aspects, the potential code z can be represented as a three-dimensional tensor. The three dimensions of the tensor may include a feature channel dimension, and height and width spatial dimensions (e.g., represented as code z c,w,h ). Each code c,w,h (denoting codes indexed by channel and horizontal and vertical position) can be predicted based on previous codes, which can be fixed-order and theoretically arbitrary-order codes. In some examples, the codes can be generated by analyzing a given image file from beginning to end and analyzing each block in the image in raster scan order.

[0143] The code model 704 can use a probabilistic autoregressive model to learn the probability distribution of the input code z. The probability distribution can be conditioned on its previous value (as described above). In some examples, the probability distribution can be represented by the following formula:

[0144]

[0145] Wherein c is the channel index of all image channels C (such as R, G and B channels, Y, Cb and Cr channels or other channels), w is the width index of the total image frame width W, and h is the height index of the total image frame height H.

[0146] In some examples, the probability distribution P(z) can be predicted by a fully convolutional neural network with causal convolution. In some aspects, the kernel of each layer of the convolutional neural network can be masked so that the convolutional network knows the previous value z when calculating the probability distribution 0:c,0:w,0:h And other values ​​may not be known.In some aspects, the last layer of the convolutional network can include a softmax function that determines the probability that a code in the latent space is applicable to the input value (e.g., the likelihood that a given code can be used to compress a given input).

[0147] Arithmetic decoder 706 uses the probability distribution P(z) generated by code model 704 to generate a bitstream 715 corresponding to a prediction of code z (in Figure 7 . . ”). The prediction of code z may be represented as the code with the highest probability score in the probability distribution P(z) generated over a set of possible codes. In some aspects, the arithmetic decoder 706 may output a variable length bitstream based on the accuracy of the prediction of code z and the actual code z generated by the autoencoder 701. For example, if the prediction is accurate, the bitstream 715 may correspond to a short codeword, while as the magnitude of the difference between code z and the prediction of code z increases, the bitstream 715 may correspond to a longer codeword.

[0148] In some cases, the bitstream 715 may be output by the arithmetic decoder 706 for storage in a compressed image file. The bitstream 715 may also be output for transmission to a requesting device (e.g., a receiving device 720, such as Figure 7 Typically, the bitstream 715 output by the arithmetic decoder 706 may losslessly encode z so that z may be accurately recovered during a decompression process applied to the compressed image file.

[0149] The bit stream 715 generated by the arithmetic decoder 706 and transmitted from the transmitting device 710 may be received by the receiving device 720. Transmission between the transmitting device 710 and the receiving device 720 may occur using any of a variety of suitable wired or wireless communication techniques. Communication between the transmitting device 710 and the receiving device 720 may be direct or may be performed through one or more network infrastructure components (e.g., base stations, relay stations, mobile stations, network hubs, routers, and / or other network infrastructure components).

[0150] As illustrated, the receiving device 720 may include an arithmetic decoder 726, a code model 724, and an autoencoder 721. The autoencoder 721 includes an encoder 722 and a decoder 723. For a given input, the decoder 723 may generate the same or similar output as the decoder 703. Although the autoencoder 721 is illustrated as including the encoder 722, the encoder 722 need not be used during the decoding process to obtain a reconstruction from the code z received from the sending device 710. (e.g., an approximation of the original image x compressed at the sending device 710). In some examples, the decoder 723 may include a decoder of a codec machine learning model (e.g., Figure 5 decoder 524) and a residual model (e.g., Figure 5 In such examples, the reconstruction Can include Figure 5 The final reconstructed image 529. In addition, in such examples, the receiving device 720 may receive an input defining the number of sampling steps for the residual model (e.g., Figure 5 The sampling step input 539).

[0151] The received bitstream 715 may be input into an arithmetic decoder 726 to obtain one or more codes z from the bitstream. The arithmetic decoder 726 may extract the decompressed code z based on a probability distribution P(z) generated by the code model 724 over a set of possible codes and information associating each generated code z with the bitstream. Given a received portion of the bitstream and a probabilistic prediction of the next code z, the arithmetic decoder 726 may generate a new code z as it was encoded by the arithmetic decoder 706 at the transmitting device 710. Using the new code z, the arithmetic decoder 726 may make probabilistic predictions for consecutive codes z, read additional portions of the bitstream, and decode consecutive codes z until the entire received bitstream is decoded. The decompressed code z may be provided to a decoder 723 in the autoencoder 721. The decoder 723 decompresses the code z and outputs an approximation of the image content x. (which may be referred to as a reconstructed or decoded image.) In some cases, an approximation of the content x may be stored for later retrieval. In some cases, the approximation of content x The received data may be recovered by the receiving device 720 and displayed on a screen communicatively coupled or integrated with the receiving device 720 .

[0152] As described above, the automatic encoder 701 and the code model 704 of the transmitting device 710 are Figure 7 , which is illustrated as a previously trained machine learning system. In some aspects, the autoencoder 701 and the code model 704 can be trained together using image data. For example, the encoder 702 of the autoencoder 701 can receive a first training image n as input, and can map the first training image n to a code z in a latent code space. The code model 704 can use a probabilistic autoregressive model (similar to the above-mentioned techniques) to learn the probability distribution P(z) of the code z. The arithmetic decoder 706 can use the probability distribution P(z) generated by the code model 704 to generate an image bitstream. Using the bitstream and the probability distribution P(z) from the code model 704, the arithmetic decoder 706 can generate the code z, and can output the code z to the decoder 703 of the autoencoder 701. The decoder 703 can then decompress the code z to obtain a reconstruction of the first training image n. (Reconstruction is an approximation of the uncompressed first training image n).

[0153] In some cases, a backpropagation engine used during training of the transmitting device 710 may perform a backpropagation process to tune the parameters (e.g., weights, biases, etc.) of the neural networks of the autoencoder 701 and the code model 704 based on one or more loss functions. In some cases, the backpropagation process may be based on a stochastic gradient descent technique. Backpropagation may include a forward pass, one or more loss functions, a backward pass, and a weight (and / or other parameter) update. The forward pass, loss function, backward pass, and parameter update may be performed for one training iteration. The process may be repeated for a certain number of iterations for each set of training data until the weights and / or other parameters of the neural network are accurately tuned.

[0154] For example, the autoencoder 701 may compare n and To determine the first training image n and the reconstructed first training image The loss between n and n (e.g., represented by a distance vector or other difference). The loss function can be used to analyze the error in the output. In some examples, the loss can be based on maximum likelihood. As an illustrative example of output, the loss function Loss = D + beta * R can be used to train the neural network system of the autoencoder 701 and the code model 704, where R is the rate, D is the distortion, * represents the multiplication function, and beta is a tradeoff parameter that is set to a value that defines the bit rate. In another example, the loss function can be expressed as shown in equation (6) above, reproduced as follows:

[0155]

[0156] In yet another example, the loss function can be expressed as Any of the above loss functions may be used to train the neural network system of the autoencoder 701 and the code model 704. In some cases, such as when other training data is used, other loss functions may be used. An example of another loss function includes the mean squared error (MSE), defined as MSE is calculated as one-half times the sum of the actual answer minus the square of the predicted (output) answer.

[0157] Based on the determined loss (e.g., distance vector or other difference) and using a back-propagation process, parameters (e.g., weights, biases, etc.) of the neural network system of the autoencoder 701 and the code model 704 may be adjusted (effectively adjusting the mapping between the received image content and the latent code space) to reduce the loss between the input uncompressed image and the compressed image content generated as output by the autoencoder 701.

[0158] For the first training image, the loss (or error) may be high because the actual output value (reconstructed image) may be very different from the input image. The goal of training is to minimize the amount of loss for the predicted output. The neural network can perform a backward pass by determining which nodes of the neural network (with corresponding weights) contribute most to the loss of the neural network, and the weights (and / or other parameters) can be adjusted to reduce and ultimately minimize the loss. The derivative of the loss with respect to the weights (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weights that contribute most to the loss of the neural network. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Where w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, while a lower value indicates a smaller weight update.

[0159] The neural network system of the autoencoder 701 and the code model 704 may continue to be trained in this manner until the desired output is obtained. For example, the autoencoder 701 and the code model 704 may repeat the backpropagation process to minimize or otherwise reduce the difference between the input image n and the reconstructed image resulting from the decompression of the generated code z. The difference between.

[0160] The autoencoder 721 and code model 724 may be trained using techniques similar to those described above for training the autoencoder 701 and code model 704 of the transmitting device 710. In some cases, the autoencoder 721 and code model 724 may be trained using the same or different training data sets used to train the autoencoder 701 and code model 704 of the transmitting device 710.

[0161] exist Figure 7 In the example shown, the rate-distortion autoencoder (sending device 710 and receiving device 720) is trained and runs inference based on the bit rate. In some implementations, the rate-distortion autoencoder can be trained at multiple bit rates to allow generation and output of high-quality reconstructed images or video frames (e.g., without or with limited artifacts due to distortion relative to the input image) when different amounts of information are provided in the latent code z.

[0162] In some embodiments, a set of continuous potentials (e.g., which can convey a lot of information) and corresponding quantized discrete potentials (e.g., which contain less information) can be used. After training the RD-AE model, the auxiliary dequantization model can be trained. In some cases, when using RD-AE, only the discrete potential is transmitted, and the continuous potential is inferred from the discrete potential using the auxiliary dequantization model at the decoder.

[0163] Although system 700 is shown as including certain components, one of ordinary skill will appreciate that system 700 may include more than Figure 7 For example, the transmitting device 710 and / or the receiving device 720 of the system 700 may also include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices and / or Figure 7 Other hardware or processing devices not shown. Figure 7 The components shown and / or other components of system 700 may be implemented using one or more computing or processing components. The one or more computing components may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or an image signal processor (ISP). Fig.13Illustrative examples of computing devices and hardware components that may be implemented with system 1600 are described.

[0164] System 700 may be part of or implemented by a single computing device or multiple computing devices. In some examples, sending device 710 may be part of a first device and receiving device 720 may be part of a second computing device. In some examples, sending device 710 and / or receiving device 720 may be included as part of an electronic device (or multiple devices) such as: a telephone system (e.g., a smartphone, a cellular phone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, a video streaming device, a SOC, an IoT (Internet of Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), or any other suitable electronic device. In some cases, system 700 may be implemented by Figure 1 The system 700 is implemented by the image processing system 100 shown. In other cases, the system 700 may be implemented by one or more other systems or devices.

[0165] Figure 8 is an example that can be used to implement Figure 5 Schematic diagram of another example neural network compression system 800 for a codec machine learning model 521. In some examples, the neural network compression system 800 may include a RD-AE system. Figure 8 , neural network compression system 800 includes encoder 802, arithmetic encoder 808, arithmetic decoder 812, and decoder 814. In some cases, encoder 802 and / or decoder 814 may be the same as encoder 402 and / or decoder 403, respectively. In other cases, encoder 802 and / or decoder 814 may be different from encoder 702 and / or decoder 703, respectively.

[0166] Encoder 802 may receive image 801 (image x) as input and may map and / or convert image 801 (image x) to a latent code 804 (latent z) in a latent code space. Image 801 may represent a still image and / or a video frame associated with a frame sequence (e.g., a video). In some cases, encoder 802 may perform a forward pass to generate latent code 804. In some examples, encoder 802 may implement a learnable function. In some cases, encoder 802 may implement a latent code 804. Parameterized learnable function. For example, encoder 802 may implement the function In some examples, the learnable function need not be shared with or known by the decoder 814 .

[0167] Arithmetic encoder 808 may generate bitstream 810 based on latent code 804 (latent quantity z) and latent prior 806. In some examples, latent prior 806 may implement a learnable function. In some cases, latent prior 806 may implement a learnable function parameterized by ψ. For example, latent prior 806 may implement function p ψ (z). Latent prior 806 may be used to convert latent code 804 (latent quantity z) into bitstream 810 using lossless compression. Latent prior 806 may be shared and / or available at both the sender (e.g., encoder 802 and / or arithmetic encoder 808) and the receiver (e.g., arithmetic decoder 812 and / or decoder 814).

[0168] The arithmetic decoder 812 may receive the encoded bitstream 810 from the arithmetic encoder 808 and use the potential prior 806 to decode the potential code 804 (potential quantity z) in the encoded bitstream 810. The decoder 814 may decode the potential code 804 (potential quantity z) into an approximate reconstructed image 816 (reconstructed image 816). ). In some cases, decoder 814 may implement a learnable function parameterized by θ. For example, decoder 814 may implement the function p θ (x|z). The learnable function implemented by the decoder 814 may be shared and / or available at both the sender (eg, the encoder 802 and / or the arithmetic encoder 808) and the receiver (eg, the arithmetic decoder 812 and / or the decoder 814).

[0169] The neural network compression system 800 can be trained to minimize rate distortion. In some examples, the rate reflects the length of the bitstream 810 (bitstream b), while the distortion reflects the length of the image 801 (image x) and the reconstructed image 816 (reconstructed ) between the rate-distortion ratio. The parameter β may be used to train the model for a specific rate-distortion ratio. In some examples, the parameter β may be used to define and / or implement a certain tradeoff between rate and distortion. In some examples, a loss according to equation (6) above, or any other suitable loss function, may be used to train the neural network compression system 800.

[0170] Return to Figure 5 , and as described above, the encoder 522 of the codec machine learning model 521 takes the image 520 as input (image x) and compresses it into a latent code 523 (latent quantity z), and the decoder 524 of the codec machine learning model 521 decodes the latent code 523 into an initial reconstructed image 525 (prediction The initial reconstructed image 525 (initial prediction) can be used to adjust the diffusion-based model 526, which can be expressed as (where θ represents its parameters), such as Figure 5 shown.

[0171] The diffusion model 526 can receive the initial reconstructed image 525 from the decoder 524 of the codec machine learning model 521. As described above, the diffusion model 526 is conditioned on the initial reconstructed image 525. For example, the diffusion model 526 can be trained to predict a residual 527 (referred to as a prediction residual 527) that represents a difference (e.g., a perceptual difference) between the original image 520 and the initial reconstructed image 525. In some aspects, the diffusion model can be conditioned on the initial reconstructed image and can also be conditioned on an auxiliary latent code generated by the encoder of the codec machine learning model. Fig. 9 is a diagram illustrating an example of a machine learning system that includes a diffusion model 926 conditioned on an initial reconstructed image 525 and auxiliary latent codes 930 generated by an encoder 522 of a codec machine learning system 521. The auxiliary latent codes 930 provide additional information that may assist the diffusion model in predicting a high-quality residual that indicates a perceptual difference between the original image 520 and the initial reconstructed image 525. For example, the auxiliary latent codes 930 provide the diffusion model 526 with access to additional information about the input image 520 beyond the information contained in the initial reconstructed image 525. The auxiliary latent codes 930 may be used only by the diffusion model 526 (it is not used by the decoder 524 of the codec machine learning model 521). Fig. 9 In the example of FIG. 5 , the initial reconstructed image 525 and the auxiliary latent code 930 may be used to adjust the diffusion-based model 926, which may be expressed as like Fig. 9 shown.

[0172] To predict the residual, the diffusion model 526 may process the initial reconstructed image 525 and the noise data to generate multiple predictions of the residual over multiple sampling steps (eg, as defined by a sampling schedule), Figure 5 Indicated as r T ,r T-1 , until a final step r0, where a final prediction residual 527 is determined at the final step r0. In some cases, the noise data may include a noise image that includes random noise. In some cases, sampling may be performed according to a DDIM sampling strategy. By iterative sampling, using a relatively low number of sampling steps T (e.g., 100 to 500 steps), the diffusion model 526 generates a residual r0 that can be added to the initial reconstructed image 525 to generate a final reconstructed image 529 (shown as the final reconstructed image ). For example, as shown, the machine learning system 500 may combine the final prediction residual 527 with the initial reconstructed image 525 (shown as an addition operation) to generate a final reconstructed image 529.

[0173] Fig.10 1000 is a diagram illustrating an example operation of a diffusion model 526 for predicting a residual representing a difference between an input image (e.g., input image 520) and an initial reconstructed image 1025 (e.g., similar to the initial reconstructed image 525). At a first sampling step (shown as step 0) of the diffusion model 526, the initial reconstructed image 1025 is combined (e.g., concatenated) with initial noise data 1042. The initial noise data 1042 may include an image of random noise. At the first sampling step (step 0), the diffusion model 526 may process the combined reconstructed image 1025 and the initial noise data 1042 to predict a residual (e.g., a potential representation of the prediction residual).

[0174] At a subsequent sampling step, the diffusion model 526 may combine (e.g., concatenate) the initial reconstructed image 525 with the prediction residual predicted in the previous sampling step and process the newly combined data to predict another residual. Fig.10 As shown, at the subsequent sampling step (shown as step 50), the diffusion model 526 can obtain a potential quantity 1044, which is a potential representation of the prediction residual predicted at the sampling step 49 of the diffusion model 526. At the final sampling step (shown as step 100), the diffusion model can generate a final prediction residual 1027. Figure 5 Similar to what has been described, the final prediction residual 1027 may be combined with the initial reconstructed image 1025 to generate a final reconstructed image 1029 .

[0175] In some aspects, the number of sampling steps of the diffusion model 526 can be modified to increase or decrease the perceived quality of the final reconstructed image 529. As the number of sampling steps decreases, the complexity and latency of the diffusion model become smaller (e.g., resulting in a decrease in floating point operations per pixel, floating point operations per second or FLOPS, or other metrics), at the expense of reduced fidelity. For example, the diffusion model can be tuned with a small number of steps (e.g., 100 steps) to achieve a desired perceptual quality and fidelity at low complexity (relatively small DDPM with a low number of sampling steps).

[0176] In some cases, the machine learning system 500 may receive an input indicating the number of sampling steps to use for the diffusion model 526. For example, Figure 5 As shown, the residual model 526 may receive a sampling step input 539 indicating the number of sampling steps used to predict the prediction residual 527. Based on the sampling step input 539, the diffusion model 526 may determine or modify the number of sampling steps. The sampling step input 539 may include user input, input based on system or device constraints, input based on environmental conditions, any combination thereof, and / or other input.

[0177] In some examples, the input may include user input received via a user interface that may be used to configure the machine learning system. In some cases, a user may provide input to a graphical user interface to modify a sampling schedule for the diffusion model 526 (providing a user-friendly knob to control the perception-distortion tradeoff), which may increase or decrease the number of sampling steps for the diffusion model 526 (e.g., from 250 to 100 sampling steps, from 50 to 100 sampling steps, etc.). In one illustrative example, the user input may indicate a particular number of steps that corresponds to a particular perceptual quality-fidelity tradeoff. In another illustrative example, the user input may indicate a desired perceptual quality, a desired fidelity, or a desired perceptual quality-fidelity tradeoff, and based on the input, the system may determine the number of steps required to satisfy the perceptual quality, fidelity, or perceptual quality-fidelity tradeoff.

[0178] In some examples, the inputs may be based on system or device constraints. In one illustrative example, the constraints may include computational constraints of a device implementing the residual-based machine learning system. For example, the device may have limited computational power. The residual-based machine learning system may receive inputs from a processor or other component of the device indicating that the number of sampling steps is to be set to a particular number to reduce computational load. In another illustrative example, the constraints may include power constraints of a device implementing the residual-based machine learning system. The residual-based machine learning system may receive inputs from a processor or other component of the device indicating that the number of sampling steps is to be set to a particular number to the amount of power used by the residual-based machine learning system.

[0179] In some cases, sampling may be stopped at any point where the desired tradeoff is achieved. In one illustrative example, the diffusion model may be tuned to stop sampling earlier (e.g., after 120 sampling steps), in which case the final reconstructed image will have high fidelity at the expense of visual quality. In another illustrative example, the diffusion model may be tuned to stop sampling later (e.g., after 250 sampling steps), in which case the final reconstructed image will have high visual quality at the expense of a certain fidelity. Using such a variable sampling rate for the diffusion model 526 may allow the machine learning system 500 to smoothly traverse the rate-distortion-perception tradeoff at test time. Variable sampling also enables the codec to have dynamic computational capabilities, so that performance can be tuned according to user or device constraints.

[0180] In some cases, the machine learning system 500 may be trained such that the codec machine learning model 521 is trained (e.g., as compared to Figure 7) and the parameters (e.g., weights, biases, etc.) are frozen (kept fixed) during the training of the diffusion model 526, in which case the diffusion model 526 can only perform enhancement. In other cases, the entire machine learning system 500 can be trained end-to-end.

[0181] Using the residual-based machine learning systems and techniques described herein, very high perceptual quality reconstructions of output samples (e.g., reconstructed images) with good fidelity and rate characteristics can be generated. The residual model-based machine learning system can provide a codec that performs lossy compression of images (including high-resolution images). The residual model-based machine learning system can achieve performance that is competitive or better than that of GAN-based methods and allows for smooth traversal of rate-distortion-perceptual tradeoffs at test time, while GAN-based methods cannot flexibly trade off distortion for perceptual quality because they need to be retrained with different adversarial factors.

[0182] Fig.11 1 is a diagram illustrating an example of a generative adversarial network (GAN) 1100 for reconstructing an image from an input image. GAN 1100 is a generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that are reasonably likely to have come from the original data set. A GAN may include two neural networks operating together. One of these neural networks is called a generative neural network or generator 1150 (sometimes represented as G(z)). Generator 1150 may include an encoder 1153 and a decoder 1155. Encoder 1153 may extract features from an input image x and output a latent code 1157. Decoder 1155 may receive latent code 1157 and may process latent code 1157 to generate a synthetic output 1151 (e.g., a reconstructed image ). The other neural network is called a discriminator neural network or discriminator 1152 (sometimes represented as D(X)). Discriminator 1152 evaluates the authenticity of output 1151 (whether the output came from the original data set (such as the training data set) or was generated by the generator).

[0183] The training inputs and outputs may include images as illustrative examples. The generator 1150 is trained to attempt to fool the discriminator 1152 into determining that the synthetic images generated by the generator 1150 are real images from a data set. The training process continues, and the generator 1150 becomes better at generating synthetic images that look like real images. The discriminator 1152 continues to find flaws in the synthetic images, and the generator 1150 figures out what the discriminator 1152 is looking at to determine the flaws in the image. Once the GAN 1100 is trained, the generator 1150 is able to generate realistic images that the discriminator 1152 cannot distinguish from real images.

[0184] Unlike GAN 1100, the residual-based machine learning system described in this paper (e.g., Figure 5 The diffusion-based machine learning system 500 described herein can optimize the perceptual quality optimized by the discriminator / adversarial loss (e.g., the distortion-perception tradeoff is fixed after training). The residual-based machine learning system described herein can also operate in image space rather than residual space. The GAN-based system also uses a very large decoder (e.g., 120 million (M) parameters) compared to the residual-based machine learning system described herein (e.g., 30M or fewer parameters).

[0185] The residual model-based machine learning systems described herein can also dynamically navigate rate-distortion-perception tradeoffs at test time (or at inference time). In addition, while sampling from a diffusion probability model is expensive from a computational perspective, as described above, the sampling cost can be greatly reduced by adjusting the number of sampling steps (e.g., by modifying the sampling schedule). For example, as described above, unlike other systems (e.g., GAN-based systems and methods), variable sampling of the diffusion model 526 enables decoding at multiple bit rates using the same codec (machine learning system 500).

[0186] Fig.12 1 is a flow chart illustrating an example process 1200 for processing media data. At block 1202, process 1200 may include obtaining a potential representation of an image. In one illustrative example, the image may include Figure 5 520, and the latent representation of the image 520 may include latent code 523. In some aspects, process 1200 may include obtaining an image and processing the image using an encoder of a machine learning model to generate a latent representation of the image. In an illustrative example, the encoder may include Figure 5 Encoder 524.

[0187] At block 1204, process 1200 may include processing a latent representation of an image using a decoder of a machine learning model to generate an initial reconstructed image. In one illustrative example, the decoder may include Figure 5 The decoder 524 of FIG. 1 may include the decoder 524 of FIG. 1 , and the initial reconstructed image may include the initial reconstructed image 525. In another example, the decoder may include Figure 7 Decoder 723.

[0188] At block 1206, process 1200 may include processing the initial reconstructed image and the noise data using a residual model to predict a plurality of predictions (e.g., a sequence of predictions) of the residual over a plurality of sampling steps. The residual represents the difference between the image and the initial reconstructed image. In one illustrative example, the residual model may include a diffusion model, such as Figure 5 Diffusion model 526 or Fig. 9 In another illustrative example, the residual model may include a recurrent neural network (RNN). In some cases, a prediction of a plurality of prediction residuals may be determined at each of a plurality of sampling steps. In one example, the plurality of sampling steps may include Figure 5 and / or Fig. 9 The steps shown in T 、r T-1 In another example, the plurality of sampling steps may include Fig.10 The sampling steps shown are 0 to 100.

[0189] At block 1208, process 1200 may include obtaining a final residual representing the difference between the image and the initial reconstructed image from the multiple predictions of the residual. For example, the final residual may include Figure 5 (or Fig. 9 ) shown in the sampling step r0 or Fig.10 The residual determined at sampling step 100 is shown.

[0190] In some aspects, as described herein, the number of sampling steps is adjustable to obtain a tradeoff between the rate of potential representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image. For example, process 1200 may include receiving an input indicating the number of sampling steps and determining the number of sampling steps based on the input. In one example, the input includes a user input received via a user interface, as described herein. In another example, the input is based on constraints associated with a computing device (e.g., a computing device including a decoder and a residual model), such as a computational constraint of the computing device, a power constraint of the computing device, a latency constraint associated with the computing device, any combination thereof, and / or other constraints. In another example, process 1200 may include determining the number of sampling steps based on a target perceptual quality and fidelity tradeoff for the final reconstructed image (e.g., indicated by user input, determined by a computing device, etc.).

[0191] In some cases, the residual model is conditioned on the initial reconstructed image. For example, process 1200 may include using the residual model to process the initial reconstructed image and the noise data at a first sampling step to predict a first prediction of the residual. Process 1200 may also include using the residual model to process the initial reconstructed image and the first prediction of the residual at a second sampling step to predict a second prediction of the residual. Process 1200 may continue to generate predictions of the residual until a final sampling step of the configuration is reached. Process 1200 may include generating predictions of the residual at a third sampling step (e.g., a final sampling step, such as Figure 5 (or Fig. 9 ) shown in the sampling step r0 or Fig.10The residual model is used at the sampling step 100 shown in , or other sampling steps) to process the initial reconstructed image and previous predictions of the residual (e.g., the residual prediction determined at the sampling step immediately before the final sampling step, the second prediction of the residual in the above example, or other residual predictions during the sampling step) to predict the final residual.

[0192] At block 1210, process 1200 may include combining the initial reconstructed image and the residual to generate a final reconstructed image. In one illustrative example, the final reconstructed image may include Figure 5 and / or Fig. 9 The final reconstructed image 529.

[0193] In some cases, the residual model is further conditioned on an auxiliary latent representation of the image. For example, process 1200 may include obtaining an auxiliary latent representation of the image. In one illustrative example, the auxiliary representation may include Fig. 9 The auxiliary latent code 930. The process 1200 may also include processing the initial reconstructed image, the noise data, and the auxiliary latent representation of the image using the residual model to predict a plurality of predictions of the residual.

[0194] In some examples, the processes described herein (e.g., process 1200 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, process 1200 can be performed by Fig.13 The computing system 1300 shown in FIG. 1 is used to execute the above-mentioned operations.

[0195] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein (including process 1100, process 1200, process 1300, process 1400, process 1500, and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0196] Components of a computing device may be implemented in circuits. For example, a component may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0197] The process 1200 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, and data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0198] Additionally, process 1200 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0199] Fig.13 is a diagram illustrating an example of a system for implementing certain aspects of the disclosed technology. Specifically, Fig.13 An example of a computing system 1300 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using connection 1305. Connection 1305 may be a physical connection using a bus, or a direct connection into processor 1310, such as in a chipset architecture. Connection 1305 may also be a virtual connection, a networked connection, or a logical connection.

[0200] In some embodiments, computing system 1300 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.

[0201] The example system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components including system memory 1315, such as read only memory (ROM) 1320 and random access memory (RAM) 1325, to the processor 1310. The computing system 1300 may include a cache 1312 of high-speed memory directly connected to, close to, or integrated as part of the processor 1310.

[0202] Processor 1310 may include any general purpose processor and hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, that are configured to control processor 1310 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0203] To enable user interaction, computing system 1300 includes input device 1345, which may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 1300 may also include output device 1335, which may be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with computing system 1300. Computing system 1300 may include communication interface 1340, which may generally govern and manage user input and system output.

[0204] The communication interface can perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including using audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof.

[0205] The communication interface 1340 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1300 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's Beidou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no limitation to operating on any particular hardware arrangement, and thus the underlying features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0206] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, memory storage, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, or a memory card. card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a micro / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), other memory chips or cartridges, and / or combinations thereof.

[0207] Storage device 1330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1310, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1310, connection 1305, output device 1335, etc. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media that can store data and does not include carrier waves and / or transient electronic signals that propagate wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer readable media may store thereon code and / or machine executable instructions, which may represent a process, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0208] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0209] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For the sake of clarity of explanation, in some cases, the present technology may be presented as including a separate functional block, which includes a device, device assembly, step or routine in a method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0210] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams or block diagrams. Although flow charts may describe operations as sequential processes, many operations in the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process may be terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. The process may correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0211] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that enable or configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions in other ways. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0212] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of additional examples, such functionality may also be implemented on circuit boards between different chips or different processes executed on a single device.

[0213] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0214] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0215] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0216] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0217] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0218] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0219] Claim language or other language that recites "at least one processor, the at least one processor configured to," "at least one processor configured to," "one or more processors, the one or more processors configured to," "one or more processors configured to," etc., indicates that a processor or multiple processors (in any combination) can perform the associated operations. For example, claim language that recites "at least one processor, the at least one processor configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or multiple processors are each tasked with a particular subset of operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language that recites "at least one processor, the at least one processor configured to: X, Y, and Z" may mean that any single processor can perform only at least a subset of operations X, Y, and Z.

[0220] In the case of one or more elements that perform a function (e.g., steps of a method), one element may perform all functions, or more than one element may perform functions together. When more than one element performs a function together, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed entirely by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, in the case of one or more elements that are configured to cause another element (e.g., a device) to perform a function, one element may be configured to cause another element to perform all functions, or more than one element may be configured together to cause another element to perform these functions.

[0221] In the case of referring to an entity (e.g., any entity or device described herein) that performs a function or is configured to perform a function (e.g., a step of a method), the entity may be configured to cause one or more elements to perform a function (individually or collectively). One or more components of an entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of a function, and / or any combination thereof. When referring to an entity that performs a function, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform these functions collectively. When an entity is configured to cause more than one component to perform a function collectively, each function does not need to be performed by each of those components (e.g., different functions may be performed by different components) and / or each function does not need to be performed as a whole by only one component (e.g., different sub-functions of different component executable functions).

[0222] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0223] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the above methods, algorithms, and / or operations when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0224] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0225] Exemplary embodiments of the present disclosure include:

[0226] Aspect 1. A method for processing image data, the method comprising: obtaining a latent representation of an image; processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict multiple predictions of residuals over multiple sampling steps, wherein the residuals represent the difference between the image and the initial reconstructed image; obtaining a final residual representing the difference between the image and the initial reconstructed image from the multiple predictions of the residuals; and combining the initial reconstructed image and the residual to generate a final reconstructed image.

[0227] Aspect 2. A method according to aspect 1, wherein the number of sampling steps is adjustable to obtain a compromise between the rate of the latent representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image.

[0228] Aspect 3. The method according to any one of Aspects 1 or 2 further comprises: determining the number of sampling steps.

[0229] Aspect 4. The method according to aspect 3 further comprises: determining the number of sampling steps based on settings.

[0230] Aspect 5. The method according to aspect 3 further comprises: configuring the setting based on input.

[0231] Aspect 6. The method according to aspect 5 further comprises: receiving the input.

[0232] Aspect 7. A method according to any one of aspects 5 or 6, wherein the input comprises user input received via a user interface.

[0233] Aspect 8. A method according to any one of Aspects 5 to 7, wherein the input is based on constraints associated with a computing device, the constraints comprising at least one of a computing constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device.

[0234] Aspect 9. The method according to any one of aspects 1 to 8, further comprising: determining the number of sampling steps based on a target perceptual quality and fidelity tradeoff of the final reconstructed image.

[0235] Aspect 10. A method according to any one of aspects 1 to 9, wherein the residual model is conditioned on the initial reconstructed image.

[0236] Aspect 11. The method according to aspect 10, wherein the residual model is further conditioned on an auxiliary latent representation of the image.

[0237] Aspect 12. A method according to any one of Aspects 1 to 11, the method further comprising: using the residual model to process the initial reconstructed image and the noise data at a first sampling step to predict a first prediction of the residual; and using the residual model to process the initial reconstructed image and the first prediction of the residual at a second sampling step to predict a second prediction of the residual.

[0238] Aspect 13. The method according to aspect 12, further comprising: using the residual model at the third sampling step to process the initial reconstructed image and a previous prediction of the residual to predict the final residual.

[0239] Aspect 14. An apparatus according to aspect 13, wherein the previous prediction of the residual is the second prediction of the residual, or wherein the third sampling step is a final sampling step and the previous prediction of the residual is a residual prediction determined at a sampling step immediately before the final sampling step.

[0240] Aspect 15. A method according to any one of Aspects 1 to 14, the method further comprising: obtaining the image; and processing the image using an encoder of the machine learning model to generate the latent representation of the image.

[0241] Aspect 16. A method according to any one of Aspects 1 to 15, the method further comprising: obtaining an auxiliary latent representation of the image; and using the residual model to process the initial reconstructed image, the noise data and the auxiliary latent representation of the image to predict the multiple predictions of the residual.

[0242] Aspect 17. The method according to any one of aspects 1 to 16, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

[0243] Aspect 18. A method according to any one of aspects 1 to 17, wherein the plurality of predictions comprises a sequence of predictions.

[0244] Aspect 19. A device for processing image data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory and configured to: obtain a latent representation of an image; process the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; process the initial reconstructed image and noise data using a residual model to predict multiple predictions of residuals over multiple sampling steps, wherein the residuals represent the difference between the image and the initial reconstructed image; obtain a final residual representing the difference between the image and the initial reconstructed image from the multiple predictions of the residuals; and combine the initial reconstructed image and the residual to generate a final reconstructed image.

[0245] Aspect 20. The apparatus according to aspect 19, wherein the number of sampling steps is adjustable to obtain a compromise between the rate of the latent representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image.

[0246] Aspect 21. An apparatus according to any one of Aspects 19 or 20, wherein the at least one processor is configured to: determine a number of sampling steps.

[0247] Aspect 22. The apparatus according to aspect 21, wherein the at least one processor is configured to: determine the number of sampling steps based on a setting.

[0248] Aspect 23. The apparatus according to aspect 22, wherein the at least one processor is configured to: configure the setting based on input.

[0249] Aspect 24. The apparatus according to aspect 23, wherein the at least one processor is configured to: receive the input.

[0250] Aspect 25. An apparatus according to any one of aspects 23 or 24, wherein the input comprises a user input received via a user interface.

[0251] Aspect 26. An apparatus according to any one of Aspects 23 to 25, wherein the input is based on constraints associated with a computing device, the constraints comprising at least one of a computing constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device.

[0252] Aspect 27. An apparatus according to any one of Aspects 19 to 26, wherein the at least one processor is configured to: determine the number of sampling steps based on a target perceptual quality and fidelity tradeoff of the final reconstructed image.

[0253] Aspect 28. An apparatus according to any one of aspects 19 to 27, wherein the residual model is conditioned on the initial reconstructed image.

[0254] Aspect 29. The apparatus of aspect 28, wherein the residual model is further conditioned on an auxiliary latent representation of the image.

[0255] Aspect 30. An apparatus according to any one of Aspects 19 to 29, wherein the at least one processor is configured to: use the residual model to process the initial reconstructed image and the noise data at a first sampling step to predict a first prediction of the residual; and use the residual model to process the initial reconstructed image and the first prediction of the residual at a second sampling step to predict a second prediction of the residual.

[0256] Aspect 31. The apparatus according to aspect 30, wherein the at least one processor is configured to: use the residual model to process the initial reconstructed image and a previous prediction of the residual to predict the final residual at the third sampling step.

[0257] Aspect 32. An apparatus according to aspect 13, wherein the previous prediction of the residual is the second prediction of the residual, or wherein the third sampling step is a final sampling step and the previous prediction of the residual is a residual prediction determined at a sampling step immediately before the final sampling step.

[0258] Aspect 33. An apparatus according to any one of Aspects 19 to 32, wherein the at least one processor is configured to: obtain the image; and process the image using an encoder of the machine learning model to generate the latent representation of the image.

[0259] Aspect 34. An apparatus according to any one of Aspects 19 to 33, wherein the at least one processor is configured to: obtain an auxiliary latent representation of the image; and use the residual model to process the initial reconstructed image, the noise data, and the auxiliary latent representation of the image to predict the multiple predictions of the residual.

[0260] Aspect 35. An apparatus according to any one of aspects 19 to 34, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

[0261] Aspect 36. An apparatus according to any one of aspects 19 to 35, wherein the plurality of predictions comprises a sequence of predictions.

[0262] Aspect 37. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations according to any one of aspects 1 to 18.

[0263] Aspect 38. An apparatus for processing image data, the apparatus comprising: one or more components for performing the operations according to any one of aspects 1 to 18.

Claims

1. A device for processing image data, the device comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: Obtaining a latent representation of the image; processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict a plurality of predictions of residuals over a plurality of sampling steps, wherein the residuals represent differences between the image and the initial reconstructed image; obtaining, from said plurality of predictions of said residuals, a final residual representing said difference between said image and said initial reconstructed image; as well as The initial reconstructed image and the residual are combined to generate a final reconstructed image.

2. The apparatus of claim 1, wherein the number of sampling steps is adjustable to obtain a compromise between the rate of the latent representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image.

3. The apparatus of claim 1 , wherein the at least one processor is configured to: Determines the number of sampling steps.

4. The apparatus of claim 3, wherein the at least one processor is configured to: The number of sampling steps is determined based on the settings.

5. The apparatus of claim 4, wherein the at least one processor is configured to: Configure the settings based on the input.

6. The apparatus of claim 5, wherein the at least one processor is configured to: The input is received. The apparatus of claim 5 , wherein the input comprises user input received via a user interface.

8. The apparatus of claim 5, wherein the input is based on a constraint associated with a computing device, the constraint comprising at least one of a computational constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device.

9. The apparatus of claim 1 , wherein the at least one processor is configured to: The number of sampling steps is determined based on a target perceptual quality and fidelity tradeoff for the final reconstructed image.

10. The apparatus of claim 1, wherein the residual model is conditioned on the initial reconstructed image.

11. The apparatus of claim 10, wherein the residual model is further conditioned on an auxiliary latent representation of the image.

12. The apparatus of claim 1, wherein the at least one processor is configured to: processing the initial reconstructed image and the noise data using the residual model at a first sampling step to predict a first prediction of the residual; and The initial reconstructed image and the first prediction of the residual are processed using the residual model at a second sampling step to predict a second prediction of the residual.

13. The apparatus of claim 12, wherein the at least one processor is configured to: The initial reconstructed image and a previous prediction of the residual are processed using the residual model at a third sampling step to predict the final residual. The device of claim 13 , wherein the previous prediction of the residual is the second prediction of the residual.

15. The apparatus of claim 13, wherein the third sampling step is a final sampling step, and wherein the previous prediction of the residual is a residual prediction determined at a sampling step immediately preceding the final sampling step.

16. The apparatus of claim 1, wherein the at least one processor is configured to: obtaining the image; and The image is processed using an encoder of the machine learning model to generate the latent representation of the image.

17. The apparatus of claim 1, wherein the at least one processor is configured to: obtaining an auxiliary latent representation of the image; and The initial reconstructed image, the noise data, and the auxiliary latent representation of the image are processed using the residual model to predict the plurality of predictions of the residual.

18. The device of claim 1, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

19. The apparatus of claim 1, wherein the plurality of predictions comprises a sequence of predictions.

20. A method for processing image data, the method comprising: Obtaining a latent representation of the image; processing the latent representation of the image using a decoder of a machine learning model to generate an initial reconstructed image; processing the initial reconstructed image and noise data using a residual model to predict a plurality of predictions of residuals over a plurality of sampling steps, wherein the residuals represent differences between the image and the initial reconstructed image; obtaining, from said plurality of predictions of said residuals, a final residual representing said difference between said image and said initial reconstructed image; as well as The initial reconstructed image and the residual are combined to generate a final reconstructed image.

21. The method of claim 20, wherein the number of sampling steps is adjustable to obtain a compromise between the rate of the latent representation, the perceptual quality of the final reconstructed image, and the distortion between the image and the final reconstructed image.

22. The method according to claim 20, further comprising: configuring the settings based on inputs, wherein the inputs are based on constraints associated with a computing device, the constraints comprising at least one of a computational constraint of the computing device, a power constraint of the computing device, or a latency constraint associated with the computing device; and The number of sampling steps is determined based on the settings.

23. The method according to claim 20, further comprising: The number of sampling steps is determined based on a target perceptual quality and fidelity tradeoff for the final reconstructed image.

24. The method of claim 20, wherein the residual model is conditioned on the initial reconstructed image.

25. The method of claim 24, wherein the residual model is further conditioned on an auxiliary latent representation of the image.

26. The method according to claim 20, further comprising: processing the initial reconstructed image and the noise data using the residual model at a first sampling step to predict a first prediction of the residual; as well as The initial reconstructed image and the first prediction of the residual are processed using the residual model at a second sampling step to predict a second prediction of the residual.

27. The method according to claim 26, further comprising: The initial reconstructed image and a previous prediction of the residual are processed using the residual model at a third sampling step to predict the final residual.

28. The method according to claim 20, further comprising: obtaining the image; as well as The image is processed using an encoder of the machine learning model to generate the latent representation of the image.

29. The method according to claim 20, further comprising: obtaining an auxiliary latent representation of the image; as well as The initial reconstructed image, the noise data, and the auxiliary latent representation of the image are processed using the residual model to predict the plurality of predictions of the residual.

30. The method of claim 20, wherein the residual model comprises one of a diffusion model or a recurrent neural network (RNN).

Images