Instance-adaptive image and video compression in network parameter subspaces using machine learning systems

Abstract

Techniques for compressing data using a machine learning system are described. An exemplary process can include receiving input data for compression by a neural network compression system. The process can include determining a set of updated model parameters for the neural network compression system based on the input data, where the set of updated model parameters is selected from a subspace of model parameters. The process can include generating at least one bitstream including a compressed version of the input data and a compressed version of one or more subspace coordinates corresponding to the set of updated model parameters. The process can include outputting the at least one bitstream for transmission to a receiver.

Description

[Technical Field]

[0001] This disclosure generally relates to data compression. For example, aspects of this disclosure include using machine learning systems to compress image and / or video content. [Background technology]

[0002] Many devices and systems enable image / video data to be processed and output for consumption. Digital image / video data contains large amounts of data to meet the increasing demands for image / video quality, performance, and functionality. For example, consumers of video data typically desire high-quality video with high fidelity, resolution, and frame rate. The large amounts of video data frequently required to meet these demands place a significant burden on communication networks, as well as on devices that process and store video data. Video coding techniques can be used to compress video data. One exemplary goal of video coding is to compress video data into a form that uses a lower bitrate while avoiding or minimizing degradation of video quality. As ever-evolving video services become available and the demand for large amounts of video data increases, coding techniques with better performance and efficiency are needed. [Overview of the project]

[0003] In some examples, systems and techniques for data compression and / or decompression using one or more machine learning systems are described. In some examples, machine learning systems (e.g., using one or more neural network systems) are provided for compressing and / or decompressing image / video data. According to at least one exemplary example, a method for processing image data is provided. This method may include: receiving input data for compression by a neural network compression system; determining an updated set of model parameters for the neural network compression system, selected from a subspace of model parameters, based on the input data; generating at least one bitstream containing a compressed version of the input data and a compressed version of one or more subspace coordinates corresponding to the updated set of model parameters; and outputting at least one bitstream for transmission to a receiver.

[0004] In another example, a device for processing image data is provided, comprising at least one memory and at least one processor (e.g., configured in a circuit) commutatically coupled to the at least one memory. The at least one processor may be configured to receive input data for compression by a neural network compression system, determine an updated set of model parameters for the neural network compression system, selected from a subspace of model parameters, based on the input data, generate at least one bitstream comprising a compressed version of the input data and compressed versions of one or more subspace coordinates corresponding to the updated set of model parameters, and output at least one bitstream for transmission to a receiver.

[0005] In another example, a non-temporary computer-readable medium is provided, which includes at least one instruction stored on the non-temporary computer-readable medium, which, when executed by one or more processors, causes one or more processors to receive input data for compression by a neural network compression system, to determine an updated set of model parameters for the neural network compression system, selected from a subspace of model parameters, based on the input data, to generate at least one bitstream containing a compressed version of the input data and a compressed version of one or more subspace coordinates corresponding to the updated set of model parameters, and to output at least one bitstream for transmission to a receiver.

[0006] In another example, a device for processing image data is provided. This device may include means for receiving input data to be compressed by a neural network compression system; means for determining an updated set of model parameters for the neural network compression system, selected from a subspace of model parameters, based on the input data; means for generating at least one bitstream including a compressed version of the input data and compressed versions of one or more subspace coordinates corresponding to the updated set of model parameters; and means for outputting at least one bitstream for transmission to a receiver.

[0007] Another example provides a method for processing image data. This method may include receiving a compressed version of the input data and a compressed version of one or more subspace coordinates; determining an updated set of model parameters for a neural network compression system based on the one or more subspace coordinates; and processing the input data using the updated set of model parameters to generate a reconstructed version of the input data.

[0008] In another example, an apparatus for processing image data is provided that includes at least one memory and at least one processor (e.g., configured in a circuit) communicatively coupled to the at least one memory. The at least one processor is configured to receive a compressed version of input data and a compressed version of one or more partial space coordinates, determine a set of updated model parameters for a neural network compression system based on the one or more partial space coordinates, and process the input data using the set of updated model parameters to generate a reconstructed version of the input data.

[0009] In another example, a non - transitory computer - readable medium storing at least one instruction that, when executed by one or more processors, causes the one or more processors to receive a compressed version of input data and a compressed version of one or more partial space coordinates, determine a set of updated model parameters for a neural network compression system based on the one or more partial space coordinates, and process the input data using the set of updated model parameters to generate a reconstructed version of the input data is provided.

[0010] In another example, an apparatus for processing image data is provided. The apparatus can include means for receiving a compressed version of input data and a compressed version of one or more partial space coordinates, means for determining a set of updated model parameters for a neural network compression system based on the one or more partial space coordinates, and means for processing the input data using the set of updated model parameters to generate a reconstructed version of the input data.

[0011] In some embodiments, the device may be a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smartphone," or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a 3D scanner, a multi-camera system, or other device, or a portion thereof. In some embodiments, the device includes one or more cameras for capturing one or more images. In some embodiments, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the device described above may include one or more sensors.

[0012] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0013] The above will become more apparent with reference to the following specification, claims, and attached drawings, along with other features and embodiments.

Brief Description of the Drawings

[0014] Exemplary embodiments of the present application will be described in detail below with reference to the following drawings. [Figure 1] FIG. shows an example of an image processing system according to some examples of the present disclosure. [Figure 2A] FIG. shows an example of a fully-connected neural network according to some examples of the present disclosure. [Figure 2B] FIG. shows an example of a locally-connected neural network according to some examples of the present disclosure. [Figure 2C]This figure shows an example of a convolutional neural network relating to some of the examples in this disclosure. [Figure 2D] This figure shows an example of a deep convolutional network (DCN) for recognizing visual features from images, relating to some examples of the present disclosure. [Figure 3] This block diagram shows exemplary deep convolutional networks (DCNs) relating to some examples of the present disclosure. [Figure 4] This figure shows an example of a system relating to some examples of the present disclosure, which includes a transmitting device for compressing video content and a receiving device for decompressing the received bitstream into video content. [Figure 5A] This figure shows an exemplary rate-distorted autoencoder system relating to some examples of the present disclosure. [Figure 5B] This figure shows an exemplary rate-distorted autoencoder system relating to some examples of the present disclosure. [Figure 6] This figure shows an exemplary inference process implemented by an exemplary neural network compression system, fine-tuned using model prior information, relating to some examples of the present disclosure. [Figure 7] This figure shows an exemplary instance adaptation process implemented by an exemplary neural network compression system fine-tuned within a network parameter subspace, relating to some examples of the present disclosure. [Figure 8] This is a graph showing an exemplary network parameter subspace relating to some examples of the disclosure. [Figure 9] This is a graph showing updates to network parameters in a subspace, relating to some examples of this disclosure. [Figure 10] This flowchart shows an example of a process for determining a subspace, relating to some examples of the present disclosure. [Figure 11] This figure shows an exemplary process for performing instance-adaptive neural compression, relating to some examples of the present disclosure. [Figure 12] This flowchart shows an example of the process for performing instance-adaptive neural decompression, relating to several examples in this disclosure. [Figure 13] This flowchart shows another example of the process for performing instance-adaptive neural compression, relating to some of the examples in this disclosure. [Figure 14] This flowchart shows another example of the process for performing instance-adaptive neural decompression, relating to some of the examples in this disclosure. [Figure 15] This document illustrates some exemplary computing systems relating to several examples of this disclosure. [Modes for carrying out the invention]

[0015] Specific aspects and embodiments of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some may be applied in combination. The following description includes specific details to provide a complete understanding of the embodiments of this application for illustrative purposes. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

[0016] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the Disclosure. Rather, the following description of exemplary embodiments provides a description that enables the implementation of the exemplary embodiments to those skilled in the art. It should be understood that various modifications may be made to the function and configuration of the elements without departing from the scope of this application as set forth in the appended claims.

[0017] As mentioned above, digital image and video data can contain large amounts of data, especially given the continuously increasing demand for high-quality video data. For example, consumers of image and video data typically desire increasingly higher video quality, such as high fidelity, resolution, and frame rate. However, the large amounts of data required to meet such demands can place a significant burden on communication networks, including high bandwidth and network resource requirements, as well as on devices that process and store the video data.

[0018] Because uncompressed image and video content can result in large files that require a considerable amount of memory for physical storage and a significant amount of bandwidth for transmission, techniques for compressing such image and / or video content may be utilized. For example, various compression algorithms (also called encoding algorithms or tools) can be applied to image and video content to reduce the size of the image content, and therefore the amount of storage required to store the image content and the amount of bandwidth required to deliver the video content.

[0019] In some cases, image content may be compressed using a priori defined compression algorithms, such as Joint Photographic Experts Group (JPEG) and Better Portable Graphics (BPG). For example, JPEG is a lossy compression format based on the Discrete Cosine Transform (DCT). For instance, a device performing JPEG compression on an image may compress the image by converting the image to an optimal color space (e.g., the YCbCr color space, which includes luminance (Y), chrominance blue (Cb), and chrominance red (Cr)), downsampling the chrominance components by averaging groups of pixels together, and applying the DCT function to blocks of pixels to remove redundant image data. The compression is based on identifying similar regions within an image and converting those regions to the same color code (based on the DCT function). Video content may also be compressed using a priori defined compression algorithms, such as the Motion Picture Experts Group (MPEG) algorithm, H.264, or the High Efficiency Video Coding algorithm.

[0020] These a priori-defined compression algorithms may be able to preserve most of the information in raw image and video content and may be defined a priori based on ideas of signal processing and information theory. However, while these given compression algorithms may be generally applicable (e.g., to any type of image / video content), the compression algorithms may not take into account the similarity of the content, new resolutions or frame rates for video capture and distribution, or unnatural images (e.g., radar images or other images captured through various sensors).

[0021] Compression algorithms defined a priori can be considered lossy compression algorithms. In lossy compression of an input image (or video frame), the input image cannot be encoded and then decoded / reconstructed, so the exact input image is reconstructed. Rather, lossy compression produces an approximate version of the input image after decoding / reconstruction of the compressed input image. Lossy compression results in a reduction of bitrate at the expense of distortion, resulting in artifacts present in the reconstructed image. Thus, there is a rate-distortion trade-off in lossy compression systems. In certain compression methods (e.g., JPEG, BPG, among others), distortion-based artifacts can take the form of blocking or other artifacts.

[0022] In some cases, machine learning systems can be used to perform compression and decompression of image and / or video data. For example, deep neural networks such as rate-distorted autoencoders (RD-AEs) can be used to perform compression and decompression of content (e.g., image content, video content, audio content, etc.). A deep neural network may include an autoencoder (AE) that maps images to a latent code space (e.g., containing a set of codes z). In some cases, a deep neural network may also include a probabilistic model (also called a pre-code model) that can reversibly compress the codes z from the latent code space. The probabilistic model can generate a probability distribution over a set of codes z that can represent encoded data based on the input data.

[0023] Machine learning-based compression techniques (e.g., RD-AE) can be used to generate compressed content with high quality and / or reduced bitrate. In some cases, an RD-AE can be trained to minimize the average rate distortion loss across a dataset of data points, such as image and / or video data points. In some cases, an RD-AE can also be fine-tuned to specific data points that are sent to and decoded by a receiver. In some cases, fine-tuning an RD-AE at data points can enable it to achieve high compression (rate / distortion) performance. The encoder associated with an RD-AE can send the AE model or a portion of the AE model to a receiver (e.g., a decoder) to decode the bitstream.

[0024] While machine learning-based compression systems can perform better than lossy compression systems, implementations of such systems exhibit certain drawbacks. For example, neural compression algorithms require large neural networks, which can lead to increased computational and / or memory requirements. Furthermore, the optimal settings (e.g., weights) used in deep neural networks are dependent on the training dataset and may not perform well when used with different data. For instance, a neural video codec trained on natural scenes may not perform well on animated scenes. Additionally, while a particular neural network can be fine-tuned based on each instance, sending updated and fine-tuned parameters to the decoder can significantly increase file size and / or bitrate.

[0025] In some embodiments, systems, apparatus, processes (also called methods), and computer-readable media (collectively referred to herein as “systems and techniques”) for performing compression and decompression (also called encoding and decoding, collectively referred to as coding) of data (e.g., images, videos, audio, etc.) using one or more machine learning systems are described herein. In some examples, machine learning techniques can provide image and / or video compression that produces high-quality visual output.

[0026] In some embodiments, a machine learning system (e.g., a neural network such as a rate-distortion autoencoder (RD-AE) or other neural networks) can be trained to minimize the mean rate-distortion loss across a dataset of data points, such as image and / or video data points. In some examples, a set of global model parameters (e.g., weights) can be associated with the trained machine learning system. In some cases, training a machine learning system can involve identifying parameters (e.g., a weight vector) associated with each iteration of training. For example, the weight vector could correspond to each training iteration, and the global model parameters could correspond to the weight vector of the trained neural network.

[0027] In some examples, training a machine learning system may involve determining a subspace or manifold of model parameters that has lower dimension than the entire parameter space. In some embodiments, the subspace or manifold of model parameters includes a portion of the weight vectors associated with each iteration of training. In some cases, the subspace or manifold of model parameters may be determined using principal component analysis (PCA). For example, PCA can be used to identify one or more directions and / or trajectories in the entire parameter space where the model parameters performed well during training of a neural network (e.g., if the result of the loss function falls below a threshold). In some embodiments, sparse PCA can be used to reduce the size of the subspace of model parameters. In some examples, the subspace or manifold of model parameters may be shared with a neural network configured to decode encoded data.

[0028] In some examples, a machine learning system can be fine-tuned (e.g., trained, fitted, etc.) for instances of input data (e.g., images, videos, parts of videos, 3D data, etc.) that are compressed and sent to a receiving device containing a decoder. In some embodiments, the fine-tuning of a machine learning system can be performed using a subspace or manifold of model parameters. For example, fine-tuning a neural network may involve selecting an updated set of model parameters that correspond to weight vectors that are part of a subspace or manifold of model parameters (e.g., selecting the optimal weight vectors for the instance). In some cases, the updated set of parameters (e.g., the weight vectors selected during fine-tuning) can be associated with one or more subspace coordinates. In one example, one or more subspace coordinates may correspond to a set of global parameters (e.g., the subspace coordinates may correspond to trajectories for a set of global parameters).

[0029] In some embodiments, an updated set of parameters can be used to encode the input data. In some examples, a machine learning system (e.g., a neural network such as RD-AE or other neural networks) may further include an arithmetic encoder (e.g., an arithmetic encoder, an arithmetic decoder, or a composite arithmetic encoder-decoder). The arithmetic encoder (e.g., an arithmetic encoder or a composite arithmetic encoder-decoder) can generate a bitstream containing a compressed version of the input data. In some cases, the bitstream may also contain compressed versions of one or more subspace coordinates corresponding to the updated set of model parameters. In some cases, a receiving device can receive the bitstream. The decoder in the receiving device can use one or more subspace coordinates to determine the updated model parameters of the neural network. The machine learning system of the decoder (e.g., a neural network such as RD-AE or other neural networks) can use the updated model parameters to decode the compressed input data.

[0030] In some embodiments, the systems and techniques disclosed herein may be used to improve compression performance by fitting a machine learning system (e.g., a neural network) to each data instance. In some examples, the systems and techniques disclosed herein can reduce the bitrate and / or file size for sending network parameter updates (e.g., a finely tuned neural network) to the decoder by selecting finely tuned weight vectors from a low-dimensional subspace.

[0031] One or more machine learning systems may be trained as described herein and used to perform data compression and / or decompression, such as compression and decompression of images, videos, and / or audio. The machine learning systems described herein may be trained to perform compression / decompression techniques that produce high-quality data output. The systems and techniques described herein can perform compression and / or decompression of any type of data. For example, in some cases, the systems and techniques described herein can perform compression and / or decompression of image data. As another example, in some cases, the systems and techniques described herein can perform compression and / or decompression of video data. Where used herein, the terms “image” and “frame” are used interchangeably and refer to a standalone image or frame (e.g., a photograph) or a group or sequence of images or frames (e.g., a video or other sequence of images / frames). As another example, in some cases, the systems and techniques described herein can perform compression and / or decompression of audio data. For the purposes of simplification, illustration, and explanation, the systems and techniques described herein are described with reference to the compression and / or decompression of image data (e.g., images or frames, video, etc.). However, as stated above, the concepts described herein can also be applied to other modalities such as audio data and any other type of data.

[0032] The compression models used by the encoder and / or decoder can be generalizable to different types of data. Furthermore, by utilizing instance-adaptive neural models having the various properties described herein, machine learning systems can improve the compression and / or decompression performance, bitrate, quality, and / or efficiency of specific datasets. For example, an instance-adaptive neural model-based machine learning system can reduce the bitrate by fine-tuning the neural network within a predefined subspace of network parameters and providing the decoder with parameter updates within that subspace.

[0033] As mentioned above, in some examples, a machine learning system can include one or more neural networks. Machine learning (ML) is a subset of artificial intelligence (AI). ML systems include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without using explicit instructions. An example of an ML system is a neural network (also called a pseudoneural network) which may consist of interconnected groups of pseudoneurons (e.g., nerve cell models). Neural networks can be used for a variety of applications and / or devices, particularly image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, and service robots.

[0034] Individual nodes within a neural network can emulate biological neurons by acquiring input data and performing simple operations on it. The results of these operations may be selectively passed to other neurons. Each vector and node in the network is associated with a weight value, which constrains how the input data relates to the output data. For example, the input data of each node may be multiplied by the corresponding weight value, and the products may be summed. The sum of the products may be adjusted by a discretionary bias, and an activation function may be applied to the result, potentially producing an output signal or "output activation" (sometimes called an activation map or feature map) for the node. The weight values ​​may initially be determined by an iterative flow of training data through the network (for example, the weight values ​​are established during the training phase, when the network learns how to identify specific classes based on their typical input data characteristics).

[0035] In particular, there are different types of neural networks, such as deep generative neural network models (e.g., generative adversarial networks (GANs)), recurrent neural network (RNN) models, multilayer perceptron (MLP) neural network models, convolutional neural network (CNN) models, and autoencoders (AEs). For example, a GAN is a form of generative neural network that can learn patterns in input data so that the neural network model can generate a new synthetic output that is reasonably likely to be from the original dataset. A GAN can include two neural networks working together. One neural network (called the generator, denoted as the generative neural network or G(z)) generates the synthetic output, and the other neural network (called the discriminative neural network, denoted as the discriminative, or D(X)) evaluates the output for authenticity (whether the output is from the original dataset, such as the training dataset, or whether it was generated by the generator). The training input and training output can include images as illustrative examples. The generator is trained to deceive the discriminator into believing that the synthetic images it produces are real images from the dataset. The training process continues, and the generator becomes better at producing synthetic images that look like real images. The discriminator continues to find defects in the synthetic images, and the generator learns what the discriminator is looking at to determine the defects in the images. Once the network is trained, the generator can produce images that look realistic enough that the discriminator cannot distinguish them from real images.

[0036] RNNs work by saving the output of a layer and feeding this output back into the input to help predict the results of that layer. In MLP neural networks, data may be fed into the input layer, and one or more hidden layers give the data a level of abstraction. Based on the abstracted data, predictions may then be made in the output layer. MLPs can be particularly well-suited to classification prediction problems where the input is assigned to a class or label. Convolutional neural networks (CNNs) are a type of feedforward artificial neural network. A CNN may consist of a set of artificial neurons that collectively tile the input space, each having a receptive field (e.g., a spatially localized region of the input space). CNNs have many applications, including pattern recognition and classification.

[0037] In a hierarchical neural network architecture (called a deep neural network when multiple hidden layers exist), the output of the first layer of artificial neurons becomes the input to the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to the third layer of artificial neurons, and so on. Convolutional neural networks may be trained to recognize feature hierarchies. Computations in convolutional neural network architectures may be distributed across a set of processing nodes that can be composed of one or more computational chains. These multilayer architectures may be trained one layer at a time and may be fine-tuned using backpropagation.

[0038] Autoencoders (AEs) can learn efficient data coding in an unsupervised manner. In some examples, an AE can learn a representation (e.g., data coding) of a dataset by training the network to ignore signal noise. An AE can include an encoder and a decoder. The encoder can map input data to a code, and the decoder can map the code to a reconstruction of the input data. In some examples, a rate-distortion autoencoder (RD-AE) can be trained to minimize the average rate-distortion loss across a dataset of data points, such as image and / or video data points. In some cases, an RD-AE can make a forward pass at inference to encode new data points.

[0039] Figure 1 shows an example of an image processing system 100 relating to some examples of the present disclosure. In some cases, the image processing system 100 may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., neural networks with weights), delays, frequency bin information, task information, among other information, may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from program memory associated with the CPU 102 and / or the memory block 118.

[0040] The image processing system 100 may include a GPU 104, a DSP 106, a connectivity block 110 which may include fifth-generation (5G) connectivity, fourth-generation long-term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth® connectivity, etc., and / or additional processing blocks tailored to specific functions, such as a multimedia processor 112 capable of detecting and recognizing features. In one implementation, the NPU 108 is implemented in the CPU 102, DSP 106, and / or GPU 104. The image processing system 100 may also include a sensor processor 114, one or more image signal processors (ISPs) 116, and / or a storage device 120. In some examples, the image processing system 100 may be based on the ARM instruction set.

[0041] The image processing system 100 may be part of one or more computing devices. In some examples, the image processing system 100 may be part of an electronic device (or more devices) such as a camera system (e.g., a digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., a smartphone, cellular phone, conferencing system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smartwatch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a game console, a video streaming device, a drone, an in-car computer, a system-on-a-chip (SOC), an Internet of Things (IoT) device, or any other suitable electronic device.

[0042] While the image processing system 100 is shown as including several components, it will be understood by those skilled in the art that the image processing system 100 may include more or fewer components than those shown in Figure 1. For example, in some cases, the image processing system 100 may also include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 1. Examples of computing devices and hardware components that may be implemented with the image processing system 100 are described below with reference to Figure 16.

[0043] Image processing system 100 and / or its components may be configured to perform compression and / or decompression (also known as encoding and / or decoding, collectively referred to as image coding) using the machine learning systems and techniques described herein. In some cases, image processing system 100 and / or its components may be configured to perform compression and / or decompression of images or videos using the techniques described herein. In some examples, the machine learning system may utilize a deep learning neural network architecture to perform compression and / or decompression of image, video, and / or audio data. By using a deep learning neural network architecture, the machine learning system can increase the efficiency and speed of compression and / or decompression of content on a device. For example, a device using the compression and / or decompression techniques described herein may efficiently compress one or more images using machine learning-based techniques, send one or more compressed images to a receiving device, and the receiving device may efficiently decompress one or more compressed images using the machine learning-based techniques described herein. As used herein, images may refer to still images and / or video frames associated with a sequence of frames (e.g., video).

[0044] As mentioned above, a neural network is an example of a machine learning system. A neural network can include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and an output is produced through the output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map, which can include artificial neurons (or nodes). Feature maps can include filters, kernels, etc. Nodes can include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers used to determine simple and low-level characteristics of the input, and later layers building a hierarchy of more complex and abstract characteristics.

[0045] A deep learning architecture may learn a hierarchy of features. If visual data is presented, for example, the first layer may learn to recognize relatively simple features such as edges in the input stream. In another embodiment, if auditory data is presented, the first layer may learn to recognize spectral power at a specific frequency. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, a higher layer may learn to represent complex shapes in visual data or words in auditory data. A higher layer may be learned to recognize common visual objects or speech phrases.

[0046] Deep learning architectures can sometimes work particularly well when applied to problems with natural hierarchical structures. For example, classifying electric vehicles may benefit from initially learning to recognize wheels, windshields, and other features. These features may then be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0047] Neural networks may be designed using various connectivity patterns. In a feedforward network, each neuron in a given layer communicates with neurons in higher layers to pass information from lower to higher layers. As mentioned above, a hierarchical representation may be constructed within the consecutive layers of a feedforward network. Neural networks may also have recursive or feedback (also called top-down) connectivity. In recursive connectivity, the output from a neuron in a given layer may be transmitted to another neuron in the same layer. Recursive architectures can be useful when recognizing patterns across two or more chunks of input data delivered to the neural network in a sequence. Connectivity from a neuron in a given layer to a neuron in a lower layer is called feedback (or top-down) connectivity. Networks with a large number of feedback connectivitys can be useful when the recognition of a higher-level concept can help distinguish certain lower-level features of the input.

[0048] The connections between layers of a neural network may be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, neurons in the first layer may transmit their output to any neuron in the second layer, so that each neuron in the second layer receives input from any neuron in the first layer. Figure 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, neurons in the first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured so that each neuron in a given layer has the same or similar connectivity pattern, but with different connection strengths (e.g., 210, 212, 214, and 216). Because higher-layer neurons within a given region may receive inputs that, through training, are adapted to the characteristics of a constrained portion of the total input to the network, the connectivity patterns of local connections can give rise to spatially distinct receptive fields within the higher layers.

[0049] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be suitable for problems where the spatial location of the input is meaningful. The convolutional neural network 206 may be used to perform one or more embodiments of video compression and / or decompression according to embodiments of this disclosure.

[0050] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN200 designed to recognize visual features from an image 226 input from an image capture device 230, such as an in-vehicle camera. The DCN200 in this example may be trained to identify traffic signs and the numbers given on the traffic signs. Of course, the DCN200 may also be trained for other tasks, such as identifying lane markings or traffic signals.

[0051] DCN200 may be trained using supervised learning. During training, DCN200 may be presented with images, such as image 226 of a speed limit sign, and then a forward pass may be computed to produce output 222. DCN200 may include a feature extraction section and a classification section. Upon receiving image 226, the convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. For example, the convolutional kernel for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this embodiment, four different feature maps are generated in the first set 218 of feature maps, so four different convolutional kernels were applied to image 226 in convolutional layer 232. Convolutional kernels are sometimes called filters or convolutional filters.

[0052] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218; that is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0053] In the example in Figure 2D, a second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may contain a number corresponding to a possible feature of image 226, such as "label", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN200 is the probability that image 226 contains one or more features.

[0054] In this embodiment, the probabilities in output 222 for “label” and “60” are higher than the probabilities of other outputs 222 such as “30”, “40”, “50”, “70”, “80”, “90”, and “100”. Before training, the outputs 222 generated by DCN200 may be inaccurate. Therefore, an error may be calculated between output 222 and the target output. The target output is the ground truth of image 226 (e.g., “label” and “60”). The weights of DCN200 may then be adjusted so that the output 222 of DCN200 is more closely matched with the target output.

[0055] To adjust the weights, the learning algorithm may calculate gradient vectors for the weights. The gradient can indicate the amount by which the error will increase or decrease when the weights are adjusted. In the top layer, the gradient can directly correspond to the weight values ​​connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient can depend on the weight values ​​and the error gradient calculated in the upper layers. The weights may then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0056] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be iterated until the achievable error rate for the entire system stops decreasing or until the error rate reaches a target level. After training, the DCN may be presented with a new image, and a forward pass through the network may yield an output 222 which may be considered an estimate or prediction of the DCN.

[0057] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs may be used to extract a hierarchical representation of a training dataset. DBNs may also be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn a probability distribution without any information about the class to which each input should be categorized, RBMs are frequently used in unsupervised learning. Using a hybrid paradigm of unsupervised and supervised learning, the lower RBMs of a DBN can be trained unsupervised and can function as feature extractors, while the upper RBMs can be trained supervised (on a joint distribution of inputs from previous layers and target classes) and can function as classifiers.

[0058] A deep convolutional network (DCN) is a network of convolutional networks consisting of additional pooling and normalization layers. DCNs achieve state-of-the-art performance for a wide range of tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for a large number of samples, and the network weights are modified using gradient descent.

[0059] A DCN may also be a feedforward network. In addition, as described above, connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational burden of a DCN may be significantly less than that of a similarly sized neural network, for example, one that includes recursive or feedback connections.

[0060] The processing of each layer of a convolutional network may be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input may be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension that captures color information. The output of the convolutional connections may be considered as forming a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map may be further processed using nonlinearities such as rectification and max(0,x). Values ​​from neighboring neurons may further undergo pooling corresponding to downsampling, which can result in additional local invariance and dimensionality reduction.

[0061] Figure 3 is a block diagram showing an example of a deep convolutional network 350. The deep convolutional network 350 may include several different types of layers based on connectivity and weight sharing. As shown in Figure 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may consist of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0062] The convolutional layer 356 may include one or more convolutional filters that can be applied to the input data 352 to generate a feature map. Although only two convolutional blocks 354A and 354B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks (e.g., blocks 354A and 354B) may be included in the deep convolutional network 350 according to design preferences. A normalization layer 358 may normalize the output of the convolutional filters. For example, the normalization layer 358 may result in whitening or lateral suppression. A maximum pooling layer 360 may result in downsampling aggregation across space for local invariance and dimensionality reduction.

[0063] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the image processing system 100 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the image processing system 100. Furthermore, the deep convolutional network 350 may have access to other processing blocks that may be present on the image processing system 100, such as the sensor processor 114.

[0064] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 are weights (not shown) that will be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as input to subsequent layers among the layers in the deep convolutional network 350 (e.g., 356, 358, 360, 362, 364) to learn a hierarchical feature representation from the initially supplied input data 352 (e.g., image, audio, video, sensor data, and / or other input data) in the convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0065] Image and video content may be stored and / or shared between devices. For example, image and video content can be uploaded to media hosting services and sharing platforms and transmitted to various devices. Recording uncompressed image and video content generally results in large file sizes, which increase significantly with increasing resolution of the image and video content. For example, uncompressed 16-bit / channel video recorded at 1080p / 24 (e.g., 24 frames per second, with a resolution of 1920 pixels wide and 1080 pixels high) may occupy 12.4 megabytes per frame, or 297.6 megabytes per second. Uncompressed 16-bit / channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame, or 1195.2 megabytes per second.

[0066] Because uncompressed image and video content can result in large files that require a considerable amount of memory for physical storage and a significant amount of bandwidth for transmission, techniques for compressing such video content may be utilized. For example, various compression algorithms can be applied to image and video content to reduce the size of the image content, and therefore the amount of storage required to store the image content and the amount of bandwidth required to deliver the video content.

[0067] In some cases, image content may be compressed using a priori defined compression algorithms, such as Joint Photographic Experts Group (JPEG) and Better Portable Graphics (BPG). For example, JPEG is a lossy compression format based on the Discrete Cosine Transform (DCT). For instance, a device performing JPEG compression on an image may compress the image by converting the image to an optimal color space (e.g., the YCbCr color space, which includes luminance (Y), chrominance blue (Cb), and chrominance red (Cr)), downsampling the chrominance components by averaging groups of pixels together, and applying the DCT function to blocks of pixels to remove redundant image data. The compression is based on identifying similar regions within an image and converting those regions to the same color code (based on the DCT function). Video content may also be compressed using a priori defined compression algorithms, such as the Motion Picture Experts Group (MPEG) algorithm, H.264, or the High Efficiency Video Coding algorithm.

[0068] These a priori-defined compression algorithms may be able to preserve most of the information in raw image and video content and may be defined a priori based on ideas of signal processing and information theory. However, while these given compression algorithms may be generally applicable (e.g., to any type of image / video content), the compression algorithms may not take into account the similarity of the content, new resolutions or frame rates for video capture and distribution, or unnatural images (e.g., radar images or other images captured through various sensors).

[0069] Compression algorithms defined a priori are considered lossy compression algorithms. In lossy compression of an input image (or video frame), the input image cannot be encoded and then decoded / reconstructed, so the exact input image is reconstructed. Rather, lossy compression produces an approximate version of the input image after decoding / reconstruction of the compressed input image. Lossy compression results in a reduction of bitrate at the expense of distortion, resulting in artifacts present in the reconstructed image. Thus, there is a rate-distortion trade-off in lossy compression systems. In certain compression methods (e.g., JPEG, BPG, among others), distortion-based artifacts can take the form of blocking or other artifacts. In some cases, neural network-based compression can be used, which can result in high-quality compression of image and video data. In some cases, blurring and color shift are examples of artifacts.

[0070] Whenever the bitrate falls below the true entropy of the input data, it may be difficult or impossible to reconstruct the accurate input data. However, the fact that there is distortion / loss resulting from data compression / decompression does not mean that the reconstructed image or frame will not have artifacts. In fact, it may be possible to reconstruct a compressed image into another similar but different image with high visual quality.

[0071] In some cases, compression and decompression may be performed using one or more machine learning (ML) systems. In some examples, such ML-based systems can provide image and / or video compression that produces high-quality visual output. In some examples, such systems can perform compression and decompression of content (e.g., image content, video content, audio content, etc.) using deep neural networks such as rate-distorted autoencoders (RD-AEs). A deep neural network may include an autoencoder (AE) that maps an image to a latent code space (e.g., containing a set of codes z). The latent code space may include a code space used by the encoder and decoder, where the content is encoded in codes z. The codes (e.g., codes z) may also be called latent information, latent variables, or latent representations. A deep neural network may include a probabilistic model (also called a pre-code model) that can reversibly compress the codes z from the latent code space. The probabilistic model can generate a probability distribution over a set of codes z that can represent encoded data based on the input data. In some cases, the probability distribution may be expressed as (P(z)).

[0072] In some examples, a deep neural network may include an arithmetic encoder that generates a bitstream containing compressed data output based on a set of probability distributions P(z) and / or codes z. The bitstream containing the compressed data may be stored and / or transmitted to a receiving device. The receiving device may perform the inverse process to decode or decode the bitstream, for example, using an arithmetic decoder, a probabilistic (or coded) model, and an AE decoder. The device that generated the bitstream containing the compressed data may perform a similar decode / decode process when retrieving the compressed data from storage. Similar techniques may be employed to compress / encode and decode / decode updated model parameters.

[0073] In some cases, RD-AEs can be trained and operated to run as multi-rate AEs (including high-rate and low-rate operations). For example, the latent code space generated by the encoder of a multi-rate AE may be divided into two or more chunks (e.g., code z divided into chunks z1 and z2). In high-rate operation, the multi-rate AE can transmit a bitstream based on the entire latent space (e.g., code z including z1, z2, etc.) which can be used by the receiving device to decompress the data, similar to the operation described above for RD-AEs. In low-rate operation, the bitstream transmitted to the receiving device is based on a subset of the latent space (e.g., chunk z1 but not z2). The receiving device can infer the rest of the latent space based on the transmitted subset and generate reconstructed data using the subset of the latent space and the inferred rest of the latent space.

[0074] By compressing (and decompressing) content using RD-AE or multirate AE, encoding and decoding mechanisms can be adapted to various use cases. Machine learning-based compression techniques can produce compressed content with high quality and / or reduced bitrate. In some cases, an RD-AE can be trained to minimize the average rate distortion loss across a dataset of data points, such as image and / or video data points. In some cases, an RD-AE can also be fine-tuned to specific data points that are sent to and decoded by a receiver. In some cases, fine-tuning the RD-AE at data points can result in high compression (rate / distortion) performance. The encoder associated with the RD-AE can send the AE model or a portion of the AE model to a receiver (e.g., a decoder) to decode the bitstream.

[0075] In some cases, a neural network compression system can reconstruct an input instance (e.g., input image, video, audio, etc.) from a (quantized) latent representation. The neural network compression system can also be used before reversibly compressing the latent representation. In some cases, a neural network compression system can determine a test data distribution with known relatively low entropy (e.g., a camera viewing a still scene, a dashcam in an autonomous vehicle) and fine-tune or adapt to such a distribution. Fine-tuning or adaptation can result in improved rate / distortion (RD) performance. In some examples, the model of the neural network compression system can be adapted to a single input instance being compressed. The neural network compression system can provide model updates, which, in some examples, can be quantized and compressed using parameter-space prior information along with the latent representation.

[0076] Fine-tuning can take into account the effects of model quantization and the additional costs incurred by transmitting model updates. In some examples, neural network compression systems can be fine-tuned using an additional model rate term M that measures the RD loss, as well as the number of bits required to transmit model updates under model prior information, resulting in a combined RDM loss.

[0077] Figure 4 shows a system 400 comprising a transmitting device 410 and a receiving device 420, relating to some examples of the present disclosure. The transmitting device 410 and the receiving device 420 may, in some cases, be referred to as RD-AE, respectively. The transmitting device 410 can compress image content, store the compressed image content, and / or transmit the compressed image content to the receiving device 420 for decompression. The receiving device 420 can decompress the compressed image content, output the decompressed image content onto the receiving device 420 (e.g., for display, editing, etc.), and / or output the decompressed image content to another device connected to the receiving device 420 (e.g., a television, mobile device, or other device). In some cases, the receiving device 420 can become a transmitting device by compressing image content (using an encoder 422), storing the compressed image content, and / or transmitting it to another device such as the transmitting device 410 (in which case the transmitting device 410 becomes a receiving device). While System 400 is described herein in relation to image compression and decompression, those skilled in the art will understand that System 400 can also be used with the techniques described herein to compress and decompress video content.

[0078] As shown in Figure 4, the transmitting device 410 includes an image compression pipeline, and the receiving device 420 includes an image bitstream decompression pipeline. According to aspects of this disclosure, the image compression pipeline in the transmitting device 410 and the bitstream decompression pipeline in the receiving device 420 generally use one or more artificial neural networks to compress image content and / or decompress the received bitstream into image content. The image compression pipeline in the transmitting device 410 includes an autoencoder 401, a coding model 404, and an arithmetic encoder 406. In some implementations, the arithmetic encoder 406 is optional and may be omitted in some cases. The image decompression pipeline in the receiving device 420 includes an autoencoder 421, a coding model 424, and an arithmetic decoder 426. In some implementations, the arithmetic decoder 426 is optional and may be omitted in some cases. The autoencoder 401 and coding model 404 of the transmitting device 410 are shown in Figure 4 as a machine learning system that is pre-trained and therefore configured to perform operations during inference or operation of a trained machine learning system. The autoencoder 421, the coding model 424, and the completion model 425 are also shown as pre-trained machine learning systems.

[0079] The autoencoder 401 comprises an encoder 402 and a decoder 403. The encoder 402 can perform lossy compression on the received uncompressed image content by mapping pixels in one or more images of the uncompressed image content to a latent code space (containing codes z). Generally, the encoder 402 may be configured such that the codes z representing the compressed (or encoded) image are discrete or binary. These codes may be generated based on stochastic perturbation techniques, soft vector quantization, or other techniques capable of generating distinct codes. In some embodiments, the autoencoder 401 may map the uncompressed image to codes having a compressible (low-entropy) distribution. These codes may have cross-entropy close to a predefined or learned prior distribution.

[0080] In some examples, the autoencoder 401 may be implemented using a convolutional architecture. For example, in some cases, the autoencoder 401 may be configured as a two-dimensional convolutional neural network (CNN) so that the autoencoder 401 learns spatial filters for mapping image content into latent code space. In an example where system 400 is used to encode video data, the autoencoder 401 may be configured as a three-dimensional CNN so that the autoencoder 401 learns spatiotemporal filters for mapping video into latent code space. In such a network, the autoencoder 401 may encode video with respect to keyframes (e.g., initial frames that mark the start of a sequence of frames, where subsequent frames in the sequence are described as differences to the initial frames in the sequence), warping (or differences) between keyframes and other frames in the video, and residual factors. In other embodiments, the autoencoder 401 may be implemented as a two-dimensional neural network subject to adjustments via previous frames, residual coefficients between frames, and stacked channels, or adjustments including iterative layers.

[0081] The encoder 402 of the autoencoder 401 can receive a first image (specified as image x in Figure 4) as input and can map the first image x to a code z in the latent code space. As described above, the encoder 402 can be implemented as a two-dimensional convolutional network such that the latent code space has a vector at each (x,y) position that describes a block of image x centered at that position. The x coordinate can represent the horizontal pixel position within a block of image x, and the y coordinate can represent the vertical pixel position within a block of image x. When encoding video data, the latent code space can have t variables or positions, and the t variables represent timestamps within blocks of video data (in addition to the spatial x and y coordinates). By using the two dimensions of horizontal and vertical pixel positions, the vector can describe image patches within image x.

[0082] Next, the decoder 403 of the autoencoder 401 decodes the code z and reconstructs the first image x.

[0083]

number

[0084]

number

[0085]

number

[0086] The coding model 404 receives a code z representing an encoded image or a portion thereof and generates a probability distribution P(z) over a set of compressed codewords that can be used to represent the code z. In some examples, the coding model 404 may include a stochastic autoregressive generative model. In some cases, the codes from which the probability distribution can be generated include a learned distribution that controls bit assignment based on the arithmetic encoder 406. For example, using the arithmetic encoder 406, the compressed code for a first code z can be predicted independently, the compressed code for a second code z can be predicted based on the compressed code for the first code z, the compressed code for a third code z can be predicted based on the compressed codes for the first and second code z, and so on. The compressed codes generally represent different spatiotemporal chunks of a given image being compressed.

[0087] In some embodiments, z can be represented as a three-dimensional tensor. The three dimensions of the tensor are the feature channel dimension, as well as the height and width spatial dimensions (e.g., the code z). (c,w,h) It may include (represented as). Each sign z (c,w,h)The codes (representing codes indexed by channel, horizontal position, and vertical position) can be predicted based on previous codes, which can be in a fixed, theoretically arbitrary order. In some examples, the codes can be generated by analyzing a given image file from beginning to end and analyzing each block in the image in raster scan order.

[0088] The code model 404 can learn the probability distribution of the input code z using a stochastic autoregressive model. The probability distribution can be conditioned on its previous values ​​(as described above). In some examples, the probability distribution can be expressed by the following equation:

[0089]

number

[0090] In some cases, a probability distribution P(z) can be predicted by a fully convolutional neural network of causal convolution. In some embodiments, the kernel of each layer of the convolutional neural network predicts the previous value z. 0:c,0:w,0:h These can be recognized and masked so that other values ​​do not need to be recognized when calculating the probability distribution. In some embodiments, the final layer of the convolutional network may include a softmax function that determines the probability that a code in the latent space is applicable across input values ​​(e.g., the likelihood that a given code can be used to compress a given input).

[0091] The arithmetic encoder 406 generates a bitstream 415 (shown as "0010011..." in Figure 4) corresponding to the prediction of code z, using the probability distribution P(z) generated by the code model 404. The prediction of code z can be represented as the code having the highest probability score in the probability distribution P(z) generated across the set of possible codes. In some embodiments, the arithmetic encoder 406 can output a bitstream of variable length based on the accuracy of the prediction of code z and the actual code z generated by the autoencoder 401. For example, bitstream 415 can correspond to a short codeword if the prediction is accurate, while bitstream 415 can correspond to a longer codeword as the magnitude of the difference between code z and the prediction of code z increases.

[0092] In some cases, bitstream 415 may be output by arithmetic encoder 406 for storage in a compressed image file. Bitstream 415 may also be output for transmission to a requesting device (e.g., receiving device 420, as shown in Figure 4). Generally, bitstream 415 output by arithmetic encoder 406 can reversibly encode z so that z can be accurately restored during the decompression process applied to the compressed image file.

[0093] The bitstream 415 generated by the arithmetic encoder 406 and transmitted from the transmitting device 410 can be received by the receiving device 420. Transmission between the transmitting device 410 and the receiving device 420 can be carried out using any of a variety of suitable wired or wireless communication technologies. Communication between the transmitting device 410 and the receiving device 420 may occur directly or via one or more network infrastructure components (e.g., base stations, relay stations, mobile stations, network hubs, routers, and / or other network infrastructure components).

[0094] As shown in the figure, the receiving device 420 may include an arithmetic decoder 426, a code model 424, and an autoencoder 421. The autoencoder 421 comprises an encoder 422 and a decoder 423. The decoder 423 can produce the same or similar output as the decoder 403 for a given input. Although the autoencoder 421 is shown as including an encoder 422, the encoder 422 takes code z received from the transmitting device 410.

[0095]

number

[0096] The received bitstream 415 may be input to an arithmetic decoder 426 to obtain one or more codes z from the bitstream. The arithmetic decoder 426 may extract decompressed codes z based on a probability distribution P(z) generated by a coding model 424 over a set of possible codes and information relating each generated code z to the bitstream. Given the received portion of the bitstream and a probabilistic prediction of the next code z, the arithmetic decoder 426 may generate a new code z as encoded by the arithmetic encoder 406 in the transmitting device 410. Using the new code z, the arithmetic decoder 426 may make probabilistic predictions for consecutive codes z, read additional portions of the bitstream, and decode consecutive codes z until the entire received bitstream is decoded. The decompressed codes z may be provided to a decoder 423 in an autoencoder 421. The decoder 423 decompresses the codes z and approximates the image content x.

[0097]

number

[0098]

number

[0099]

number

[0100] As described above, the autoencoder 401 and coding model 404 of the transmitting device 410 are shown in Figure 4 as a pre-trained machine learning system. In some embodiments, the autoencoder 401 and coding model 404 can be trained together using image data. For example, the encoder 402 of the autoencoder 401 can receive a first training image n as input and map the first training image n to a code z in the latent code space. The coding model 404 can learn the probability distribution P(z) of code z using a stochastic autoregressive model (similar to the technique described above). The arithmetic encoder 406 can generate an image bitstream using the probability distribution P(z) generated by the coding model 404. Using the bitstream and probability distribution P(z) from the coding model 404, the arithmetic encoder 406 can generate code z and output code z to the decoder 403 of the autoencoder 401. The decoder 403 then decrypts code z to reconstruct the first training image n.

[0101]

number

[0102]

number

[0103] In some cases, the backpropagation engine used during training of the transmitting device 410 may perform a backpropagation process to adjust the parameters of the neural network of the autoencoder 401 (e.g., weights, biases, etc.) and the coding model 404 based on one or more loss functions. In some cases, the backpropagation process may be based on stochastic gradient descent. Backpropagation may include a forward pass, one or more loss functions, a backward pass, and weight (and / or other parameter) updates. The forward pass, loss function, backward pass, and parameter updates may be performed for one training iteration. This process may be repeated for a certain number of iterations for each set of training data until the weights and / or other parameters of the neural network are precisely adjusted.

[0104] For example, the autoencoder 401 has n and

[0105]

number

[0106]

number

[0107]

number

[0108]

number

[0109]

number

[0110] Based on the determined loss (e.g., distance vector or other difference values), and using a backpropagation process, the parameters (e.g., weights, biases, etc.) of the neural network system and coding model 404 of the autoencoder 401 can be adjusted (effectively adjusting the mapping between the received image content and the latent coding space) to reduce the loss between the input uncompressed image and the compressed image content produced as output by the autoencoder 401.

[0111] The actual output value (reconstructed image) can differ significantly from the input image, which can result in a high loss (or error) in the first training image. The goal of training is to minimize the amount of loss for the predicted output. A neural network can perform a reverse pass by determining which node of the neural network (with corresponding weights) contributed the most to the network's loss, and then adjust the weights (and / or other parameters) so that the loss decreases and is eventually minimized. The derivative of the loss with respect to the weights (expressed as dL / dW, where W is the weight in a particular layer) can be calculated to determine which weight contributed the most to the neural network's loss. For example, weights can be updated so that they change in the opposite direction of the gradient. Weight updates are performed as follows:

[0112]

number

[0113] The neural network system of autoencoder 401 and coding model 404 can continue to be trained in this manner until a desired output is achieved. For example, autoencoder 401 and coding model 404 can repeat the backpropagation process to decompose the input image n and reconstructed image resulting from the decomposition of the generated code z.

[0114]

number

[0115] The autoencoder 421 and coding model 424 may be trained using techniques similar to those described above for training the autoencoder 401 and coding model 404 of the transmitting device 410. In some cases, the autoencoder 421 and coding model 424 may be trained using the same or different training datasets used to train the autoencoder 401 and coding model 404 of the transmitting device 410.

[0116] In the example shown in Figure 4, the rate-distorted autoencoder (transmitting device 410 and receiving device 420) is trained and executed in inference according to the bitrate. In some implementations, the rate-distorted autoencoder may be trained at multiple bitrates to enable the generation and output of high-quality reconstructed images or video frames (with or without artifacts due to distortion of the input image) when varying amounts of information are provided to the latent code z.

[0117] In some implementations, the latent code z can be divided into at least two chunks, z1 and z2. When the RD-AE model is used in a high-rate setting, both chunks are sent to the device for decoding. When the rate-distorted autoencoder model is used in a low-rate setting, only chunk z1 is sent, and chunk z2 is inferred from z1 on the decoder side. The inference of z2 from z1 can be performed using various techniques, as will be described in more detail below.

[0118] In some implementations, a set of continuous latents (e.g., capable of transmitting a large amount of information) and a corresponding quantized discrete latent (e.g., containing less information) may be used. After training the RD-AE model, an auxiliary inverse quantization model may be trained. In some cases, when using RD-AE, only discrete latents are transmitted, and the auxiliary inverse quantization model is used on the decoder side to infer the continuous latent from the discrete latent.

[0119] While System 400 is shown to include certain components, those skilled in the art will understand that System 400 may include more or fewer components than those shown in Figure 4. For example, the transmitting device 410 and / or receiving device 420 of System 400 may also optionally include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 4. The components shown in Figure 4, and / or other components of System 400, may be implemented using one or more computing or processing components. One or more computing components may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or an image signal processor (ISP). Exemplary examples of computing devices and hardware components that may be implemented by System 1600 are described below with respect to Figure 16.

[0120] System 400 may be part of a single computing device or multiple computing devices, or implemented by them. In some examples, the transmitting device 410 may be part of a first device, and the receiving device 420 may be part of a second computing device. In some examples, the transmitting device 410 and / or the receiving device 420 may be included as part of an electronic device such as a telephone system (e.g., smartphone, mobile phone, conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, a video streaming device, a SOC, an IoT (Internet of Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), or any other suitable electronic device. In some cases, System 400 may be implemented by the image processing system 100 shown in Figure 1. In other cases, System 400 may be implemented by one or more other systems or devices.

[0121] Figure 5A shows an exemplary neural network compression system 500. In some examples, the neural network compression system 500 may include an RD-AE system. In Figure 5A, the neural network compression system 500 comprises an encoder 502, an arithmetic encoder 508, an arithmetic decoder 512, and a decoder 514. In some cases, the encoder 502 and / or decoder 514 may be the same as the encoder 402 and / or decoder 403, respectively. In other cases, the encoder 502 and / or decoder 514 may be different from the encoder 402 and / or decoder 403, respectively.

[0122] Encoder 502 takes image 501 (image x) as input. i ) can be received, and image 501 (image x i) can be mapped and / or converted to a latent code 504 (latent z) within the latent code space. Image 501 can represent a still image and / or a video frame associated with a sequence of frames (e.g., video). In some cases, en - coder 502 can execute a forward pass to generate the latent code 504. In some examples, encoder 502 can implement a learnable function. In some cases, encoder 502 can implement a learnable function parameterized by φ. For example, encoder 502 can implement the function q i (z|x). In some examples, the learnable function need not be shared with, or known to, decoder 514. φ <0,000,540>

[0123] Arithmetic encoder 508 can generate a bit - stream 510 based on the latent code 504 (latent z i ) and the latent prior information 506. In some examples, the latent prior information 506 can implement a learnable function. In some cases, the latent prior information 506 can implement a learnable function parameterized by ψ. For example, the latent prior information 506 can implement the function p ψ (z). The latent prior information 506 can be used to convert the latent code 504 (latent z i ) to the bit - stream 510 using lossless compression. The latent prior information 506 can be shared and / or utilized on both the transmitting side (e.g., encoder 502 and / or arithmetic encoder 508) and the receiving side (e.g., arithmetic decoder 512 and / or decoder ⑤).

[0124] Decoder 514 receives the encoded bit - stream 510 from arithmetic encoder 508 and can decode the latent code 504 (latent z i ) within the encoded bit - stream 510 using the latent prior information 506. Decoder 514 can approximate a reconstructed image 516 (reconstructed i ) from the latent code 504 (latent z

[0125] ​

number

[0126] The neural network compression system 500 can be trained to minimize rate distortion. In some examples, the rate reflects the length of bitstream 510 (bitstream b), and the distortion is the length of image 501 (image x). i ) and reconstructed image 516 (reconstruction)

[0127]

number

[0128] In some examples, the loss can be expressed as follows:

[0129]

number

[0130] The rate for transmitting potential is Rz It can be expressed as (z;ψ). In some examples, the term logp ψ (z) is the rate R z (z;ψ) can be shown and / or represented. In some cases, the loss can be minimized over the entire dataset D as follows:

[0131]

number

[0132] Figure 5B shows the inference process 530 by the neural network compression system 500. As shown in the figure, the encoder 502 can convert the image 501 into a latent code 504. In some examples, the image 501 may represent a still image and / or video frame associated with a sequence of frames (e.g., video).

[0133] In some examples, encoder 502 is a single forward path

[0134]

number

[0135]

number

[0136]

number

[0137] The arithmetic decoder 512 receives the bitstream 520 from the arithmetic encoder 508 and, under the latent prior information 506, processes the latent code 504 (latent z i ) can perform arithmetic decoding. In some examples, the arithmetic decoder 512 can decode the latent code 504 from the bitstream 520 as follows:

[0138]

number

[0139]

number

[0140]

number

[0141] In some cases, an RD-AE system may be trained using a set of training data and further fine-tuned for data points (e.g., image data, video data, audio data) transmitted and decoded by a receiver (e.g., a decoder). For example, during inference, an RD-AE system may be fine-tuned to match the image data being transmitted to the receiver. Because compressed models are generally large, transmitting parameters associated with the model to the receiver can be very costly in terms of resources such as network (e.g., bandwidth), storage, and computing resources. In some cases, an RD-AE system may be fine-tuned for a single data point that is compressed and sent to the receiver for decompression. This can limit the amount of information (and associated costs) transmitted to the receiver while maintaining and / or improving the efficiency, performance, and / or quality of compression / decompression.

[0142] Figure 6 shows an exemplary inference process implemented by an exemplary neural network compression system 600, which has been fine-tuned using model prior information. In some examples, the neural network compression system 600 may include an RD-AE system fine-tuned using RDM-AE model prior information. In some cases, the neural network compression system 600 may include an AE model fine-tuned using model prior information.

[0143] In this exemplary example, the neural network compression system 600 includes an encoder 602, an arithmetic encoder 608, an arithmetic decoder 612, a decoder 614, model prior information 616, and potential prior information 606. In some cases, encoder 602 may be the same as or different from encoder 402 or encoder 502, and decoder 614 may be the same as or different from decoder 403 or decoder 514. Arithmetic encoder 608 may be the same as or different from arithmetic encoder 406 or arithmetic encoder 508, and arithmetic decoder 612 may be the same as or different from arithmetic decoder 426 or arithmetic decoder 512.

[0144] The neural network compression system 600 uses latent code 604 (latent z) for image 601. i The neural network compression system 600 can generate image 601 (image x). i ) encodes and reconstructs image 620 (reconstruction

[0145]

number

[0146] In some examples, the neural network compression system 600 can be fine-tuned using RDM-AE loss. The neural network compression system 600 can be trained by minimizing the rate-distortion model rate (RDM) loss. In some examples, on the encoder side, the AE model is trained using RDM loss as follows: image 601 (image x i ) can be fine-tuned on:

[0147]

number

[0148] The finely tuned encoder 602 generates latent code 604 from image 601 (image x i ) can be encoded. In some cases, the finely tuned encoder 602 can encode image 601 (image x) using a single forward pass as follows. i ) can be encoded:

[0149]

number

[0150] In some cases, the finely tuned potential prior information 606 can be entropy-encoded under the model prior information 616 as follows:

[0151]

number

[0152]

number

[0153]

number

[0154]

number

[0155]

number

[0156]

number

[0157] Decoder 614 outputs latent code 604 (latent z i) approximate reconstructed image 620 (reconstruction

[0158]

number

[0159]

number

[0160] As mentioned above, the neural network compression system 600 can be trained by minimizing the RDM loss. In some cases, the rate can reflect the length of bitstream b (e.g., bitstreams 610 and / or 611), and the distortion is the input image 601 (image x i ) and reconstructed image 620 (reconstruction)

[0161]

number

[0162] In some examples, the loss for data point x can be minimized during inference as follows:

[0163]

number

[0164]

number

[0165] term-logp θ (x│z) can indicate and / or represent the strain D(x|z;θ). Term βlogp ψ (z) is latent R z The rate for transmitting (z;ψ) can be indicated and / or represented by the term βlogp ω (ψ,θ) is the finely tuned model update R ψ,θ This can indicate and / or represent the rate for transmitting (ψ,θ;ω).

[0166] In some cases, the model pre-information 616 may reflect the length of the bitrate overhead for sending the model update. In some examples, the bitrate for sending the model update may be described as follows:

[0167]

number

[0168]

number

[0169] In some cases, using the RDM loss function, the neural network compression system 600 updates the model only if the latent rate or distortion is reduced by at least the same number of bits.

[0170]

number

[0171]

number

[0172] The neural network compression system 600 can be trained end-to-end. In some cases, RDM loss can be minimized during end-to-end inference. In some examples, a certain amount of computation can be spent once (e.g., fine-tuning the model), and a high compression ratio can be subsequently obtained without incurring extra costs on the receiver. For example, a content provider may spend a large amount of computation to train and fine-tune the neural network compression system 600 over a wide range of videos delivered to numerous receivers. A highly trained and fine-tuned neural network compression system 600 can provide high compression performance for that video. Having spent a large amount of computation, the video provider can store updated parameters of the model pre-information and efficiently provide them to each receiver of the compressed video for decompression. The video provider can achieve a significant advantage in compression (and reduction of network and computational resources) with each transmission of video, which can far outweigh the initial computational cost of training and fine-tuning the model.

[0173] Due to the large number of pixels in video and images (e.g., high-resolution images), the training / learning and fine-tuning techniques described above can be very advantageous for video compression and / or high-resolution images. In some cases, complexity and / or decoder computation can be used as additional considerations for the overall system design and / or implementation. For example, a very small network that performs inference quickly can be fine-tuned. As another example, a cost term can be added due to the complexity of the receiver, which can force and / or make the model remove one or more layers. In some examples, machine learning can be used to learn prior information for more complex models to achieve even greater gains.

[0174] Model pre-design can include various attributes. In some examples, the implemented model pre-design has a high probability of submitting the model without any updates.

[0175]

number

[0176]

number

[0177]

number

[0178] Despite the acceleration of research and development, the implementation of such machine learning-based compression systems exhibits certain drawbacks. For example, neural compression algorithms require large neural networks, which can lead to increased computational and / or memory requirements. Furthermore, the optimal settings or parameters (e.g., weights) used in deep neural networks are dependent on the training dataset and may not perform well when used with different data. For example, a neural video codec trained on natural scenes may not perform well on animated scenes. Moreover, while a particular neural network can be fine-tuned based on each instance, sending updated and fine-tuned parameters to the decoder can significantly increase file size and / or bitrate.

[0179] As described above, systems and techniques for compressing and decompressing content using one or more machine learning (ML) systems capable of addressing the problems described herein are described herein. For example, aspects of the disclosure include machine learning systems (e.g., one or more neural networks, such as a rate-distortion autoencoder (RD-AE) or other neural networks) that can be trained to minimize the mean rate-distortion loss across a dataset of data points, such as image and / or video data points. In some examples, a set of global model parameters (e.g., weights) can be associated with the trained machine learning system. In some cases, training the machine learning system can include identifying parameters (e.g., a weight vector) associated with each iteration of training. For example, the weight vector may correspond to each training iteration, and the global model parameters may correspond to the weight vector of the trained neural network.

[0180] In some examples, training a machine learning system may involve determining a subspace or manifold of model parameters that has lower dimension than the entire parameter space. In some embodiments, the subspace or manifold of model parameters includes a portion of the weight vectors associated with each iteration of training. In some cases, the subspace or manifold of model parameters may be determined using principal component analysis (PCA). For example, PCA can be used to identify one or more directions and / or trajectories in the entire parameter space where the model parameters performed well during training of a neural network (e.g., where the loss function fell below a threshold). In some embodiments, sparse PCA can be used to reduce the size of the subspace of model parameters. In some examples, the subspace or manifold of model parameters may be shared with a neural network configured to decode encoded data.

[0181] In some examples, a machine learning system can be fine-tuned (e.g., trained, fitted) to input data that is compressed and sent to a receiving device containing a decoder (which may also contain a machine learning system). In some embodiments, the fine-tuning of a machine learning system can be performed using a subspace or manifold of model parameters. For example, fine-tuning a neural network may involve selecting an updated set of model parameters corresponding to weight vectors that are part of a subspace or manifold of model parameters (e.g., selecting the optimal weight vectors for an instance). In some cases, the updated set of parameters (e.g., the weight vectors selected during fine-tuning) can be associated with one or more subspace coordinates. In one example, one or more subspace coordinates may be relative to a set of global parameters (e.g., the subspace coordinates may correspond to trajectories relative to a set of global parameters).

[0182] In some embodiments, an updated set of parameters can be used to encode the input data. In some examples, the machine learning system may further include an arithmetic encoder (e.g., an arithmetic encoder, an arithmetic decoder, or a composite arithmetic encoder-decoder). The arithmetic encoder (e.g., an arithmetic encoder or a composite arithmetic encoder-decoder) can generate a bitstream containing a compressed version of the input data. In some cases, the bitstream may also contain compressed versions of one or more subspace coordinates corresponding to the updated set of model parameters. In some cases, a receiving device can receive the bitstream. The decoder in the receiving device can use one or more subspace coordinates to determine the updated model parameters of the neural network. The machine learning system of the decoder (e.g., a neural network such as RD-AE or other neural networks) can use the updated model parameters to decode the compressed input data.

[0183] In some embodiments, the systems and techniques disclosed herein can be used to improve compression performance by fitting a neural network to each data instance. In some examples, the systems and techniques disclosed herein can reduce the bitrate and / or file size for sending network parameter updates (e.g., a finely tuned neural network) to a decoder by selecting a finely tuned weight vector from a low-dimensional subspace.

[0184] Figure 7 shows an instance adaptation process implemented by an exemplary neural network compression system 700. In some embodiments, the neural network compression system 700 may include an encoder 710 and a decoder 714 that can be configured to perform compression and / or decompression of image, video, and / or audio data. In some cases, the encoder 710 and the decoder 714 may be part of the same device. In other cases, the encoder 710 and the decoder 714 may be part of different devices. In some examples, the encoder 710 and / or the decoder 714 may include, determine, or otherwise access one or more aspects of the shared knowledge 706 associated with the neural network compression system 700. In some cases, the shared knowledge 706 may include a network architecture (e.g., a deep learning neural network architecture), global model parameters (e.g., network weights), and / or subspace definitions (further described below).

[0185] In some examples, the neural network compression system 700 may be trained using one or more training datasets, which may include images, videos, audio, etc. (e.g., training data 702). For example, network training 704 may be implemented using the training data 702 to determine the parameters of the neural network compression system 700 (e.g., weights, biases, etc.). In some cases, network training 704 may include an iterative flow of the training data 702 through the neural network compression system 700 (e.g., using backpropagation training techniques). In some embodiments, the parameters of the trained neural network compression system 700 (e.g., weights, biases, etc.) may be called global model parameters.

[0186] In some embodiments, network training 704 can be used to generate, construct, determine, and / or define a subspace or manifold of network parameters. In some cases, the subspace or manifold of network parameters may be determined based on a set of model parameters (e.g., weights) associated with network training 704. For example, a weight vector (e.g., a set of network parameters) may be determined during each iteration of network training 704. In some examples, a weight vector associated with a performance threshold level during network training 704 can be used to define a subspace or manifold of network parameters. In one exemplary example, the performance threshold level may be determined based on a loss function associated with the data rate and / or a distortion metric (e.g., rate-distorted model rate (RDM) loss). In some embodiments, the subspace or manifold of network parameters may be defined based on "N" of the best-performing weight vectors during network training 704 (e.g., 100 best-performing weight vectors).

[0187] For example, a subspace or manifold of network parameters can be represented as a matrix M, which may be defined according to the following relationship given by equation (1), where θ corresponds to the optimized parameters of the compression network for a given instance (e.g., video), and θ global δ corresponds to global model parameters (e.g., weights, biases, etc. of the trained neural network that can be stored in the transmitting and / or receiving devices), M corresponds to a matrix that defines the subspace, and δ corresponds to a vector of subspace coordinates (e.g., updates that can be determined for each instance). θ = θ global +Mδ (1)

[0188] In some embodiments, the network parameter subspace M can be determined using machine learning methods. For example, a meta-learning machine learning algorithm (e.g., model-independent meta-learning (MAML)) may be implemented to determine the subspace M. In some embodiments, MAML may be used to train a neural network compression system 700 (e.g., a global model) along with a definition of the subspace M (e.g., a function that can map transmitted coordinates δ into the network parameter space).

[0189] Figure 10 shows an exemplary process 1000 for training a neural network compression system (e.g., system 700) and determining a subspace of network parameters. In block 1002, process 1000 may include initializing a machine learning algorithm. In some embodiments, the machine learning algorithm may correspond to a model-independent meta-learning (MAML) algorithm. In some cases, the initialization of the machine learning algorithm may include defining and / or initializing one or more parameters, constraints, variables, etc. For example, the initialization of the machine learning algorithm may include defining a training dataset ρ(x) having x instances. In some cases, the initialization of the machine learning algorithm may define a compression model f with weights θ. θ This may include defining (x). In some examples, the initialization of a machine learning algorithm is done by defining a hypernetwork θ=g with instance parameters δ and weights ψ. ψ This may include defining (δ). In some embodiments, the initialization of a machine learning algorithm may include defining a rate-distortion loss L(x,δ,ψ) which may include the model update rate (e.g., rate-distortion model loss). In some cases, the initialization of a machine learning algorithm may include the initialization of weights (e.g., ψ←ψ0), the batch size N of the training data, the learning rates α, β, and / or some optimizer steps n.

[0190] In block 1004, process 1000 may include zeroing the gradient. In some embodiments, zeroing the gradient can be expressed as Δψ←ψ0. In block 1006, method 1000 may include sampling an instance (e.g., input data). In some cases, sampling an instance can be expressed as follows:x i ~ρ(x);δ i ←0. In block 1008, process 1000 may perform instance fine-tuning. In some embodiments, instance fine-tuning is δ i ←δ i -α∇ δ L(x i ,δ i It can be expressed as ,ψ). In block 1010, process 1000 can determine whether k is less than or equal to n optimizer steps. If so, process 1000 can increment the value of index k, return to block 1008, and repeat the instance fine-tuning. If k is greater than n, the process can proceed to block 1012 and accumulate the hypernetwork gradient. In some examples, accumulating the hypernetwork gradient can be expressed as follows: Δψ←Δψ-β∇ ψ L(x i ;δ i ,ψ).

[0191] In block 1014, process 1000 can determine whether i is less than or equal to the N batch size. If so, process 1000 can increment the value of index i, return to block 1006, and repeat the operations from blocks 1006 to 1012. If i is greater than N, the process can proceed to block 1016 and perform a hypernetwork update. In some embodiments, performing a hypernetwork update can be expressed as ψ←ψ+Δψ. In block 1018, process 1000 can determine whether the model has converged (for example, based on the error range or loss function). If the model has not converged, process 1000 can return to block 1004 (for example, the zero-output gradient) and repeat the operations from blocks 1004 to 1016. If the model has converged, process 1000 can proceed to block 1020, where process 1000 can terminate.

[0192] Referring back to Figure 7, in some examples, network training 704 may include a principal component analysis (PCA) algorithm that can be used to determine a subspace M of the network parameters. For example, PCA can be used to determine one or more directions and / or trajectories in the total parameter space where the model parameters performed well during network training 704 (e.g., based on a loss function). In some examples, PCA can be used to reduce the dimension of the parameter space to its principal components. In some embodiments, PCA may be used to determine the subspace M by training a k-step compressed model and saving the values ​​of n network parameters after each step. In some examples, the corresponding network parameters may be arranged in a k×n matrix that may be called A. In some cases, singular value decomposition may be applied to matrix A. In some embodiments, the first m eigenvectors can be obtained according to singular value decomposition (e.g., sorted by decreasing the eigenvalues) and may be arranged in a k×m matrix. In some examples, the k×m matrix may correspond to matrix M (e.g., as used in equation (1)). In some cases, the matrix components M ij This can correspond to the i-th component of the j-th eigenvector of matrix A.

[0193] In some embodiments, sparse PCA can be used to reduce the size of a subspace M of network parameters (e.g., by applying sparsity constraints to input variables). In one exemplary example, sparse PCA can be implemented by first applying PCA as described above to determine a subspace matrix M. In some embodiments, sparse PCA can include reducing the size of M by maintaining that p components have the largest absolute values and setting all other components to a value of 0. In some cases, p can be selected such that p < m. In some examples, sparse PCA can be used to determine a k×m sized matrix having k×p non-zero entries. In some embodiments, the memory requirements for M can be substantially reduced by storing only the non-zero entries determined using sparse PCA.

[0194] FIG. 8 is a graph 800 showing an example of a one-dimensional subspace 806. As shown, graph 800 includes a number of training parameter points �04. In some embodiments, each of the training parameter points 804 can correspond to a weight vector (e.g., a set of network parameters) used or accessed during network training 704. Graph 800 also includes an optimal parameter point 802 that can correspond to a set of global model parameters of a fully trained neural network compression system 700.

[0195] As described above, in some examples, PCA can be used to process training parameter points 804 and determine subspaces 806. For example, during network training of a global model 704, PCA can be used (e.g., based on the weight vectors corresponding to the training iterations) to find the major principal components of the optimizer trajectory. In one example, the dataset on which PCA is performed is constructed by observing how the model unfolds during training. For example, at each training iteration, a snapshot of the model's parameters may be determined. Given a certain number of parameter snapshots (e.g., 1000 sets of parameters, 2000 sets of parameters, etc.), there are a number of data points on which PCA can be performed to determine the most important directions (corresponding to the major principal components) during training. In some embodiments, the major principal components of the optimizer trajectory can be used to define subspaces 806. In some cases, subspace 806 can correspond to a matrix (e.g., subspace matrix M) that parameterizes the directions from the optimal parameter points 802. Subspace 806 can be used to constrain the values ​​that the neural network can update (e.g., the neural network's parameters can only be updated to values ​​contained within the subspace). By restricting the neural network's parameter updates to a subspace of 806, the rate is significantly reduced while minimizing the impact on distortion (for example, by sending a set of subspace coordinates as described below, so that only changes along the subspace are sent to the receiver), and thus the overall coding performance of the system is improved.

[0196] In some cases, the subspace 806 may be defined during and / or after network training 704. For example, a manifold recognition type of training of a global model may be implemented to define the subspace 806 during training of the neural network compression system 700.

[0197] Although subspace 806 is shown as a one-dimensional subspace, those skilled in the art will recognize that the art can be used to define subspaces having any number of dimensions. In one exemplary example, the neural network compression system 700 may be associated with a parameter space containing millions of weights, and subspace 806 may have thousands of dimensions.

[0198] Referring back to Figure 7, the neural network compression system 700 may include an encoder 710. In some examples, the encoder 710 may be configured to receive, process, compress, and / or encode input data 708. In some cases, the input data 708 may include image data, video data, and / or audio data. In some embodiments, the encoder 710 may fine-tune global model parameters based on the input data 708 by selecting and using a modified set of parameters from a subspace.

[0199] In some cases, fine-tuning of global model parameters may be based on the bit size (e.g., rate cost) of the compressed version of the input data and / or the distortion between the input data and the reconstructed data generated from the compressed version of the input data. In one exemplary example, the encoder 710 may iteratively encode the input data 708 using a set of parameters that are part of a subspace or manifold of network parameters defined during network training 704. In some embodiments, the encoder 710 may select a modified set of parameters (e.g., from a subspace of network parameters) that optimizes the loss function.

[0200] In some examples, the encoder 710 can determine a model update corresponding to the difference between the global model parameters and the modified set of parameters. In some embodiments, the model update can correspond to a vector, a set of coordinates, a matrix, an equation, and / or any other suitable representation that can show one or more shifts of the global model parameters.

[0201] Figure 9 is a graph 900 showing a model update 906 within a subspace 902. In some cases, the subspace 902 corresponds to a subspace 806 defined during and / or after network training 704. In some aspects, the encoder 710 can be configured to operate using a set 904 of global model parameters. In some examples, the encoder 710 can fine-tune a set of global model parameters based on input data 708 (a particular instance being encoded). In some aspects, fine-tuning the set of global model parameters can include selecting a set 908 of modified parameters from within the subspace 902. In some cases, the modified set 908 of parameters from within the subspace 902 can be selected based on a loss function. In one exemplary example, the loss function can include a combination of the bitrate required to transmit all the information output and the distortion indicating the quality of the information. In some aspects, the loss function can be represented by a relationship given by the following equation (2), where the first term corresponds to the distortion measured as the mean squared error between the original data x and the reconstructed data x', the second term corresponds to the rate required to transmit the latent code z weighted by a hyperparameter β, and the last term corresponds to the rate required to transmit the model update vector δ weighted by the same coefficient β. L = MSE(x, x') - β log p(z) - β log p(δ) (2)

[0202] In some embodiments, the encoder 710 can determine the model update 906 by determining the difference between the global model parameters 904 and the set of modified parameters 908. In some cases, the model update 906 can include a set of subspace coordinates. For example, the set of subspace coordinates can indicate the delta between the subspace 902 and the model update 906. The decoder 714 can use the set of subspace coordinates to determine the set of modified parameters 908. For example, since the decoder already has the definition of the subspace 902, the decoder uses the set of subspace coordinates to reference the subspace 902 (e.g., the matrix defining the subspace) and can determine new parameters (e.g., weights, biases, etc.) to use for decoding the data instance corresponding to the input data 708. In some examples, the model update 906 (e.g., the set of subspace coordinates) can correspond to or reference a matrix of values that define the subspace 902 that can be used by the decoder 714 to determine the modified set of parameters 908. In one exemplary example, the modified set of parameters 908 can be determined as follows: Updated parameter point (e.g., 908) = (Global parameter point (e.g., 904)+Subspace matrix (e.g., 902)) * Subspace update (e.g., 906).

[0203] In one exemplary example, as follows, the three-dimensional parameter space can be represented by the matrix M, and the subspace coordinates can be represented by δ:

[0204]

Number

[0205] In some examples, the columns of matrix M can indicate subspace directions (for example, the first column can point diagonally between the first and second network parameters). In some embodiments, an update in which the first component δ is non-zero can modify the values ​​of the first and second neural network parameters. In some cases, the second column can be used to represent changes to the second and third network parameters. In some cases, the above model update δ can correspond to a change of +0.5 for the first network parameter, 0 for the second network parameter, and -0.5 for the third network parameter.

[0206] Referring back to Figure 7, in some cases, the encoder 710 can encode the input data 708 into latent variables using an optimized network (e.g., using a modified set of parameters from a subspace of network parameters). In some examples, the neural network compression system 700 can also include a probabilistic model (e.g., latent prior information 606) that can reversibly compress the latent variables. In some cases, the probabilistic model can generate a probability distribution over a set of latent variables that can represent the encoded data based on the input data 708.

[0207] In some embodiments, the encoder 710 can encode model updates (e.g., model updates 906 such as the set of subspace coordinates and / or matrix of values ​​described above) into latent variables (e.g., under model prior information). In some examples, the neural network compression system 700 may include an arithmetic encoder (e.g., arithmetic encoder 608) that can entropy encode the compressed model updates and / or compressed latent variables to generate one or more bitstreams (e.g., bitstream 712).

[0208] In some embodiments, the bitstream 712 may be sent to or transmitted to the decoder 714. In some examples, the decoder 714 may receive the bitstream 712 and decompress it to obtain latent variables and / or model updates (e.g., a set of subspace coordinates, a matrix of values, etc.). In some cases, the decoder 714 may determine a modified set of parameters based on the model updates and subspace definitions (e.g., determined by the decoder 714 from shared knowledge 706 received from the encoder 710). For example, the decoder 714 may determine a modified set of parameters by applying the model updates to the global model parameters. Referring to Figure 9, the decoder 714 may determine a modified set of parameters 908 by applying the model update 906 to the global model parameter 904. For example, the decoder 714 may determine the updated parameter of the global model parameter 904: updated parameter point (e.g., 908) = (global parameter point (e.g., 904) + subspace matrix (e.g., 902)). * Subspace update (e.g., 906).

[0209] In some examples, the decoder 714 can decode the latent variables corresponding to the input data 708 using a modified set of parameters (e.g., updated weights from the parameter subspace). In some embodiments, the decoder 714 uses the modified set of parameters to reconstruct the image 716 (reconstruction).

[0210]

number

[0211] Figure 11 shows an exemplary process 1100 for performing instance adaptive neural compression. In block 1102, process 1100 may include training a neural network to determine global model parameters. For example, a neural network compression system 700 may be trained using training data 702 and determine global model parameters (e.g., network weights) that may be included as part of shared knowledge 706.

[0212] In block 1104, process 1100 may include defining a subspace of the neural network parameters. In some embodiments, the subspace may be based on weight vectors implemented during the training phase. For example, training parameter points 804 may be used to define the subspace 806. In some cases, principal component analysis (PCA) and / or sparse PCA may be used to determine the subspace. For example, PCA may be used to determine one or more directions and / or trajectories within the full parameter space where the model parameters performed well during the training of the neural network.

[0213] In block 1106, process 1100 may include receiving input data for compression by a neural network compression system. For example, input data 708 (e.g., image data, video data, audio data) may be received by decoder 714. In block 1108, process 1100 may include using the input data to fine-tune neural network parameters in a subspace. In some embodiments, encoder 710 may use one or more weight vectors in the subspace to determine a modified set of parameters (e.g., fine-tuned parameters). In some examples, the modified set of parameters may be selected based on the performance of the encoder, measured by a loss function (e.g., rate and / or distortion).

[0214] In block 1110, process 1100 may include compressing the input data using a finely tuned neural network (for example, using a modified set of parameters from within the subspace). In block 1112, process 1100 may include determining parameter updates based on global parameter points and the finely tuned parameters. Referring to Figure 9, parameter update 906 may be determined based on a shift from the global parameter 904 in subspace 902 to the finely tuned parameter 908.

[0215] In block 1114, process 1100 may include sending compressed input data and parameter updates to a receiver. For example, encoder 710 may send compressed latent variables and compressed subspace coordinates to decoder 714. In some examples, an arithmetic encoder (e.g., arithmetic encoder 608) may entropy encode the compressed subspace coordinates and / or compressed latent variables to generate one or more bitstreams (e.g., bitstream 712).

[0216] Figure 12 shows an exemplary process 1200 for performing instance adaptive neural decompression. In block 1202, process 1200 may include receiving compressed input data and neural network parameter updates. For example, decoder 714 may receive a bitstream 712 which may include compressed latent variables and compressed subspace coordinates.

[0217] In block 1204, process 1200 may include determining updated neural network parameters based on parameter updates in a subspace. In some embodiments, the decoder 714 may determine updated neural network parameters (e.g., finely tuned parameters used to encode input data) by applying parameter updates (e.g., subspace coordinates) to global model parameters. For example, network update 906 may be used to determine updated neural network parameters 908 by shifting global model parameters 904 in subspace 902.

[0218] In block 1206, process 1200 may include decompressing the input data using updated network parameters. For example, updated neural network parameters 908 can be used to decompress compressed latent variables corresponding to the input data 708. In some embodiments, the decoder 714 may generate a reconstructed image 716 using the updated network parameters.

[0219] Figure 13 shows an exemplary process 1300 for performing instance-adaptive neural compression. In block 1302, process 1300 may include receiving input data for compression by a neural network compression system. In some embodiments, the neural network compression system may correspond to neural network compression system 600 and / or neural network compression system 700. In some cases, the input data may include image data, video data, and / or audio data (e.g., input data 708).

[0220] In block 1304, process 1300 can include determining a set of updated model parameters of the neural network compression system based on input data, where the set of updated model parameters is selected from a subspace of the model parameters. In some examples, the subspace of the model parameters includes a portion of a plurality of weight vectors. For example, the subspace of model parameters 806 can include a portion (e.g., a weight vector) of training parameter point 804. In some aspects, each of the plurality of weight vectors can correspond to a weight vector used during training of the neural network compression system. For example, each of training parameter points 804 can correspond to a weight vector (e.g., a set of training parameters) used or accessed during network training 704.

[0221] In some cases, a portion of the plurality of weight vectors can be determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-agnostic meta-learning (MAML). For example, PCA can be used to process training parameter point 804 to determine subspace 806. In some cases, SPCA can be used to reduce the size of subspace 806 by retaining components with the largest absolute values and setting all other components (e.g., subspace matrix M) to zero. In some aspects, a portion of the plurality of weight vectors can be determined using the MAML algorithm according to the operation of process 1000.

[0222] In some examples, process 1300 may include generating a set of global model parameters based on a training dataset used to train a neural network compression system, where one or more subspace coordinates corresponding to the updated set of model parameters are for the global set of model parameters. In some cases, the training dataset may correspond to training data 702. In some cases, training data 702 may be used to perform network training 704 and determine the optimal parameter points 802 (e.g., the set of global model parameters). In some examples, the updated parameter points 908 are for the global parameter points 904 (e.g., based on the subspace matrix 902 and subspace update 906).

[0223] In some examples, process 1300 may include tuning a set of global model parameters using input data, the set of global model parameters being tuned based on the bit size of the compressed version of the input data and the distortion between the input data and the reconstructed data generated from the compressed version of the input data. In some embodiments, the set of global model parameters may correspond to global parameter points 904 that can be tuned based on input 708. For example, the encoder 710 may iteratively encode the input data 708 using a set of parameters that are part of a subspace (e.g., subspace 902). In some examples, tuning a set of global parameters 904 may include selecting a modified set of parameters 908 based on a loss function (e.g., the loss function of equation (2)).

[0224] In block 1306, process 1300 may include generating at least one bitstream containing a compressed version of the input data and a compressed version of one or more subspace coordinates corresponding to the updated set of model parameters. For example, an arithmetic encoder (e.g., arithmetic encoder 608) may be used to entropically encode the compressed model update (e.g., subspace update 906) and the compressed latent variables into bitstream 712. In some cases, at least one bitstream may include a plurality of encoding quantization parameters used to quantize one or more subspace coordinates. For example, bitstream 712 may include quantization parameters used to quantize subspace update 906. In some embodiments, generating at least one bitstream may include entropically encoding one or more subspace coordinates using model prior information. For example, model prior information 616 may be used by arithmetic encoder 608 to entropically encode one or more subspace coordinates (e.g., subspace update 906).

[0225] In block 1308, process 1300 may include outputting at least one bitstream for transmission to a receiver. For example, bitstream 712 may be output for transmission to decoder 714. In some examples, process 1300 may include sending a subspace of model parameters to the receiver. For example, subspace 808 may be encoded as part of bitstream 712 and sent to decoder 714.

[0226] Figure 14 shows an exemplary process 1400 for performing instance adaptive neural decompression. In block 1402, process 1400 may include receiving a compressed version of the input data and a compressed version of one or more subspace coordinates. For example, decoder 714 may receive a bitstream 712 which may include a compressed version of the input 708 and a compressed version of the model update 906.

[0227] In block 1404, process 1400 may include determining an updated set of model parameters for the neural network compression system based on one or more subspace coordinates. For example, the updated set of model parameters may correspond to updated parameter points 908 determined based on the updated subspace 906. In some embodiments, determining the updated set of model parameters may include determining a shift from the global set of model parameters based on the subspace of the model parameters and one or more subspace coordinates. In some examples, the updated set of model parameters may be global parameter point 904 + (subspace matrix 902 * This can correspond to updated parameter points 908, which can be equivalent to subspace updates 906).

[0228] In some embodiments, the subspace of model parameters may include a portion of a plurality of weight vectors, each of which corresponds to a weight vector used during training of the neural network compression system. For example, the subspace 806 may include a portion of training parameter points 804, which can correspond to training parameter points used during training of the neural network compression system 700.

[0229] In block 1406, process 1400 may include processing the input data with an updated set of model parameters to generate a reconstructed version of the input data. For example, the input data 708 (compressed, e.g., into a bitstream 712) can be processed with updated parameter points 908 to generate a reconstructed image 716.

[0230] In some examples, process 1400 may include receiving a subspace of model parameters. For example, subspace 902 may be encoded as part of bitstream 712 and sent to decoder 714.

[0231] In some examples, the processes described herein (e.g., process 1000, process 1100, process 1200, process 1300, process 1400, and / or other processes described herein) may be executed by a computing device or apparatus. In one example, processes 1300 and / or 1400 may be executed by the transmitting device 410 of the system 400 illustrated in Figure 4. In another example, processes 1300 and / or 1400 may be executed by a computing device relating to the system 400 shown in Figure 4 or the computing system 1500 shown in Figure 15.

[0232] The computing device may include any suitable device such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capacity to perform the processes described herein, including process 1100, process 1200, and / or other processes described herein. In some cases, the computing device or apparatus may include a variety of components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0233] Components of a computing device may be implemented in a circuit. For example, a component may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), may include electronic circuits or other electronic hardware, and / or may be implemented using them, and / or may include computer software, firmware, or any combination thereof to perform the various operations described herein, and / or may be implemented using them.

[0234] Processes 1300 and 1400 are illustrated as logical flow diagrams, and their operations represent sequences of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be interpreted as limiting, and any number of described operations can be combined in any order and / or in parallel to implement a process.

[0235] Furthermore, processes 1300, 1400, and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively by hardware or in combination thereof on one or more processors. As described above, the code may be stored on a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0236] Figure 15 shows an example of a system for implementing a particular aspect of the present technology. More specifically, Figure 15 shows an example of a computing system 1500, which can be, for example, an internal computing system, a remote computing system, a camera, or any computing device comprising any of these components, where the components of the system communicate with each other using a connection 1505. The connection 1505 can be a physical connection using a bus, or a direct connection to a processor 1510 in a chipset architecture, for example. The connection 1505 can also be a virtual connection, a network connection, or a logical connection.

[0237] In some embodiments, the computing system 1500 is a distributed system in which the functions described herein can be distributed across a data center, multiple data centers, a peer network, and so on. In some embodiments, one or more of the described system components represent many components, each performing some or all of the functions described for the components. In some embodiments, the components may be physical or virtual devices.

[0238] An exemplary system 1500 includes at least one processing unit (CPU or processor) 1510 and connections 1505 that connect various system components to the processor 1510, including system memory 1515 such as read-only memory (ROM) 1520 and random access memory (RAM) 1525. The computing system 1500 may include a high-speed memory cache 1512 that is directly connected to the processor 1510, very close to the processor, or integrated as part of the processor.

[0239] The processor 1510 may include a dedicated processor in which any general-purpose processor, and hardware or software services such as services 1532, 1534, and 1536 stored in memory device 1530, as well as software instructions, are incorporated into the actual processor design and configured to control the processor 1510. The processor 1510 may essentially be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. The multicore processor may be symmetrical or asymmetrical.

[0240] To enable user interaction, the computing system 1500 includes an input device 1545 which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, and speech. The computing system 1500 may also include an output device 1535 which may be one or more of several output mechanisms. In some cases, a multimode system may allow the user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1540 which can generally control and manage user input and system output.

[0241] Communication interfaces include audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, Bluetooth® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, and 802.11. It is possible to perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communications (VLC), worldwide interoperability for microwave access (WiMAX), infrared (IR) communications wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.

[0242] The communication interface 1540 may also include one or more GNSS receivers or transceivers used to determine the position of the computing system 1500 based on the reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite Systems (GNSS). GNSS systems include, but are not limited to, the US Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's Beidou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no restrictions on operation on any particular hardware configuration, and therefore the basic features described herein may be readily replaced for improved hardware or firmware configurations as they are developed.

[0243] The memory device 1530 may be a non-volatile and / or non-temporary and / or computer-readable memory device, such as a magnetic cassette, flash memory card, solid-state memory device, digital multipurpose disk, cartridge, floppy disk, flexible disk, hard disk, magnetic tape, magnetic strip / stripe, any other magnetic storage medium, flash memory, memory storage, any other solid-state memory, compact disc read-only memory (CD-ROM) optical disc, rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, Blu-ray disc (BDD) optical disc, holographic optical disc, another optical medium, secure digital (SD) card, micro secure digital (microSD) card, Memory Stick® card, smart card chip, EMV chip, subscriber identity module (SIM) card, mini / micro / nano / pico SIM card, or another integrated circuit (integrated) Circuits, ICs, chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory.This may include a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0244] The storage device 1530 may include software services, servers, services, etc., which cause the system to perform functions when code defining such software is executed by the processor 1510. In some embodiments, a hardware service that performs a particular function may include software components stored in a computer-readable medium with respect to the necessary hardware components, such as the processor 1510, connection 1505, and output device 1535, in order to perform the function. The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. The computer-readable medium may also include non-transient media on which data is stored and which do not contain carrier waves and / or transient electronic signals that propagate wirelessly or via wired connections. Examples of non-transient media include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored on the computer-readable medium that can represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any preferred means, including memory sharing, message passing, token passing, network transmission, etc.

[0245] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals, such as bitstreams. However, as used to refer to them, non-transient computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0246] Specific details are provided in the above description to give a complete understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments can be practiced without these specific details. For clarity of description, in some cases the technology may be presented as including individual functional blocks comprising a device, device components, and software, or steps or routines in a method embodied in a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments.

[0247] Individual embodiments may be described above as processes or methods shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of steps may be rearranged. A process terminates when its operation is complete, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or main function.

[0248] The processes and methods described above may be implemented using computer-executable instructions stored in or otherwise available from computer-readable media. Such instructions may include instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform some functions or groups of functions, or otherwise configure a general-purpose computer, a dedicated computer, or a processing device to perform some functions or groups of functions. The portion of computer resources used may be accessible via a network. Computer-executable instructions may be binary or intermediate format instructions, such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created in the methods described above include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and network-connected storage devices.

[0249] Devices that implement processes and methods in accordance with these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. Processors may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-space personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may also, as a further example, be implemented on circuit boards of different chips or on different processes running in a single device.

[0250] Instructions, a medium for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0251] In the above description, aspects of the present application are described with reference to their specific embodiments, but those skilled in the art will recognize that the application is not limited thereto. Therefore, while exemplary embodiments of the present application are described in detail herein, it should be understood that the concepts of the present invention can be embodied and employed in various other ways, and that, unless limited by the prior art, the appended claims are intended to be interpreted as including such variations. The various features and aspects of the present application described above may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader scope of this specification. Therefore, this specification and the drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods have been described in a particular order. In alternative embodiments, it should be understood that the methods may be performed in a different order than described.

[0252] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this specification.

[0253] When components are described as “configured” to perform a particular operation, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0254] The phrase "combined" refers to any component that is physically connected to another component, either directly or indirectly, and / or communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0255] The wording of a claim that includes "at least one of" and / or "one or more" of a set, or any other wording, indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, the wording of a claim that includes "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the wording of a claim that includes "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording "at least one of" and / or "one or more" of a set does not limit the set to items enumerated within the set. For example, the wording of a claim that includes "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not enumerated within the set of A and B.

[0256] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps have been schematically described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functions in various ways for each specific application, but such a decision on implementation should not be construed as a departure from the scope of this application.

[0257] The technologies described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technologies may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices having multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. When implemented in software, the technology may be at least partially implemented by a computer-readable data storage medium containing program code including instructions, which, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media such as random access memory (RAM) including synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (non-volatile RAM, NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, and magnetic or optical data storage media. The technology may also be at least partially implemented by computer-readable communication media, such as propagating signals or waves, which can carry or communicate program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0258] The program code may be executed by a processor which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to implement any of the technologies described herein. Instead of a general-purpose processor being a microprocessor, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. Accordingly, the term “processor” as used herein may refer to any of the above structures, any combination thereof, or any other structure or device suitable for implementing the technologies described herein.

[0259] Examples of the disclosure include the following: Embodiment 1: An apparatus comprising at least one memory and at least one processor coupled to the at least one memory. The apparatus is configured to receive input data for compression by a neural network compression system, determine an updated set of model parameters for the neural network compression system, selected from a subspace of model parameters, based on the input data, generate at least one bitstream including a compressed version of the input data and compressed versions of one or more subspace coordinates corresponding to the updated set of model parameters, and output at least one bitstream for transmission to a receiver.

[0260] Embodiment 2: The apparatus according to Embodiment 1, wherein the subspace of the model parameters includes a portion of a plurality of weight vectors.

[0261] Embodiment 3: The apparatus according to Embodiment 2, wherein each of the multiple weight vectors corresponds to a weight vector used during training of the neural network compression system.

[0262] Embodiment 4: The apparatus according to Embodiment 2 or 3, wherein a portion of a plurality of weight vectors is determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-independent meta-learning (MAML).

[0263] Embodiment 5: The apparatus according to any one of Embodiments 1 to 4, wherein at least one processor is further configured to generate a set of global model parameters based on a training dataset used to train a neural network compression system, and one or more subspace coordinates corresponding to the updated set of model parameters are relative to the set of global model parameters.

[0264] Embodiment 6: The apparatus according to Embodiment 5, wherein at least one processor is further configured to adjust a set of global model parameters using input data to determine an updated set of model parameters from a subspace of model parameters, the set of global model parameters being adjusted based on the bit size of a compressed version of the input data and the distortion between the input data and reconstructed data generated from the compressed version of the input data.

[0265] Embodiment 7: The apparatus according to any one of Embodiments 1 to 6, wherein at least one processor is further configured to quantize one or more subspace coordinates to generate one or more quantized subspace coordinates, and at least one bitstream includes compressed versions of one or more quantized subspace coordinates.

[0266] Embodiment 8: The apparatus according to Embodiment 7, wherein at least one bitstream includes a plurality of coding quantization parameters used to quantize one or more subspace coordinates.

[0267] Embodiment 9: The apparatus according to any one of embodiments 1 to 8, wherein at least one processor is further configured to entropy encode one or more subspace coordinates using model prior information in order to generate at least one bitstream.

[0268] Embodiment 10: The apparatus according to any one of embodiments 1 to 9, wherein at least one processor is further configured to send a subspace of model parameters to a receiver.

[0269] Embodiment 11: A method for performing any of the operations described in Embodiments 1 to 10.

[0270] Embodiment 12: A computer-readable storage medium that, when executed, stores instructions causing one or more processors to perform any of the operations described in Embodiments 1 to 10.

[0271] Embodiment 13: An apparatus comprising means for performing any of the operations of Embodiments 1 to 10.

[0272] Embodiment 14: An apparatus comprising at least one memory and at least one processor coupled to at least one memory. The at least one processor is configured to receive a compressed version of input data and a compressed version of one or more subspace coordinates, to determine an updated set of model parameters for a neural network compression system based on the one or more subspace coordinates, and to process the input data using the updated set of model parameters to generate a reconstructed version of the input data.

[0273] Embodiment 15: The apparatus according to Embodiment 14, wherein at least one processor is further configured to determine a shift from a global set of model parameters based on a subspace of the model parameters and one or more subspace coordinates, in order to determine an updated set of model parameters.

[0274] Embodiment 16: The apparatus according to Embodiment 15, wherein the subspace of model parameters includes a portion of a plurality of weight vectors, each of which corresponds to a weight vector used during training of the neural network compression system.

[0275] Embodiment 17: The apparatus according to Embodiment 16, wherein some of the weight vectors are determined using principal component analysis (PCA).

[0276] Embodiment 18: The apparatus according to any one of embodiments 15 to 17, wherein at least one processor is further configured to receive a subspace of model parameters.

[0277] Aspect 19: A method for performing any of the operations described in Aspects 14 to 18.

[0278] Embodiment 20: A computer-readable storage medium that, when executed, stores instructions causing one or more processors to perform any of the operations described in Embodiments 14 to 18.

[0279] Embodiment 21: An apparatus comprising means for performing any of the operations described in Embodiments 14 to 18.

Claims

1. A method for processing image data, The process involves receiving input data for compression by a neural network compression system, Determining an updated set of model parameters for the neural network compression system based on the input data, wherein the updated set of model parameters is selected from a subspace of model parameters having a lower dimension than the entire parameter space. The method for generating at least one bitstream comprising a compressed version of the input data and a compressed version of one or more subspace coordinates corresponding to the updated set of model parameters, wherein the compressed version of the input data is a bitstream encoded using the updated set of model parameters, and the compressed version of the subspace coordinates is a bitstream encoded with information indicating the position of the updated model parameters in the subspace of the model parameters, Outputting at least one bitstream for transmission to the receiver, Methods that include...

2. The method according to claim 1, wherein the subspace of the model parameters includes a portion of a plurality of weight vectors.

3. The method according to claim 2, wherein each of the plurality of weight vectors corresponds to a weight vector used during training of the neural network compression system.

4. The method according to claim 2, wherein a portion of the plurality of weight vectors is determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-independent meta-learning (MAML).

5. The method according to claim 1, further comprising generating a set of global model parameters based on a training dataset used to train the neural network compression system, wherein the one or more subspace coordinates corresponding to the updated set of model parameters are relative to the set of global model parameters.

6. Determining the updated set of model parameters from the subspace of the model parameters is The method according to claim 5, comprising adjusting the set of global model parameters using the input data, wherein the set of global model parameters is adjusted based on the bit size of the compressed version of the input data and the distortion between the input data and the reconstructed data generated from the compressed version of the input data.

7. The method according to claim 1, further comprising quantizing one or more subspace coordinates to generate one or more quantized subspace coordinates, wherein at least one bitstream includes a compressed version of the one or more quantized subspace coordinates.

8. The method according to claim 7, wherein the at least one bitstream includes a plurality of coding quantization parameters used to quantize the one or more subspace coordinates.

9. Generating the aforementioned at least one bitstream is The method according to claim 1, comprising entropy coding one or more subspace coordinates using model prior information.

10. The method according to claim 1, further comprising sending the subspace of the model parameters to the receiver.

11. At least one memory, At least one processor coupled to the at least one memory, A device including, wherein the at least one processor, The neural network compression system receives input data for compression. An updated set of model parameters for the neural network compression system is determined based on the input data, and the updated set of model parameters is selected from a subspace of model parameters having a lower dimension than the entire parameter space. At least one bitstream is generated, comprising a compressed version of the input data and a compressed version of one or more subspace coordinates corresponding to the updated set of model parameters, wherein the compressed version of the input data is a bitstream encoded using the updated set of model parameters, and the compressed version of the subspace coordinates is a bitstream encoded with information indicating the position of the updated model parameters in the subspace of the model parameters. Outputting the at least one bitstream for transmission to the receiver, A device configured in such a way.

12. The apparatus according to claim 11, wherein the subspace of the model parameters includes a portion of a plurality of weight vectors.

13. The apparatus according to claim 12, wherein each of the plurality of weight vectors corresponds to a weight vector used during training of the neural network compression system.

14. The apparatus according to claim 12, wherein a portion of the plurality of weight vectors is determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-independent meta-learning (MAML).

15. The aforementioned at least one processor, The apparatus according to claim 11, further configured to generate a set of global model parameters based on a training dataset used to train the neural network compression system, wherein the one or more subspace coordinates corresponding to the updated set of model parameters are relative to the set of global model parameters.

16. To determine the updated set of model parameters from the subspace of the model parameters, at least one processor, The apparatus according to claim 15, further configured to adjust the set of global model parameters using the input data, wherein the set of global model parameters is adjusted based on the bit size of the compressed version of the input data and the distortion between the input data and the reconstructed data generated from the compressed version of the input data.

17. The aforementioned at least one processor, The apparatus according to claim 11, further configured to quantize one or more subspace coordinates to generate one or more quantized subspace coordinates, wherein at least one bitstream includes compressed versions of the one or more quantized subspace coordinates.

18. The apparatus according to claim 17, wherein the at least one bitstream includes a plurality of encoding quantization parameters used to quantize one or more subspace coordinates.

19. To generate the at least one bitstream, the at least one processor, The apparatus according to claim 11, further configured to entropy encode one or more subspace coordinates using model prior information.

20. The aforementioned at least one processor, The apparatus according to claim 11, further configured to send the subspace of the model parameters to the receiver.

21. A method for processing image data, The system receives a compressed version of the input data and a compressed version of one or more subspace coordinates, wherein the subspace coordinates are information indicating the position of updated model parameters in a subspace of model parameters having a lower dimension than the total parameter space for a neural network compression system, the compressed version of the input data is a bitstream encoded using the updated set of model parameters, and the compressed version of the subspace coordinates is a bitstream encoded with information indicating the position of updated model parameters in the subspace of model parameters. Based on the aforementioned one or more subspace coordinates, determine the updated set of model parameters for the neural network compression system. To generate a reconstructed version of the input data, the input data is processed using the updated set of model parameters, Methods that include...

22. Determining the updated set of model parameters is The method according to claim 21, comprising determining a shift from a global set of model parameters based on a subspace of model parameters and the coordinates of one or more subspaces.

23. The method according to claim 22, wherein the subspace of the model parameters includes a portion of a plurality of weight vectors, each of which corresponds to a weight vector used during training of the neural network compression system.

24. The method according to claim 23, wherein a portion of the plurality of weight vectors is determined using principal component analysis (PCA).

25. The method according to claim 22, further comprising receiving a subspace of the model parameters.

26. At least one memory, At least one processor coupled to the at least one memory, A device including, wherein the at least one processor, The system receives a compressed version of the input data and a compressed version of one or more subspace coordinates, where the subspace coordinates are information indicating the position of updated model parameters in a subspace of model parameters having a lower dimension than the total parameter space for the neural network compression system; the compressed version of the input data is a bitstream encoded using the updated set of model parameters; and the compressed version of the subspace coordinates is a bitstream encoded with information indicating the position of updated model parameters in a subspace of model parameters. Based on the one or more subspace coordinates, the updated set of model parameters for the neural network compression system is determined. A device configured to process the input data using the updated set of model parameters in order to generate a reconstructed version of the input data.

27. To determine the updated set of model parameters, at least one processor, The apparatus according to claim 26, further configured to determine a shift from a set of global model parameters based on a subspace of model parameters and the coordinates of one or more subspaces.

28. The apparatus according to claim 27, wherein the subspace of the model parameters includes a portion of a plurality of weight vectors, each of which corresponds to a weight vector used during training of the neural network compression system.

29. The apparatus according to claim 28, wherein a portion of the plurality of weight vectors is determined using principal component analysis (PCA).

30. The aforementioned at least one processor, The apparatus according to claim 27, further configured to receive a subspace of the model parameters.

Images