Machine Learning Based Flow Decision for Video Coding

Abstract

Systems and techniques for processing video data are described herein. In some aspects, the method may include obtaining input video data by a machine learning system. The input video data includes one or more luminance components of a current frame. The method may include determining, by the machine learning system, motion information for the luminance component of the current frame and motion information for one or more chrominance components of the current frame using the luminance component of the current frame. In some examples, the method may include determining the motion information for the luminance component based on a luma component of the current frame and at least one restored luma component of a previous frame. In some examples, the method may further include determining motion information for the chrominance components of the current frame using the motion information determined for the luminance component of the current frame.

Description

[Technical field]

[0001]

[0001] The present disclosure generally relates to image and video coding, including encoding (or compression) and decoding (decompression) of images and / or videos. For example, aspects of the present disclosure relate to techniques for determining flow information of luma and chroma components of one or more image frames or pictures (e.g., video frames / pictures). [Background technology]

[0002]

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes large amounts of data to meet the demands of consumers and video providers. For example, consumers of video data desire high quality video, including high fidelity, resolution, frame rate, etc. As a result, the large amounts of video data required to meet these demands strain communication networks and devices that process and store the video data.

[0003]

[0003] Video coding techniques can be used to compress video data. The goal of video coding is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation to the video quality. As ever-evolving video services become available, encoding techniques with better coding efficiency are needed. Summary of the Invention

[0004]

[0004] Systems and techniques are described for coding (e.g., encoding and / or decoding) image and / or video content using one or more machine learning systems. According to at least one example, a method for processing video data is provided. The method includes obtaining, by the machine learning system, input video data including at least one luminance component of a current frame, and determining, by the machine learning system, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using the at least one luminance component of the current frame.

[0005] In another example, an apparatus is provided for processing video data that includes at least one memory (e.g., configured to store data, such as virtual content data, one or more images, etc.) and one or more processors coupled to the at least one memory (e.g., implemented in circuitry). The one or more processors are configured and capable of using a machine learning system to obtain input video data including at least one luminance component of a current frame, and to determine, using the machine learning system, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame.

[0006]

[0006] In another example, a non-transitory computer-readable medium is provided that stores instructions that, when executed by one or more processors, cause the one or more processors to use a machine learning system to obtain input video data including at least one luminance component of a current frame, and to determine, using the machine learning system, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using the at least one luminance component of the current frame.

[0007] In another example, an apparatus for processing video data is provided that includes means for obtaining input video data including at least one luminance component of a current frame, and means for determining, using the at least one luminance component of the current frame, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame.

[0008]

[0008] In some aspects, one or more of the above methods, apparatus, and computer-readable media further comprise determining a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame by a machine learning system using motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, and determining one or more inter-frame predictions for the current frame using the warping parameter for at least one luminance component of the current frame and the one or more warping parameters for one or more chrominance components of the current frame.

[0009]

[0009] In some aspects, one or more interframe predictions are determined at least in part by applying an interpolation operation using a warping parameter of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame.

[0010] In some aspects, the interpolation operation includes a trilinear interpolation operation.

[0011]

[0011] In some aspects, the warping parameters of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame include space-scale flow (SSF) warping parameters.

[0012] In some aspects, the SSF warping parameters include a learned scale-flow vector.

[0013]

[0013] In some aspects, to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using at least one luminance component of the current frame, one or more of the above methods, apparatus, and computer-readable media further comprise determining motion information for at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame, and determining motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame.

[0014]

[0014] In some aspects, motion information for one or more chrominance components of a current frame is determined using a convolutional layer of a machine learning system.

[0015]

[0015] In some aspects, to determine motion information for one or more chrominance components of the current frame, one or more of the above-mentioned methods, apparatus, and computer-readable media further comprise sampling the motion information determined for at least one luminance component of the current frame.

[0016] In some aspects, the current frame includes a video frame.

[0017] In some aspects, the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component.

[0018] In some aspects, the current frame has a luminance-chrominance (YUV) format. In some cases, the YUV format is a YUV 4:2:0 format.

[0019] In some aspects, the apparatus described herein may comprise or be part of a mobile device (e.g., a mobile phone or so-called “smartphone”, tablet computer, or other type of mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device of the vehicle), or other device. In some aspects, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more videos including one or more images and / or video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to transmit one or more video frames and / or syntax data over a transmission medium to at least one device. In some aspects, the processor may include a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0020]

[0020] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter, which subject matter should be understood with reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0021]

[0021] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0022]

[0022] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0023] [Figure 1]

[0023] FIG. 1 illustrates an example implementation of a system on a chip (SOC). [Figure 2A]

[0024] FIG. 1 is a diagram showing an example of a fully connected neural network. [Figure 2B]

[0025] FIG. 1 is a diagram showing an example of a locally connected neural network. [Figure 2C]

[0026] FIG. 1 illustrates an example of a convolutional neural network. [Figure 2D]

[0027] A detailed example of a deep convolutional network (DCN) designed to recognize visual features from images. [Diagram 3]

[0028] A block diagram showing a deep convolutional network (DCN). [Figure 4]

[0029] FIG. 1 illustrates an example of a system including a device operable to perform image and / or video coding (encoding and decoding) using a neural network-based system, according to some examples. [Diagram 5]

[0030] FIG. 1 illustrates an example of an end-to-end neural network-based image and video coding system for input having a Red-Green-Blue (RGB) format, in accordance with some examples. [Figure 6]

[0031] FIG. 1 illustrates an example of a spatial-scale flow (SSF) neural network architecture configured to process one or more luminance-chrominance (YUV) input formats (e.g., a 4:2:0 YUV input format), which may be part of an end-to-end neural network-based image and video coding system, according to some examples. [Figure 7A]

[0032] FIG. 1 illustrates an example of a machine learning based flow engine operating on luma input, according to some examples. [Figure 7B]

[0033] 4A-4C are diagrams illustrating an example of subsampling of luma motion information to obtain chroma motion information, in accordance with some examples. [Figure 8A]

[0034] FIG. 1 illustrates an example of a machine learning based architecture with YUV (e.g., YUV4:2:0) residuals, in accordance with some examples. [Figure 8B]

[0035] FIG. 1 illustrates an example operation of a 1×1 convolutional layer, in accordance with some examples. [Figure 9]

[0036] FIG. 1 illustrates an example of a machine learning based architecture (e.g., an end-to-end neural network based image and video coding system) that works directly with a YUV input (Y, U, and V), such as a YUV4:2:0 input, in accordance with some examples. [Figure 10]

[0037] FIG. 1 illustrates another example of a machine learning based architecture (e.g., an end-to-end neural network based image and video coding system) that works directly with a YUV input (Y, U, and V), such as a YUV4:2:0 input, according to some examples. [Figure 11]

[0038] 1 is a flow diagram illustrating an example of a process for processing video data, according to some examples. [Figure 12]

[0039] FIG. 1 illustrates an example computing device architecture for an example computing device capable of implementing various techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024]

[0040] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are described to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not limiting.

[0025]

[0041] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0026]

[0042] Digital video data can contain large amounts of data, especially as the demand for high quality video data continues to grow. For example, consumers of video data typically desire increasingly higher quality video having high fidelity, resolution, frame rates, etc. However, the large amounts of video data required to meet such demands can place a significant strain on communication networks, as well as devices that process and store the video data.

[0027]

[0043] Various techniques may be used to code video data. Video coding may be performed according to a particular video coding standard. Exemplary video coding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding, and Versatile Video Coding (VVC). Video coding often uses prediction methods such as inter-prediction or intra-prediction that exploit redundancy present in a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality. As the demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0028]

[0044] Machine learning (ML) based systems may be used to perform image and / or video coding. In general, ML is a subset of artificial intelligence (AI). ML systems may include algorithms and statistical models that computer systems may use to perform various tasks by relying on patterns and inferences without explicit instructions. One example of an ML system is a neural network (also called an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks may be used in various applications and / or devices, such as image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, among others.

[0029]

[0045] Individual nodes in a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, each node's input data may be multiplied by a corresponding weight value, and the products may be summed. The sums of the products may be adjusted by an optional bias, and an activation function is applied to the result, resulting in the node's output signal or "output activation" (sometimes called an activation map or feature map). The weight values ​​may initially be determined by an iterative flow of training data through the network (e.g., the weight values ​​are established during a training phase in which the network learns how to discriminate between certain classes by input data characteristics typical of those classes).

[0030]

[0046] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and multi-layer perceptron (MLP) neural networks, among others. For example, convolutional neural networks (CNNs) are a type of feed-forward artificial neural network. Convolutional neural networks may include a collection of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) that collectively tile the input space. RNNs operate on the principle of saving the output of a layer and feeding this output back to the input to help predict the outcome of the layer. GANs are a form of generative neural network that can learn patterns in the input data so that the neural network model can generate new synthetic outputs that can reasonably be from the original dataset. GANs can include two neural networks operating together, including a generative neural network that generates the synthesized output, and a discriminative neural network that evaluates the output for reliability. In MLP neural networks, data may be fed into an input layer, and one or more hidden layers provide a level of abstraction to the data. Predictions may then be made on the output layer based on the abstracted data.

[0031]

[0047] In a hierarchical neural network architecture (called a deep neural network when there are multiple hidden layers), the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input of a third layer of artificial neurons, and so on. A CNN may be trained, for example, to recognize a hierarchy of features. Computations in a CNN architecture may be distributed across a population of processing nodes that may be organized in one or more computational chains. These multi-layered architectures may be trained one layer at a time and may be fine-tuned using backpropagation.

[0032]

[0048] Deep learning-based systems are often designed as a combination of an autoencoder sub-network (encoder sub-network) and a second sub-network (decoder sub-network) involved in learning a probabilistic model on the quantization latent used for entropy coding (sometimes also called a hyperprior network). In some cases, there may be other sub-networks of the decoder. Such a deep learning-based system architecture can be seen as a combination of a transform plus quantization module (or encoder sub-network) and an entropy modeling sub-network module.

[0033]

[0049] Most existing deep learning-based architectures for video compression are designed to operate with non-subsampled input formats, such as RGB, YUV4:4:4, or other non-subsampled input formats. However, video coding standards such as HEVC and VVC are designed to support the YUV4:2:0 color format within their respective main profiles. In order to support the 4:2:0 YUV format, deep learning-based architectures designed to operate with non-subsampled input formats need to be modified.

[0034]

[0050] Systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to as "systems and techniques") are described herein to provide ML-based systems (e.g., deep learning-based systems) that can use one color component of one or more frames (e.g., video frames) to estimate information about a color component and another color component of the frame. In some aspects, the ML-based system may be designed to process input data having a luminance-chrominance (YUV) input format. In such aspects, the ML-based system may use luma components of both a current frame and a previously restored frame (e.g., restored by the ML-based system) to estimate motion information (e.g., flow information such as optical flow information) for both the luma component and one or more chroma components. In some cases, after learning the motion information of the luma component, a convolutional layer with downsampling may be used to learn the motion information (e.g., flow information) of one or more chroma components. In some cases, the motion information of one or more chroma components may be obtained by directly subsampling the motion information of the luma component (e.g., without using a convolutional layer). Such techniques can be performed for all components of a frame. Using such techniques, an ML-based system can determine chroma motion information (e.g., flow information) without requiring latent data or chroma information coded as part of the bitstream (e.g., reducing the need to transmit side information along with the chroma information).

[0035]

[0051] As mentioned above, the ML-based system may be designed to process input data having a YUV input format. The YUV format includes a luminance channel (Y) and a pair of chrominance channels (U and V). The U channel may be referred to as a chrominance (or chroma)-blue channel, and the U channel may be referred to as a chrominance (or chroma)-red channel. In some cases, the luminance (Y) channel or component may also be referred to as a luma channel or component. In some cases, the chrominance (U and V) channels or components may also be referred to as chroma channels or components. The YUV input format may include YUV4:2:0, YUV4:4:4, YUV4:2:2, among others. In some cases, the systems and techniques described herein may be designed to process other input formats, such as a Y-chroma blue (Cb)-chroma red (Cr) (YCbCr) format, a red-green-blue (RGB) format, and / or other formats. The ML-based system described herein may encode and / or decode video data including a standalone frame (also referred to as an image) and / or multiple frames.

[0036]

[0052] Further details and additional aspects of the disclosure are described with respect to the figures.

[0037]

[0053] 1 illustrates an exemplary implementation of a system-on-chip (SOC) 100 that may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Among other information, parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., neural networks with weights), delays, frequency bin information, task information, may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, and / or may be distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from the memory block 118.

[0038]

[0054] The SOC 100 may also include additional processing blocks adapted for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 that may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and a multimedia processor 112 that may, for example, detect and recognize gestures. In one implementation, the NPU is implemented within the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120 that may include a global positioning system.

[0039]

[0055] The SOC 100 may be based on an ARM instruction set. In one aspect of the disclosure, the instructions loaded into the CPU 102 may comprise code for searching a stored multiplication result in a look-up table (LUT) corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also comprise code for disabling a multiplier during a multiplication operation of the multiplication product when a look-up table hit of the multiplication product is detected. Additionally, the instructions loaded into the CPU 102 may comprise code for storing the calculated multiplication product of the input value and the filter weight when a look-up table miss of the multiplication product is detected.

[0040]

[0056] SOC 100 and / or its components may be configured to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video coding) using machine learning techniques in accordance with aspects of the disclosure described herein. By using deep learning architectures to perform video compression and / or decompression, aspects of the disclosure can increase the efficiency of video compression and / or decompression on a device. For example, a device using the described video coding techniques can compress video more efficiently using machine learning based techniques and transmit the compressed video to another device, which can decompress the video more efficiently using the machine learning based techniques described herein.

[0041]

[0057] As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processing is performed by hidden nodes in one or more hidden layers, and output is generated via output nodes in the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network may include a feature map or activation map that may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. The nodes may include one or more weights that are used to indicate the importance of one or more nodes of the layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers being used to determine simple, low-level characteristics of the input, and later layers building a hierarchy of more complex, abstract characteristics.

[0042]

[0058] Deep learning architectures can learn hierarchies of features. For example, when presented with visual data, a first layer can be trained to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer can be trained to recognize the spectral power of specific frequencies. A second layer, which takes as input the output of the first layer, can be trained to recognize combinations of features, such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers can be trained to represent complex shapes in visual data, or words in auditory data. Even higher layers can be trained to recognize common visual objects or spoken phrases.

[0043]

[0059] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, motor vehicle classification can benefit from first learning to recognize wheels, windshields, and other features. These features may be combined in different ways in higher layers to recognize cars, trucks, and planes.

[0044]

[0060] Neural networks may be designed with various connection patterns. In feedforward networks, information is passed from lower layers to higher layers, with each neuron in a given layer informing neurons in higher layers. As described above, hierarchical representations may be constructed in successive layers of feedforward networks. Neural networks may also have recurrent connections or feedback connections (also called top-down). In recurrent connections, the output from a neuron in a given layer may be conveyed to another neuron in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks that are delivered sequentially to the neural network. The connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognition of high-level concepts may help distinguish certain low-level features of the input.

[0045]

[0061] The connections between layers of a neural network can be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, a neuron in a first layer can communicate its output to every neuron in a second layer, such that each neuron in the second layer receives input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layer of the locally connected neural network 204 may be configured such that each neuron in the layer has the same or similar connection pattern, but with connection strengths that may have different values ​​(e.g., 210, 212, 214, and 216). The locally connected connection pattern may result in spatially distinct receptive fields in the upper layer, since upper layer neurons in a given region receive inputs that are conditioned via training to the properties of a limited portion of the total inputs to the network.

[0046]

[0062] An example of a locally connected neural network is a convolutional neural network. FIG. 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g., 208). The convolutional neural network may be suitable for problems in which the spatial location of the inputs is meaningful. The convolutional neural network 206 may be used to perform one or more aspects of video compression and / or decompression according to aspects of the present disclosure.

[0047]

[0063] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230, such as an on-board camera. The DCN 200 of this example may be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or identifying traffic signals.

[0048]

[0064] The DCN 200 may be trained using supervised learning. During training, the DCN 200 may be presented with an image, such as the image 226 of a speed limit sign, and then a forward pass may be computed to generate the output 222. The DCN 200 may include a feature extraction section and a classification section. Upon receiving the image 226, the convolution layer 232 may apply a convolution kernel (not shown) to the image 226 to generate the first set of feature maps 218. As an example, the convolution kernel for the convolution layer 232 may be a 5×5 kernel that generates a 28×28 feature map. In this example, four different feature maps are generated in the first set of feature maps 218, so four different convolution kernels were applied to the image 226 in the convolution layer 232. The convolution kernel may also be referred to as a filter or a convolution filter.

[0049]

[0065] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0050]

[0066] In the example of FIG. 2D , the second set of feature maps 220 are convolved to generate a first feature vector 224. Moreover, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as “sign”, “60”, and “100”. A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 contains one or more features.

[0051]

[0067] In this example, the probability in the output 222 for "sign" and "60" is higher than the probability of others of the output 222 such as "30", "40", "50", "70", "80", "90", and "100". Before training, the output 222 generated by the DCN 200 may be inaccurate. Thus, an error may be calculated between the output 222 and a target output. The target output is the ground truth of the image 226 (e.g., "sign" and "60"). The weights of the DCN 200 may then be adjusted so that the output 222 of the DCN 200 is more closely aligned with the target output.

[0052]

[0068] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error would increase or decrease if the weights were adjusted. In the top layer, the gradient may correspond directly to the values ​​of the weights connecting the activating neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient may depend on the values ​​of the weights and the calculated error gradients of the upper layers. The weights may then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward pass" through the neural network.

[0053]

[0069] In practice, the error gradient of the weights may be calculated over a small number of examples such that the calculated gradient approximates the true error gradient. This approximation method may be called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the overall system no longer decreases or until the error rate reaches a target level. After training, the DCN may be presented with new images and a forward pass through the network may result in an output 222, which may be considered the inference or prediction of the DCN.

[0054]

[0070] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs may be used to extract hierarchical representations of a training dataset. DBNs may be obtained by stacking layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. RBMs are often used in unsupervised learning because they can learn probability distributions in the absence of information about the class into which each input should be classified. Using a hybrid unsupervised and supervised paradigm, the lower RBM of the DBN may be trained in an unsupervised manner and can act as a feature extractor, and the upper RBM may be trained in a supervised manner (on the joint distribution of inputs and target classes from the previous layer) and can act as a classifier.

[0055]

[0071] A deep convolutional network (DCN) is a network of convolutional networks composed of additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the network weights by using gradient descent.

[0056]

[0072] The DCN may be a feedforward network. In addition, as described above, the connections from neurons in the first layer of the DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of the DCN may be exploited for high-speed processing. The computational burden of the DCN may be much less than that of a similarly sized neural network with, for example, recurrent or feedback connections.

[0057]

[0073] The processing of each layer of the convolutional network may be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input may be considered to be three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The output of the convolutional connections may be considered to form a feature map in a subsequent layer, where each element of the feature map (e.g., 220) receives inputs from various neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map may be further processed using nonlinearities such as rectification, max(0,x), etc. Values ​​from neighboring neurons may be further pooled, which corresponds to downsampling and can provide further local invariance and dimensionality reduction.

[0058]

[0074] 3 is a block diagram illustrating an example of a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connections and weight sharing. As shown in FIG. 3, the deep convolutional network 350 includes convolutional blocks 354A, 354B. Each of the convolutional blocks 354A, 354B may be composed of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0059]

[0075] The convolution layer 356 may include one or more convolution filters, which may be applied to the input data 352 to generate feature maps. Although only two convolution blocks 354A, 354B are shown, the disclosure is not so limited and instead any number of convolution blocks (e.g., blocks 354A, 354B) may be included in the deep convolution network 350 depending on design preferences. The normalization layer 358 may normalize the output of the convolution filters. For example, the normalization layer 358 may provide whitening or lateral inhibition. The max pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0060]

[0076] For example, a parallel filter bank of a deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. Additionally, the deep convolutional network 350 may have access to other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0061]

[0077] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362A, 362B, 364 of the deep convolutional network 350, there are weights (not shown) that must be updated. The output of each of the layers (e.g., 356, 358, 360, 362A, 362B, 364) can serve as an input of a subsequent one of the layers (e.g., 356, 358, 360, 362A, 362B, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) provided in the initial convolution block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0062]

[0078] As mentioned above, digital video data can contain large amounts of data, which can place a significant burden on communication networks as well as devices that process and store the video data. For example, recording uncompressed video content generally results in large file sizes that increase significantly as the resolution of the recorded video content increases. In one illustrative example, uncompressed 16-bit per channel video recorded at 1080p / 24 (e.g., a resolution of 1920 pixels wide by 1080 pixels high captured at 24 frames per second) may occupy 12.4 megabytes per frame or 297.6 megabytes per second. Uncompressed 16-bit per channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame or 1195.2 megabytes per second.

[0063]

[0079] Network bandwidth is another constraint that can become problematic for large video files. For example, video content is often delivered over wireless networks (e.g., over LTE, LTE Advanced, New Radio (NR), WiFi, Bluetooth, or other wireless networks) and can constitute a large portion of consumer Internet traffic. Despite advances in the amount of available bandwidth in wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content in these networks.

[0064]

[0080] Because uncompressed video content can result in large files that may require significant memory for physical storage and significant bandwidth for transmission, video coding techniques may be utilized to compress and then decompress such video content.

[0065]

[0081] In order to reduce the size of video content and therefore the amount of storage required to store the video content and the amount of bandwidth required to deliver the video content, various video coding techniques may be implemented according to specific video coding standards such as HEVC, AVC, MPEG, VVC, among others. Video coding often uses prediction methods such as inter-prediction or intra-prediction that exploit redundancy present in a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality. As the demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0066]

[0082] In general, an encoding device encodes video data according to a video coding standard to generate an encoded video bitstream. In some examples, an encoded video bitstream (or "video bitstream" or "bitstream") is a sequence of one or more coded video sequences. An encoding device may generate a coded representation of a picture by partitioning each picture into multiple slices. A slice is independent of other slices such that information in a slice is coded without dependency on data from other slices in the same picture. A slice includes one or more slice segments, including independent slice segments, and one or more dependent slice segments that depend on previous slice segments, if any. In HEVC, slices are partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, together with syntax for the samples, are referred to as a coding tree unit (CTU). A CTU may also be referred to as a "treeblock" or a "largest coding unit" (LCU). A CTU is the basic processing unit for HEVC encoding. A CTU may be divided into multiple coding units (CUs) of various sizes. A CU contains an array of luma and chroma samples called a coding block (CB).

[0067]

[0083] The luma CB and the chroma CB may be further divided into prediction blocks (PBs). A PB is a block of luma or chroma component samples that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). The luma PB and one or more chroma PBs, together with associated syntax, form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) is signaled in the bitstream for each PU and is used for inter prediction of the luma PB and one or more chroma PBs. The motion parameters may also be referred to as motion information. The CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples of a color component to which a residual transform (e.g., possibly the same two-dimensional transform) is applied to code the prediction residual signal. A transform unit (TU) represents a TB of luma samples and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

[0068]

[0084] According to the HEVC standard, the transform may be performed using TUs. The TUs may be sized based on the size of the PUs in a given CU. The TUs may be the same size as the PUs or smaller than the PUs. In some examples, the residual samples corresponding to the CU may be subdivided into smaller units using a quadtree structure known as a residual quadtree (RQT). Leaf nodes of the RQT may correspond to TUs. Pixel difference values ​​associated with the TUs may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoding device.

[0069]

[0085] Once a picture of video data is partitioned into CUs, the encoding device predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, a prediction mode may be signaled in the bitstream using syntax data. The prediction mode may include intra prediction (or intra-picture prediction) or inter prediction (or inter-picture prediction). Intra prediction exploits the correlation between spatially adjacent samples in a picture. For example, using intra prediction, each PU is predicted from neighboring image data in the same picture using, for example, DC prediction to find an average value for the PU, planar prediction to fit a flat surface to the PU, directional prediction to extrapolate from neighboring data, or any other suitable type of prediction. Inter prediction uses the temporal correlation between pictures to derive a motion compensated prediction for a block of image samples. For example, using inter prediction, each PU is predicted using a motion compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision of whether to code a picture area using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0070]

[0086] After performing prediction using intra prediction and / or inter prediction, the encoding device may perform transformation and quantization. For example, after prediction, the encoding device may calculate a residual value corresponding to the PU. The residual value may comprise a pixel difference value between a current block of pixels being coded (PU) and a predictive block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a predictive block (e.g., issuing an inter prediction or intra prediction), the encoding device may generate a residual block by subtracting the predictive block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantify differences between pixel values ​​of the current block and pixel values ​​of the predictive block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0071]

[0087] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., size 32×32, 16×16, 8×8, 4×4, or other suitable size) may be applied to the residual data in each CU. In some embodiments, TUs may be used for the transform and quantization processes implemented by the encoding device. A given CU having one or more PUs may also include one or more TUs. As described in more detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TUs to generate serialized transform coefficients for entropy coding.

[0072]

[0088] The encoding device may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0073]

[0089] Once quantization is performed, the coded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partitioning information, and any other suitable data, such as other syntax data. Different elements of the coded video bitstream may then be entropy coded by an encoding device. In some examples, the encoding device may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector that may be entropy coded. In some examples, the encoding device may perform an adaptive scan. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoding device may entropy code the vector. For example, the encoding device may use context-adaptive variable length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partitioned entropy coding, or another suitable entropy coding technique.

[0074]

[0090] The encoding device can send the encoded video bitstream data over a communication link to a receiving device, which can store the encoded video bitstream and / or can include a decoding device. The encoding device can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) to extract elements of one or more coded video sequences that make up the encoded video data. The decoding device can then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to a prediction stage of the decoding device. The decoding device then predicts blocks of pixels (e.g., PUs) using intra-prediction, inter-prediction, IBC, and / or other types of prediction. In some examples, the prediction is added to the output of the inverse transform (the residual data). The decoding device can output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a consumer of the content.

[0075]

[0091] Video coding systems and techniques defined by various video coding standards (e.g., the HEVC video coding techniques described above) may be able to preserve most of the information in the raw video content and may be defined a priori based on concepts of signal processing and information theory. However, in some cases, machine learning (ML)-based image and / or video systems may provide benefits over non-ML-based image and video coding systems, such as deep learning-based end-to-end video coding (DLEC) systems. As described above, many deep learning-based systems are designed as a combination of an autoencoder sub-network (encoder sub-network) and a second sub-network responsible for learning a probability model on the quantization latents used for entropy coding. Such an architecture can be viewed as a combination of a transform, quantization module (encoder sub-network) and an entropy modeling sub-network module.

[0076]

[0092] FIG. 4 depicts a system 400 including a device 402 configured to perform video encoding and decoding using a deep learning-based system 410. The device 402 is coupled to a camera 407 and a storage medium 414 (e.g., a data storage device). In some implementations, the camera 407 is configured to provide image data 408 (e.g., a video data stream) to the processor 404 for encoding by the deep learning-based system 410. In some implementations, the device 402 may be coupled to and / or include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 402 may be coupled to a microphone and / or other input devices (e.g., a keyboard, a mouse, a touch input device such as a touch screen and / or a touch pad, and / or other input devices). In some examples, the camera 407, the storage medium 414, the microphone, and / or other input devices may be part of the device 402.

[0077]

[0093] The device 402 is also coupled to the second device 490 via a transmission medium 418, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 418 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. The transmission medium 418 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 418 may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from a source device to a receiving device. The wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi, radio frequency (RF), UWB, WiFi-Direct, cellular, Long Term Evolution (LTE), WiMax, etc.). The wired network may include any wired interface (e.g., fiber, Ethernet, power line Ethernet, Ethernet over coaxial cable, Digital Signal Line (DSL), etc.). Wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0078]

[0094] The device 402 includes one or more processors 404 (referred to herein as “processors”) coupled to a memory 406, a first interface (“I / F1”) 412, and a second interface (“I / F2”) 416. The processor 404 is configured to receive image data 408 from a camera 407, from the memory 406, and / or from a storage medium 414. The processor 404 is coupled to the storage medium 414 via the first interface 412 (e.g., via a memory bus) and to a transmission medium 418 via a second interface 416 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0079]

[0095] The processor 404 includes a deep learning based system 410. The deep learning based system 410 includes an encoder portion 462 and a decoder portion 466. In some implementations, the deep learning based system 410 can include one or more autoencoders. The encoder portion 462 is configured to receive input data 470 and process the input data 470 to generate output data 474 based at least in part on the input data 470.

[0080]

[0096] In some implementations, the encoder portion 462 of the deep learning-based system 410 is configured to perform lossy compression of the input data 470 to generate the output data 474, such that the output data 474 has fewer bits than the input data 470. The encoder portion 462 may be trained to compress the input data 470 (e.g., an image or video frame) based on any previous representation (e.g., one or more previously reconstructed frames) without using motion compensation. For example, the encoder portion 462 may compress a video frame using only video data from that video frame and without using any data from previously reconstructed frames. The video frames processed by the encoder portion 462 may be referred to herein as intra-predicted frames (I-frames). In some examples, the I-frames may be generated using traditional video coding techniques (e.g., according to HEVC, VVC, MPEG-4, or other video coding standards). In such examples, the processor 404 may include or be coupled to a video coding device (e.g., an encoding device) configured to perform block-based intra prediction, such as that described above with respect to the HEVC standard. In such examples, the deep learning-based system 410 may be excluded from the processor 404.

[0081]

[0097] In some implementations, the encoder portion 462 of the deep learning based system 410 may be trained to compress input data 470 (e.g., a video frame) using motion compensation based on a previous representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 462 may compress a video frame using video data from that video frame and using data from a previously reconstructed frame. The video frames processed by the encoder portion 462 may be referred to herein as intra-predicted frames (P-frames). Motion compensation may be used to determine data for a current frame by describing how pixels from a previously reconstructed frame move to new positions within the current frame along with residual information.

[0082]

[0098] As shown, the encoder portion 462 of the deep learning based system 410 may include a neural network 463 and a quantizer 464. The neural network 463 may include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures that generate intermediate data 472. The intermediate data 472 is input to the quantizer 464. Examples of components that may be included in the encoder portion 462 are illustrated in FIGS. 6-10.

[0083]

[0099] The quantizer 464 is configured to perform quantization and possibly entropy coding of the intermediate data 472 to generate the output data 474. The output data 474 may include the quantized (and possibly entropy coded) data. The quantization operation performed by the quantizer 464 may result in the generation of a quantized code (or data representing the quantized code generated by the deep learning based system 410) from the intermediate data 472. The quantized code (or data representing the quantized code) may also be referred to as a latent code (denoted as z) or a latent. The entropy model applied to the latent may be referred to herein as a "prior." In some examples, the quantization and / or entropy coding operation may be performed using existing quantization and entropy coding operations performed when encoding and / or decoding video data according to existing video coding standards. In some examples, the quantization and / or entropy coding operation may be performed by the deep learning based system 410. In one illustrative example, the deep learning based system 410 may be trained using supervised training, where residual data is used as input during training and quantization codes and entropy codes are used as known outputs (labels).

[0084]

[0100] The decoder portion 466 of the deep learning based system 410 is configured to receive the output data 474 (e.g., directly from the quantizer 464 and / or from the storage medium 414). The decoder portion 466 can process the output data 474 to generate a representation 476 of the input data 470 based at least in part on the output data 474. In some examples, the decoder portion 466 of the deep learning based system 410 includes a neural network 468, which may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures. Examples of components that may be included in the decoder portion 466 are illustrated in FIGS. 6-10.

[0085]

[0101] The processor 404 is configured to send the output data 474 to at least one of the transmission medium 418 or the storage medium 414. For example, the output data 474 may be stored in the storage medium 414 for later retrieval and decoding (or reconstruction) by the decoder portion 466 to generate a representation 476 of the input data 470 as reconstruction data. The reconstruction data may be used for various purposes, such as for playback of the video data that was encoded / compressed to generate the output data 474. In some implementations, the output data 474 may be decoded in another decoder device that matches the decoder portion 466 (e.g., in the device 402, in the second device 490, or in another device) to generate a representation 476 of the input data 470 as reconstruction data. For example, the second device 490 may include a decoder that matches (or substantially matches) the decoder portion 466, and the output data 474 may be transmitted to the second device 490 via the transmission medium 418. The second device 490 can process the output data 474 to generate a representation 476 of the input data 470 as the reconstructed data.

[0086]

[0102] The components of system 400 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuitry (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0087]

[0103] Although system 400 is shown to include several components, one skilled in the art will appreciate that system 400 can include more or fewer components than those shown in Figure 4. For example, system 400 can also include, or be part of, a computing device including input and output devices (not shown). In some implementations, system 400 may also include, or be part of, a computing device that includes one or more memory devices (e.g., one or more random access memory (RAM) components, read only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) in communication with and / or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and a baseband processor per wireless interface) for performing wireless communications, one or more wired interfaces (e.g., serial interfaces such as universal serial bus (USB) inputs, Lightning connectors, and / or other wired interfaces) for performing communications over one or more hardware connections, and / or other components not shown in FIG. 4 .

[0088]

[0104] In some implementations, system 400 may be implemented locally by and / or included in a computing device. For example, the computing device may include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses, or other AR device), a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), a television, and / or any other computing device having resource capabilities to perform the techniques described herein.

[0089]

[0105] In one example, the deep learning-based system 410 may be incorporated into a portable electronic device including a memory 406 coupled to the processor 404 and configured to store instructions executable by the processor 404, and a wireless transceiver coupled to an antenna and the processor 404 and operable to transmit output data 474 to a remote device.

[0090]

[0106] As mentioned above, deep learning-based systems are typically designed to handle non-subsampled input formats such as RGB or YUV4:4:4. Examples of image and video coding schemes targeting RGB inputs are described in J. Balle, D. Minnen, S. Singh, SJ Hwang, N. Johnston, "Variational image compression with a scale hyperprior," ICLR, 2018 (referred to as the "J. Balle Paper") and D. Minnen, J. Balle, G. Toderici, "Joint Autoregressive and Hierarchical Priors for Learned Image Compression," CVPR, 2018 (referred to as the "D. Minnen Paper"), which are incorporated by reference in their entireties for all purposes.

[0091]

[0107] FIG. 5 is a diagram illustrating an example of a deep learning-based system 500. a Sub-network and g s The sub-networks correspond to the encoder sub-network (e.g., the encoder portion 462) and the decoder sub-network (e.g., the decoder portion 466), respectively. a Sub-network and g sThe sub-network is designed for a three-channel RGB input, and all three R, G, and B input channels pass through and are processed by the same neural network layers (convolutional layers and generalized neural activity normalization (GDN) layers). The neural network layers can include convolutional layers (including convolutional layer 510) that perform convolutional operations, and GDN and / or inverse GDN (IGDN) nonlinear layers that implement local partition normalization. Local neural activity normalization is a type of transformation that has been shown to be particularly suitable for density modeling and compression of images. Deep learning-based systems (such as those shown in FIG. 5) target input channels with similar statistical properties, such as RGB data (where the statistical properties of different R, G, and B channels are similar).

[0092]

[0108] Although many deep learning-based systems are designed to process RGB inputs, most image and video coding systems use YUV input formats (e.g., often YUV4:2:0 input format). The chrominance (U and V) channels in the YUV format may be subsampled relative to the luminance (Y) channel. The subsampling has minimal impact on visual quality (e.g., no significant or noticeable impact on visual quality). Subsampling formats include YUV4:2:0 format, YUV4:2:2 format, and / or other YUV formats. Correlation between channels is reduced in YUV formats, which may not be the case in other color formats (e.g., RGB formats). Furthermore, the statistics of the luminance (Y) channel and the chrominance (U and V) channels are different. For example, the U and V channels have less variance compared to the luminance channel, while, for example, the RGB format has similar statistical characteristics for the R, G, and B channels. A video coder-decoder (or codec) is designed according to the input characteristics of the data (e.g., the codec can encode and / or decode data according to the input format of the data). For example, if the chrominance channels of a frame are subsampled (e.g., the chrominance channels are half the resolution compared to the luminance channels), then when the codec predicts blocks of the frame for motion compensation, the luminance blocks are twice as large in both width and height compared to the chrominance blocks. In another example, the codec can determine, among other things, how many pixels are going to be encoded or decoded for chrominance and luminance.

[0093]

[0109] To support YUV formats (e.g., YUV4:2:0 format), deep learning-based architectures must be modified. For example, if RGB input data (which, as noted above, most deep learning-based systems are designed to process) is replaced with YUV4:4:4 input data (with all channels having the same dimensions), the performance of the deep learning-based system processing the input data will degrade due to the different statistical properties of the luminance (Y) and chrominance (U and V) channels. As noted above, the chrominance (U and V) channels are subsampled in some YUV formats, such as in the case of YUV4:2:0. For example, for content having a YUV4:2:0 format, the resolution of the U and V channels is half that of the Y channel (the U and V channels have a quarter of the size of the Y channel due to their half width and half height). Such subsampling may render the input data incompatible with the input of the deep learning-based system. The input data is the information that the deep learning based system is attempting to encode and / or decode (e.g., a YUV frame containing three channels, including a luminance (Y) channel and a chrominance (U and V) channel).

[0094]

[0110] In some end-to-end video coding deep learning based systems, autoencoders are used to code intraframes, motion vectors (e.g., dense optical flow), and the residual of a motion compensated frame relative to the original frame. In one example, a flow autoencoder can be used to learn to jointly code optical flow and scale space, and a residual autoencoder codes the residual between the warped predicted frame and the original frame, all in the RGB domain.

[0095]

[0111] As mentioned above, systems and techniques are described herein that provide an ML-based system (e.g., including one or more deep learning-based architectures) that efficiently supports one or more YUV formats (e.g., YUV4:2:0 format). The deep learning-based architecture can encode and / or decode video data including a standalone frame (or image) and / or multiple frames. For example, the ML-based system can take as input a luma component of a current frame and a luma component of a previously restored frame, which may be restored by a previous instance of the ML-based system. The ML-based system can process the luma components of the current and previous frames to estimate motion information (e.g., flow information, such as optical flow information) of the luma component of the current frame. Then, using the luma component of the current frame, the ML-based system can determine (e.g., estimate) a motion estimate (e.g., flow information, such as optical flow information) of one or more chroma components of the current frame. Such techniques can be performed for all components of a frame. Further details are described below.

[0096]

[0112] FIG. 6 illustrates an example of a neural network architecture of a deep learning-based system 600 configured to perform video coding. The neural network architecture of FIG. 6 includes an intra-prediction engine 602 and an inter-prediction engine 610. The intra-prediction engine 602 and the inter-prediction engine 610 may include autoencoders (e.g., variational autoencoders (VAE)) as shown in FIG. 6, but may include other types of neural network architectures in other implementations. As shown, the intra-prediction engine 602 processes pixel information of an input frame 604 to generate a latent representation (

[0097]

number

[0098] The input frame 604 generates a luma component (

[0099]

number

[0100] ) and two chroma components (

[0101]

number

[0102] and

[0103]

number

[0104] The latent representation may also be referred to as a bitstream, which includes a number of bits that are a coded version of the input frame 604.

[0105]

number

[0106] (or a latent representation / bitstream received from another device), the decoder sub-network of the intra-prediction engine 602 generates a reconstructed frame 606 (where the "hats" on the components indicate the reconstructed values,

[0107]

number

[0108] ,

[0109]

number

[0110] ,

[0111]

number

[0112] , which is a reconstructed version of the input frame 604.

[0113] The inter-prediction engine 610 includes a flow engine 618, a residual engine 620, and a warping engine 622. As shown, the flow engine 618 calculates the luma component (

[0114]

number

[0115] ) and the reconstructed luma component of the previous frame 615 (at a previous time t−1) (

[0116]

number

[0117] The input is the luma component

[0118]

number

[0119] and the luma component

[0120]

number

[0121] , the flow engine 618 calculates the luma component of the current frame 614.

[0122]

number

[0123] A latent representation of the motion information (e.g., flow information) of

[0124]

number

[0125] The motion information may include optical flow information (e.g., multiple motion or displacement vectors, possibly with per-pixel or per-sample scale components), which indicates the motion of pixels in the current frame 614 (at time t) relative to the previous frame 615 (at time t-1).

[0126]

number

[0127] may also be referred to as the bitstream, and the luma component of the current frame 614

[0128]

number

[0129] The flow engine 618 may include a number of bits representing a coding version of the luma component of the current frame 614, rather than the chroma components.

[0130]

number

[0131] To process the latent representation

[0132]

number

[0133] The bitstream is reduced in size compared to using all components of the current frame 614 to determine the motion information.

[0134]

[0114] Luma component

[0135]

number

[0136] Latent expression of

[0137]

number

[0138] (or a latent representation or bitstream received from another device representing the luma component of the frame), the flow engine 618 calculates the luma component of the current frame 614.

[0139]

number

[0140] Movement information about (f L ) and also determines the chroma components of the current frame 614

[0141]

number

[0142] ,

[0143]

number

[0144] Movement information about (f CDetails of determining or estimating the motion information of the chroma components based on the determined motion information of the luma component are described below with respect to Figures 7A and 7B.

[0145]

[0115] The warping engine 622 calculates the luma of the current frame 614 (at time t).

[0146]

number

[0147] and chroma components

[0148]

number

[0149] ,

[0150]

number

[0151] The motion information (f L and f C For example, the warping engine 622 may be configured to perform warping using the luma

[0152]

number

[0153] and chroma components

[0154]

number

[0155] ,

[0156]

number

[0157] Movement information (f L and f C ) to warp pixels of the current frame 614 (at time t). In some aspects, the warping engine 622 may perform spatial scale flow (SSF) warping. For example, SSF warping may apply trilinear interpolation to generate interframe predictions from learned scale flow vectors, where the predictor may be formulated as follows:

[0158]

number

[0159]

[0116] The above trilinear interpolation is

[0160]

number

[0161] and chroma components

[0162]

number

[0163] Movement information (f L and f C ) may be performed on a component-by-component basis (e.g., for each luma component and each individual U and V chroma component) based on one or more warping parameters determined based on (a) (b) (c) (d) (e.g., for each luma component and each individual U and V chroma component). For example, the warping parameters may represent the horizontal component (x-direction) of a motion or displacement vector.

[0164]

number

[0165] and represents the vertical component (y direction) of the motion or displacement vector.

[0166]

number

[0167] and spatial motion / displacement information (v x and v y ), combined with s (called the scale field), which represents a progressively smoothed version of the reconstructed frame.

[0168] The output from the warping engine 622 (after warping has been performed by the warping engine 622) is P Y , P U , P V 6, where P Y is the luma component

[0169]

number

[0170] corresponds to the prediction of P U is the chroma component

[0171]

number

[0172] corresponds to the prediction of P V is the chroma component of the current frame 614

[0173]

number

[0174] corresponds to the prediction.

[0175]

[0118] The deep learning based system 600 then calculates the corresponding luma of the current frame 614.

[0176]

number

[0177] and Chroma

[0178]

number

[0179] ,

[0180]

number

[0181] Prediction from components Y , P U , P V We can obtain the residual signal by subtracting the residual signal r Y , chroma components

[0182]

number

[0183] The residual signal r U , and the chroma components

[0184]

number

[0185] The residual signal r V The residual engine 620 generates a latent representation of the residual (

[0186]

number

[0187] (denoted as ) can be generated.

[0188]

number

[0189] (or a latent representation of the residual received from another device), the residual engine 620 generates a reconstructed residual signal for the luma component.

[0190]

number

[0191] , chroma components

[0192]

number

[0193] The restored residual signal of

[0194]

number

[0195] , and the chroma components

[0196]

number

[0197] The restored residual signal of

[0198]

number

[0199] The deep learning based system 600 can generate a reconstructed residual for the current frame, including the predicted P Y , P U , P V The restored residual

[0200]

number

[0201] ,

[0202]

number

[0203] ,

[0204]

number

[0205] to generate a reconstructed frame 616.

[0206] FIG. 7A shows the luma component of the current frame (at time t).

[0207]

number

[0208] and the reconstructed luma component of the previous frame (at time t-1).

[0209]

number

[0210] 7 illustrates an example of a flow engine 718 operating in a multi-threaded block, collectively shown as a luma component 722. As mentioned above, in some cases, the flow engine 718 may operate as an autoencoder (VAE flowIn some cases, a combined deep learning based architecture can be designed, as shown in FIG. 7A, where the flow engine 718 calculates the current frame

[0211]

number

[0212] and the previously restored frame

[0213]

number

[0214] Both luma components of the SSFf L ) and chroma motion information (e.g., SSFf C ) as described herein. C ) is the luma motion information (e.g., f L ) can be derived based on

[0215]

[0120] As shown in FIG. 7A, the luma component of the current frame

[0216]

number

[0217] Movement information (f L ) to determine the luma component of the current frame.

[0218]

number

[0219] , and the restored luma component of the previous frame

[0220]

number

[0221] is processed by several convolutional and activation layers (collectively denoted as forward path 723). The notations "↓2" and "↑2" in FIG. 7A refer to stride values, with ↓2 referring to stride 2 (for downsampling, denoted by "↓") and ↑2 also referring to stride 2 (for upsampling, denoted by "↑"). For example, convolutional layer 724 converts the input luma component

[0222]

number

[0223] and

[0224]

number

[0225] The resulting output of the convolution layer 724 is the luma component of the current frame.

[0226]

number

[0227] The luma motion information (f L) are N arrays of feature values ​​(corresponding to the N channels) representing the input and output channels. The notation "2 / N" represents two input channels and N output channels. The nonlinear layers following the convolutional layer 724 can then process the feature values ​​output by the convolutional layer 724. Each successive convolutional and nonlinear layer can process the features output by the previous layer until the final convolutional layer 725 of the forward path 723 outputs a feature to the bottleneck portion 726 of the flow engine 718.

[0228] The output of the forward path 723 is processed by a bottleneck portion 726 of the flow engine 718 to obtain the luma component of the current frame.

[0229]

number

[0230] The luma motion information (f L ) to generate a bitstream or latency representing the features output by the final convolutional layer 725 of the forward path 723. The bottleneck portion 726 may include a quantization engine and an entropy coding engine in the forward path 723, and an entropy decoding engine and an inverse quantization engine on the backward path 728 of the flow engine 718. For example, the quantization engine may perform quantization on the features output by the final convolutional layer 725 of the forward path 723 to generate a quantized output. The entropy coding engine may entropy code the quantized output from the quantization engine to generate a bitstream. In some cases, the entropy coding engine may perform entropy coding using priors generated by a hyper prior network. The neural network system may output the bitstream to a server device or system for storage, for transmission to another device, and / or in other ways.

[0231]

[0122] The backward path 728 may be a decoder sub-network of the neural network system of the flow engine 718, or a decoder sub-network of a neural network system of another flow engine (of another device) in some cases. The entropy decoding engine of the flow engine 718 may entropy decode the bit stream output by the entropy coding engine of the bottleneck 726 (or the entropy coding engine of the bottleneck of another flow engine) and output the entropy decoded data to the inverse quantization engine of the backward path 728. The entropy decoding engine may perform entropy decoding using the priors generated by the hyper prior network. The dequantization engine may dequantize the data.

[0232] Next, the convolutional and deactivation layers of the backward pass 728 process the dequantized data from the bottleneck 726 to obtain the luma component of the current frame.

[0233]

number

[0234] Movement information 729(f L ) can be generated. L ) is the luma component of the current frame

[0235]

number

[0236] In some cases, the motion information 729 (f L ) may further include a scale component. For example, as shown in FIG. 7A for illustrative purposes, the motion information 729 may be

[0237]

number

[0238] Ingredients and

[0239]

number

[0240] Ingredients and S L As mentioned above,

[0241]

number

[0242] ,

[0243]

number

[0244] , and S L The components are used in equation (1) by the warping engine 622 to warp the pixels of the current frame 614 (at time t) to obtain the prediction P Y , P U , P V can be generated.

[0245]

[0124] Luma component of the current frame

[0246]

number

[0247] Movement information 729(f L After learning the chrominance information 731 (f CFor example, the flow engine 718 may determine or predict the motion information 731 (f C ) to obtain the luma component

[0248]

number

[0249] Movement information 729(f L ) can be subsampled. C ) may include motion vectors (e.g., having a horizontal or x-direction magnitude and a vertical or y-direction magnitude), such as a motion vector for each sample of a chroma component of the current frame. C ) may further include a scale component. For example, as shown in FIG. 7A for illustrative purposes, the motion information 731(f C )teeth,

[0250]

number

[0251] Ingredients and

[0252]

number

[0253] Ingredients and S C The motion information of the luma component 729(f L ) as well as the chroma motion information 731(f C )of

[0254]

number

[0255] ,

[0256]

number

[0257] , and S C The components are used in equation (1) by the warping engine 622 to warp the pixels of the current frame 614 (at time t) to obtain the prediction P Y , P U , P V can be generated.

[0258] In some aspects, the convolutional layer with downsampling 730 may be configured to convert the luma component of the current frame

[0259]

number

[0260] Movement information 729(f L ) based on the chroma component motion information 731(f C ) may be trained (e.g., using unsupervised learning or training) to learn. In one illustrative example, a training set that may be used to train the flow engine 718 may include luma and chroma motion information (as ground truth). The luma motion information may be input to the neural network of the flow engine 718, and the resulting chroma motion information output from the flow engine 718 may be minimized using the ground truth chroma motion information using a loss function (e.g., L1 or sum of absolute differences, L2Norm or sum of squared differences, or other loss function).

[0261]

[0126] The convolution layer 730 is shown in Figure 7A as |3 / 3|5x5conv↓2|. The notation "3 / 3" indicates that there are three input channels that become three output channels. As above, the notations "↓2" and "↑2" refer to stride values, where ↓2 refers to stride 2 for downsampling (denoted by "↓") and ↑2 refers to stride 2 for upsampling (denoted by "↑"). For example, the convolution layer 730 convolves the luma component by applying a 5x5 convolution filter in the horizontal and vertical dimensions by a stride value of 2.

[0262]

number

[0263] Movement information 729(f L ) by a factor of four (e.g., for the YUV4:2:0 format). In some examples, the convolutional layer 730 may be trained to downsample by other factors for other formats (e.g., the YUV4:2:2 format, etc.). The resulting output of the convolutional layer 724 is a 3×3 array of feature values ​​(corresponding to the three channels), which are the luma component

[0264]

number

[0265] Movement information 729(f L ) is a downsampled version of

[0266] In another aspect (not shown in FIG. 7A), the motion information 731(f) of the chroma components of the current frame C ) is the luma component

[0267]

number

[0268] Movement information 729(f L ) directly. For example, the flow engine 718 can determine the chroma flow without processing the luma flow using the convolutional layer 730. In one illustrative example, instead of the convolutional layer 730, the flow engine 718 uses the luma motion information 729 (f L ) is directly subsampled to obtain chrominance motion information 731(f C ), which can obtain the neural network of the flow engine 718.

[0269]

[0128] Figure 7B is a diagram showing an example of a subsampling engine 735 for subsampling luma motion information determined for a current frame (e.g., using the flow engine 718 of Figure 7A) to obtain chroma motion information for the current frame. For illustrative purposes, a simplified example is provided with each channel N (N = 2) of luma motion information 732 having a resolution of 4x4 (4 rows and 4 columns) with a total of 16 flow motion or displacement vectors. The subsampling engine 735 subsamples or downsamples the luma motion information 732 to generate or obtain chroma motion information 738, which is a subsampled / downsampled version of the luma motion information 732.

[0270]

[0129] The illustrative example of Figure 7B shows chroma motion information 738 that is a quarter of the size of the luma motion information 732. For example, as mentioned above, for content having a YUV 4:2:0 format, the resolution of the U and V channels is half the Y channel resolution (the width and height are halved, so the size of the U and V channels is a quarter of the Y channel). The subsampling engine 735 may be trained or otherwise configured to process other formats besides the 4:2:0 format, in which case subsampling may include generating chroma information having a different resolution than that shown in Figure 7A.

[0271] 7A , and can be trained (e.g., using unsupervised learning or training) to determine chroma motion information 738 from luma motion information 732. In other aspects, the subsampling engine 735 can include a subsampler that directly subsamples the luma motion information 732 to obtain the chroma motion information 738.

[0272]

[0131] The number of channels (denoted as N in FIG. 7A) in the convolutional or transform layers of the forward path 723 and the backward path 728, as well as the bottleneck (M), may be set to any suitable value. In one illustrative example, the number of channels N may be selected as N=192 and M=128. Successive smoothed versions of the reconstructed frame (associated with the scale field s) may be obtained using filtering or smoothing operators. In one example, Gaussian blur filters of different widths may be used. In another example, a Gaussian pyramid with successive filtering and interpolation may be used to generate the smoothed version of the reconstructed frame. Furthermore, an arbitrarily large number of scales S may be used. In one example, the scale S may be set to S=3, and the scale levels are

[0273]

number

[0274] where σ can represent the Gaussian filter width.

[0275]

[0132] Meanwhile, the nonlinear activation layer in FIG. 7A is shown as PReLU for illustrative purposes, and other types of nonlinear activation layers can be used, such as a generalized partition normalization (GDN) layer, a combination of a PReLU layer and a GDN layer, etc.

[0276] In some examples, to effectively support one or more YUV formats (e.g., YUV4:2:0), the intra prediction engine 602 and the residual engine 620 of FIG. 6 may be designed based on the general neural network architecture shown in FIG. 8A, FIG. 9, or FIG. 10. For example, the architectures shown in FIG. 8A, FIG. 9, and FIG. 10 may be configured to process input data having a YUV4:2:0 format. In some examples, neural network architectures similar to those shown in FIG. 8A, FIG. 9, or FIG. 10 may be used to encode and / or decode other types of YUV content (e.g., content having a YUV4:4:4 format, a YUV4:2:2 format, etc.) and / or content having other input formats. In some cases, each architecture shown in FIG. 8A, FIG. 9, and FIG. 10 includes a residual autoencoder that operates on a YUV (e.g., 4:2:0) residual.

[0277]

[0134] Figure 8A is a diagram illustrating an example of a front-end neural network system 800 that may be configured to directly cooperate with 4:2:0 input (Y, U and V) data. As shown in Figure 8A, in the encoder sub-network (also called forward path) of the neural network system, the split luma and chroma channels (luma Y channel 802 and U and V chroma channels 804) are combined using a 1x1 convolution layer 806, and then a nonlinear layer 808 (also called a nonlinear operator) is applied. Similar operations are performed in the decoder sub-network (also called backward path) of the neural network system, but in the reverse order. For example, as shown in Figure 8A, an inverse nonlinear layer 809 (also called an inverse nonlinear operator) is applied, and the Y and U, V channels are separated using a 1x1 convolution layer 813, and the separate Y and U, V channels are processed using respective inverse nonlinear layers 815, 816 and convolution layers 817, 818.

[0278] The first two neural network layers in the encoder sub-network of the neural network system 800 of Fig. 8A include a first convolutional layer 811 (denoted as Nconv|3×3|↓1), a second convolutional layer 810 (denoted as Nconv|5×5|↓2), a first nonlinear layer 814, and a second nonlinear layer 812. The last two neural network layers in the decoder sub-network of the front-end neural network architecture of Fig. 8A include a first inverse nonlinear layer 816, a second inverse nonlinear layer 815, a first convolutional layer 818 (denoted as 2conv|3×3|↑1) for generating restored chrominance (U and V) components of the frame, and a second convolutional layer 817 (denoted as 1conv|5×5|↑2) for generating restored luminance (Y) components of the frame. The notation "Nconv" refers to the number of output channels (N) (corresponding to the number of output features) of a given convolutional layer (with the value of N defining the number of output channels). The notations 3x3 and 5x5 indicate the size of the respective convolution kernels (e.g., 3x3 and 5x5 kernels). The notations "↓1" and "↓2" refer to stride values, where ↓1 refers to a stride of 1 (for downsampling, denoted by "↓") and ↓2 refers to a stride of 2 (for downsampling). The notations "↑1" and "↑2" refer to stride values, where ↑1 refers to a stride of 1 (for upsampling, denoted by "↑") and ↑2 refers to a stride of 2 (for upsampling).

[0279]

[0136] For example, the convolution layer 810 downsamples the input luma channel 802 by 4 times by applying a 5x5 convolution filter in the horizontal and vertical dimensions with a stride value of 2. The resulting output of the convolution layer 810 is an N array (corresponding to the N channels) of feature values. The convolution layer 811 processes the input chroma (U and V) channels 804 by applying a 3x3 convolution filter in the horizontal and vertical dimensions with a stride value of 1. The resulting output of the convolution layer 811 is an N array (corresponding to the N channels) of feature values. The array of feature values ​​output by the convolution layer 810 has the same dimensions as the array of feature values ​​output by the convolution layer 811. The nonlinear layer 812 can then process the feature values ​​output by the convolution layer 810, and the nonlinear layer 814 can process the feature values ​​output by the convolution layer 811.

[0280]

[0137] The 1x1 convolution layer 806 may then process the feature values ​​output by the nonlinear layers 812, 814. The 1x1 convolution layer 806 may generate a linear combination of features associated with the luma channel 802 and the chroma channel 804. The linear combination operation acts as a cross-channel mixing per value of the Y and UV components, resulting in a prediction of the cross components (e.g., cross-luminance and chrominance components) that improves coding performance. Each 1x1 convolution filter of the 1x1 convolution layer 806 may include a respective scaling factor applied to the corresponding Nth channel of the luma channel 802 and the corresponding Nth channel of the chroma channel 804.

[0281]

[0138] Figure 8B is a diagram illustrating an example operation of the 1x1 convolution layer 838. As described above, N represents the number of output channels. As shown in Figure 8B, 2N channels are provided as inputs to the 1x1 convolution layer 838, including an N channel chroma (combined U and V) output 832 and an N channel luma (Y) output 834. In the example of Figure 8B, the value of N is equal to 2, showing two channels of values ​​for the N channel chroma output 832 and two channels of values ​​for the N channel luma output 834. Referring to Figure 8A, the N channel chroma output 832 may be an output from the nonlinear layer 814, and the N channel luma output 834 may be an output from the nonlinear layer 812.

[0282]

[0139] The 1x1 convolution layer 838 processes 2N channels, performs feature-wise linear combination of the 2N channels, and then outputs an N channel set of features or coefficients. The 1x1 convolution layer 838 includes two 1x1 convolution filters (based on N=2). The first 1x1 convolution filter is indicated with a value of S1, and the second 1x1 convolution filter is indicated with a value of S2. The value of S1 represents a first scaling factor, and the value of S2 represents a second scaling factor. In one illustrative example, the value of S1 is equal to 3, and the value of S2 is equal to 4. Each of the 1x1 convolution filters of the 1x1 convolution layer 838 has a stride value of 1, indicating that the scaling factors S1 and S2 are applied to each value at the UV output 832 and the Y output 834.

[0283] For example, a scaling factor S1 of the first 1×1 convolution filter is applied to each value in the first channel (C1) of the UV output 832 and each value in the first channel (C1) of the Y output 834. Once each value in the first channel (C1) of the UV output 832 and each value in the first channel (C1) of the Y output 834 are scaled by the scaling factor S1 of the first 1×1 convolution filter, the scaled values ​​are combined into the first channel (C1) of the output values ​​839. A scaling factor S2 of the second 1×1 convolution filter is applied to each value in the second channel (C2) of the UV output 832 and each value in the second channel (C2) of the Y output 834. After each value of the second channel (C2) of the UV output 832 and each value of the second channel (C2) of the Y output 834 are scaled by a scaling factor S2 of the second 1×1 convolution filter, the scaled values ​​are combined into the second channel (C2) of the output values ​​839. As a result, the four Y and UV channels (two Y channels and two combined UV channels) are mixed and combined into two output channels C1 and C2.

[0284] Returning to FIG. 8A, the output of the 1×1 convolutional layer 806 is processed by additional nonlinear layers and additional convolutional layers of the encoder sub-network. The bottleneck 820 may include a quantization engine and an entropy coding engine on the encoder sub-network (or forward path), and an entropy decoding engine and an inverse quantization engine (or backward path) on the decoder sub-network. The quantization engine may perform quantization on the features output by the last neural network layer 819 of the encoder sub-network to generate a quantized output. The entropy coding engine may entropy code the quantized output from the quantization engine to generate a bitstream. In some cases, the entropy coding engine may use priors generated by the hyper prior network to perform the entropy coding. The neural network system may output a bitstream for storage, for transmission to another device, a server device, or a server system, and / or otherwise.

[0285]

[0142] The decoder sub-network of the neural network system or the decoder sub-network of another neural network system (in another device) can decode the bitstream. The entropy decoding engine of the bottleneck 820 (of the decoder sub-network) can entropy decode the bitstream and output the entropy decoded data to the inverse quantization engine of the decoder sub-network. The entropy decoding engine can use the priors generated by the hyper prior network to perform the entropy decoding. The inverse quantization engine can inverse quantize the data. The inverse quantized data can be processed by several convolutional layers and several inverse nonlinear layers of the decoder sub-network.

[0286] After being processed by several convolutional and nonlinear layers, the 1×1 convolutional layer 813 can process the data output by the final inverse nonlinear layer 809. The 1×1 convolutional layer 813 can include 2N convolutional filters that can split the data into Y channel features and combined UV channel features. For example, each of the N channels output by the inverse nonlinear layer 809 can be processed using 2N 1×1 convolutions (resulting in scaling) of the 1×1 convolutional layer 813. The scaling factor n corresponding to the output channel (out of a total of 2N output channels) applied to the N input channels is i For each, the decoder sub-network can perform summations across the N input channels, resulting in 2N outputs. In one illustrative example, for a scaling factor n1, the decoder sub-network can apply the scaling factor n1 to the N input channels and sum the results, resulting in one output channel. The decoder sub-network can apply 2N different scaling factors (e.g., scaling factor n1, scaling factor n2, scaling factor n3, scaling factor n4, scaling factor n5, scaling factor n6, scaling factor n7, scaling factor n8, scaling factor n9, scaling factor n10, scaling factor n111, scaling factor n12, scaling factor n13, scaling factor n14, scaling factor n15, scaling factor n16, scaling factor n17, scaling factor n18, scaling factor n19, scaling factor n20, scaling factor n21, scaling factor n22, scaling factor n23, scaling factor n24, scaling factor n25, scaling factor n26, scaling factor n27, scaling factor n28, scaling factor n29, scaling factor n30, scaling factor n31, scaling factor n32, scaling factor n33, scaling factor n34, scaling factor n35, scaling factor n36, scaling factor n37, scaling factor n38, scaling factor n39, scaling factor n40, scaling factor n41, scaling factor n42, scaling factor n43, scaling factor n44, scaling factor n45, scaling factor n46, scaling factor n47, scaling factor n48, scaling factor n49, scaling factor n5 ...2, scaling factor n43, scaling factor n44, scaling factor n45, scaling factor n46, scaling factor n47, scaling factor n48, scaling factor n49, scaling factor n53, scaling factor n44, scaling factor n45, scaling factor n46, scaling factor n47, scaling factor n48 2N This operation can be performed for up to

[0287]

[0144] The Y channel features output by the 1x1 convolution layer 813 may be processed by an inverse nonlinearity 815. The combined UV channel features output by the 1x1 convolution layer 813 may be processed by an inverse nonlinearity 816. The convolution layer 817 may process the Y channel features and output a reconstructed Y channel or reconstructed frame sample (e.g., luminance sample or pixel) per pixel, shown as a reconstructed Y component 824. The convolution layer 818 may process the combined UV channel features and output a reconstructed U channel or reconstructed frame sample (e.g., chrominance blue sample or pixel) per pixel and a reconstructed V channel or reconstructed frame sample (e.g., chrominance red sample or pixel) per pixel, shown as a reconstructed U and V components 825.

[0288] In some examples, different variations of the architecture of FIG. 8A with different nonlinear operators may be used as the intra prediction engine 602 and the residual engine 620. For example, FIG. 9 and FIG. 10 illustrate the front-end architecture of FIG. 8A configured to process data having a YUV format (e.g., YUV4:2:0 input data with Y, U, and V components). In the neural network system 900 of FIG. 9, at the encoder side, the branched luma and chroma channels are combined using a 1×1 convolution layer (similar to that of FIG. 8A), and then the GDN nonlinear operator is applied. In the neural network system 1000 of FIG. 10, at the encoder side, the branched luma and chroma channels are combined using a 1×1 convolution layer (similar to that of FIG. 8A), and then the PReLU nonlinear operator is applied. In one example, the VAE res and VAE intra Both of these can use the variant shown in Figure 9. In another example, the VAE res and VAE intra Both can use the variant of Figure 10. In another example, the VAE res can use the variant in Figure 9, VAEintra can use the variant of Figure 10. In another example, the VAE intra can use the variant in Figure 9, VAE res can use the variant of FIG.

[0289]

[0146] Figure 11 is a flow diagram illustrating an example of a process 1100 for processing video data. At block 1102, the process 1100 includes obtaining, by a machine learning system, input video data. The input video data includes a luma component of a current frame (e.g., the current frame (at time t) of Figure 7A).

[0290]

number

[0291] ) of a previously restored frame. In some cases, the input video data may include at least one luminance component of a previously restored frame (e.g., the restored luma component of the previous frame (at time t-1) of FIG. 7A

[0292]

number

[0293] ), which may be referred to as at least one restored luminance component. In some aspects, the current frame comprises a video frame. In some cases, the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component. In some aspects, the current frame has a luminance-chrominance (YUV) format. In some cases, the YUV format is a YUV 4:2:0 format.

[0294]

[0147] At block 1104, the process includes determining, by the machine learning system, motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using at least one luminance component of the current frame. In some aspects, the process 1100 may include determining motion information for at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one restored luma component of the previous frame. In some cases, the process 1100 may further include determining motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame. In some cases, the motion information for one or more chrominance components of the current frame is determined using a convolution layer of the machine learning system. For example, referring to FIG. 7A as an illustrative example, the flow engine 718 may determine motion information for at least one luminance component of the current frame based on at least one restored luma component of the previous frame.

[0295]

number

[0296] and the previously restored frame

[0297]

number

[0298] The current frame is

[0299]

number

[0300] The luma motion information (e.g., SSFf L ) and chroma motion information (e.g., SSFf C) can be estimated. As mentioned above, the chroma motion information (e.g., f C ) 731 uses a convolutional layer 730 to extract luma motion information (e.g., f L ) 729. In some cases, the motion information for one or more chrominance components of the current frame may be determined at least in part by sampling motion information determined for at least one luminance component of the current frame.

[0301]

[0148] In some aspects, the process 1100 includes determining a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame by a machine learning system using motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame. In some aspects, the warping parameter for at least one luminance component of the current frame and the one or more warping parameters for one or more chrominance components of the current frame include a spatial scale flow (SSF) warping parameter. In some cases, the spatial scale flow (SSF) warping parameter includes a learned scale flow vector. Referring to FIG. 6 as an illustrative example, the warping parameter represents a horizontal component (in the x-direction) of the motion or displacement vector.

[0302]

number

[0303] and represents the vertical component (in the y direction) of the motion or displacement vector.

[0304]

number

[0305] and spatial motion / displacement information (v xand v y ) and s (called the scale field), which represents a progressively smoothed version of the combined reconstructed frame.

[0306]

[0149] The process 1100 may include performing one or more interframe predictions (e.g., predictor P in FIG. 6 ) of the current frame using a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame. Y , P U , and P V In some cases, the one or more inter-frame predictions are determined at least in part by applying an interpolation operation that uses a warping parameter of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame. In one illustrative example, the interpolation operation includes a trilinear interpolation operation.

[0307] In some examples, the processes described herein may be performed by a computing device or apparatus, such as a computing device having the computing device architecture 1200 shown in FIG. 11. In one example, one or more processes may be performed by a computing device having the computing device architecture 1200 to implement any one or more of the neural network architectures shown in FIG. 6 and / or the neural network architectures shown in FIG. 7A, FIG. 7B, FIG. 8A, FIG. 9, and / or FIG. 10. In some examples, the computing device may include or be part of a mobile device (e.g., a mobile phone, a tablet computing device, etc.), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device of a vehicle), a robotic device, and / or any other computing device having resource capabilities to perform the processes described herein.

[0308] In some cases, a computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers, or combined transmitter-receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other components configured to perform steps of processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0309]

[0152] Components of a computing device may be implemented in circuits. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0310]

[0153] The processes described herein may be illustrated as logic flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described should not be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0311]

[0154] Furthermore, the processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0312]

[0155] Figure 12 illustrates an exemplary computing device architecture 1200 of an exemplary computing device that can implement various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other devices. For example, the computing device architecture 1200 can implement the system of Figure 6. The components of the computing device architecture 1200 are illustrated as being in electrical communication with each other using a connection 1205 such as a bus. The exemplary computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and computing device connections 1205 that couple various computing device components, including a computing device memory 1215 such as a read only memory (ROM) 1220 and a random access memory (RAM) 1225, to the processor 1210.

[0313]

[0156] The computing device architecture 1200 may include a cache of high-speed memory directly connected to the processor 1210, in close proximity to the processor 1210, or integrated as part of the processor 1210. The computing device architecture 1200 may copy data from the memory 1215 and / or storage device 1230 to the cache 1212 for quick access by the processor 1210. In this manner, the cache may provide performance improvements that avoid delays of the processor 1210 while waiting for data. These and other modules may control or be configured to control the processor 1210 to perform various actions. Other computing device memories 1215 may also be available for use. The memory 1215 may include multiple different types of memory having different performance characteristics. Processor 1210 may include any general purpose processor, hardware or software services such as service 1 1232, service 2 1234, and service 3 1236 stored in storage device 1230 configured to control processor 1210, as well as special purpose processors where software instructions are built into the processor design. Processor 1210 may be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0314] To enable user interaction with the computing device architecture 1200, the input device 1245 may represent any number of input mechanisms, such as a microphone for voice, a touch screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 1235 may also be one or more of several output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 1200. The communication interface 1240 may generally govern and manage user input and computing device output. There is no restriction to operating on any particular hardware configuration, and thus the basic features herein may be readily substituted with improved hardware or firmware configurations as they are developed.

[0315]

[0158] The storage device 1230 is a non-volatile memory, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 1225, a read-only memory (ROM) 1220, and hybrids thereof. The storage device 1230 may include services 1232, 1234, 1236 for controlling the processor 1210. Other hardware or software modules are possible. The storage device 1230 may be connected to a computing device connection 1205. In one aspect, a hardware module that performs a particular function may include software components stored in a computer-readable medium along with necessary hardware components, such as the processor 1210, the connection 1205, the output device 1235, etc., to perform the function.

[0316] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, a smartphone, a tablet, a laptop computer, a vehicle, a drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0317]

[0160] The term "device" is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system, etc.). A device as used herein may be any electronic device having one or more parts capable of implementing at least some parts of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a specific configuration, type, or number of objects. Furthermore, the term "system" is not limited to multiple components or a specific embodiment. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. The following description and examples use the term "system" to describe various aspects of the present disclosure, but the term "system" is not limited to a specific configuration, type, or number of objects.

[0318]

[0161] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method implemented in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as to avoid obscuring the embodiments.

[0319]

[0162] Individual embodiments may be described above as a process or method depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process ends when the operations of a process are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to a calling function or a main function.

[0320]

[0163] The processes and methods according to the above-mentioned examples may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0321]

[0164] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media that do not include carrier waves and / or transitory electronic signals on which data may be stored and that propagate wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as flash memory, memories or memory devices, magnetic or optical disks, flash memories, USB devices with non-volatile memory, network-connected storage devices, compact discs (CDs) or digital versatile discs (DVDs), among others, and any suitable combination thereof. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0322] In some embodiments, computer-readable storage devices, media, and memories can include cables or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0323]

[0166] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may also be implemented on a circuit board between different chips or different processes executing within a single device, as further examples.

[0324]

[0167] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0325]

[0168] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application are described in detail herein, it should be understood that the inventive concept may be embodied and employed in various ways, except as limited by the prior art, and the appended claims are to be construed to include such variations. The various features and aspects of the above-described applications may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of explanation, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0326]

[0169] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used in this specification may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this specification.

[0327]

[0170] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform that operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform that operation, or any combination thereof.

[0328]

[0171] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or any component that is in communication with another component, either directly or indirectly (e.g., connected to the other component via a wired or wireless connection, and / or other suitable communications interface).

[0329]

[0172] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or more than one member of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include unrecited items within the set of A and B.

[0330]

[0173] Various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly show this compatibility of hardware and software, various exemplary components, blocks, modules, circuits, and steps are described above generally with respect to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of this application.

[0331]

[0174] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0332]

[0175] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0333]

[0176] Illustrative examples of the present disclosure include the following:

[0334]

[0177] Aspect 1: A method for processing video data, the method comprising: obtaining, by a machine learning system, input video data including at least one luminance component of a current frame; and determining, by the machine learning system, motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using the at least one luminance component of the current frame.

[0335]

[0178] Aspect 2: The method of aspect 1, further comprising determining a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame by a machine learning system using motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, and determining one or more inter-frame predictions of the current frame using the warping parameter for at least one luminance component of the current frame and the one or more warping parameters for one or more chrominance components of the current frame.

[0336]

[0179] Aspect 3: A method as described in aspect 2, wherein the one or more interframe predictions are determined at least in part by applying an interpolation operation using a warping parameter of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame.

[0337]

[0180] Aspect 4: The method of aspect 3, wherein the interpolation operation includes a trilinear interpolation operation.

[0338]

[0181] Aspect 5: A method described in any one of aspects 2 to 4, wherein the warping parameters of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame include spatial scale flow (SSF) warping parameters.

[0339]

[0182] Aspect 6: The method described in aspect 5, wherein the SSF warping parameters include learned scale flow vectors.

[0340]

[0183] Aspect 7: A method as described in any one of aspects 1 to 6, wherein determining motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using at least one luminance component of the current frame includes determining motion information for at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one restored luma component of a previous frame, and determining motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame.

[0341]

[0184] Aspect 8: The method described in aspect 7, wherein motion information for one or more chrominance components of a current frame is determined using a convolutional layer of a machine learning system.

[0342]

[0185] Aspect 9: A method as described in aspect 7, wherein motion information for one or more chrominance components of the current frame is determined at least in part by sampling motion information determined for at least one luminance component of the current frame.

[0343]

[0186] Aspect 10: A method according to any one of aspects 1 to 9, wherein the current frame comprises a video frame.

[0344]

[0187] Aspect 11: A method according to any one of aspects 1 to 10, wherein the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.

[0345]

[0188] Example 12: The method of any one of examples 1 to 11, wherein the current frame has a luminance-chrominance (YUV) format.

[0346]

[0189] Aspect 13: The method described in aspect 12, wherein the YUV format is YUV4:2:0 format.

[0347]

[0190] Aspect 14: An apparatus for processing video data, comprising at least one memory and one or more processors coupled to the at least one memory, wherein the one or more processors are configured to use a machine learning system to obtain input video data including at least one luminance component of a current frame, and to determine, using the at least one luminance component of the current frame, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame.

[0348]

[0191] Aspect 15: The apparatus described in aspect 14, wherein the one or more processors are configured to use a machine learning system to determine a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame based on motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, and to determine one or more inter-frame predictions of the current frame using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame.

[0349]

[0192] Aspect 16: The apparatus described in aspect 15, wherein the one or more interframe predictions are determined at least in part by applying an interpolation operation using a warping parameter of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame.

[0350]

[0193] Aspect 17: The apparatus of aspect 16, wherein the interpolation operation includes a trilinear interpolation operation.

[0351]

[0194] Aspect 18: An apparatus described in any one of aspects 15 to 17, wherein the warping parameters of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame include spatial scale flow (SSF) warping parameters.

[0352]

[0195] Aspect 19: The apparatus described in aspect 18, wherein the SSF warping parameters include learned scale flow vectors.

[0353]

[0196] Aspect 20: An apparatus described in any one of aspects 14 to 19, wherein to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using at least one luminance component of the current frame, one or more processors are configured to determine motion information for at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one restored luma component of a previous frame, and to determine motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame.

[0354]

[0197] Aspect 21: An apparatus as described in aspect 20, wherein motion information for one or more chrominance components of a current frame is determined using a convolutional layer of a machine learning system.

[0355]

[0198] Aspect 22: The apparatus described in aspect 20, wherein to determine motion information for one or more chrominance components of the current frame, the one or more processors are configured to sample the motion information determined for at least one luminance component of the current frame.

[0356]

[0199] Aspect 23: An apparatus described in any one of aspects 14 to 22, wherein the current frame includes a video frame.

[0357]

[0200] Aspect 24: An apparatus described in any one of aspects 14 to 23, wherein the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.

[0358]

[0201] Aspect 25: The apparatus described in aspects 14 to 24, wherein the current frame has a luminance-chrominance (YUV) format.

[0359]

[0202] Aspect 26: An apparatus described in aspect 25, wherein the YUV format is YUV 4:2:0 format.

[0360]

[0203] Aspect 27: An apparatus described in any one of aspects 14 to 26, further comprising at least one camera configured to capture one or more frames.

[0361]

[0204] Aspect 28: An apparatus described in any one of aspects 14 to 27, further comprising at least one display configured to display one or more frames.

[0362]

[0205] Aspect 29: An apparatus described in any one of aspects 14 to 28, comprising a mobile device.

[0363]

[0206] Aspect 30: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations recited in aspects 1 to 29.

[0364]

[0207] Aspect 31: An apparatus comprising means for performing any of the operations recited in aspects 1 to 29.

Claims

1. 1. A method for processing video data, comprising the steps of: obtaining, by a machine learning system, a latent representation of at least one luminance component of a current frame, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; determining, by the machine learning system, motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame; and determining, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame.

2. determining, by the machine learning system using the motion information of the at least one luminance component of the current frame and the motion information of the one or more chrominance components of the current frame, a warping parameter for the at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame; 2. The method of claim 1 , further comprising: determining one or more inter-frame predictions of the current frame using the warping parameter of the at least one luminance component of the current frame and the one or more warping parameters of the one or more chrominance components of the current frame.

3. 3. The method of claim 2, wherein the one or more interframe predictions are determined at least in part by applying an interpolation operation that uses the warping parameters of the at least one luminance component of the current frame and the one or more warping parameters of the one or more chrominance components of the current frame.

4. The method of claim 3 , wherein the interpolation operation comprises a trilinear interpolation operation.

5. 3. The method of claim 2, wherein the warping parameters of the at least one luminance component of the current frame and the one or more warping parameters of the one or more chrominance components of the current frame comprise spatial-scale flow (SSF) warping parameters.

6. The method of claim 5 , wherein the SSF warping parameters include learned scale flow vectors.

7. The method of claim 1 , wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.

8. The method of claim 1 , wherein the motion information for the one or more chrominance components of the current frame is determined at least in part by sampling the motion information determined for the at least one luminance component of the current frame.

9. The method of claim 1 , wherein the current frame comprises a video frame.

10. The method of claim 1 , wherein the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.

11. The method of claim 1 , wherein the current frame has a luminance-chrominance (YUV) format.

12. The method of claim 11 , wherein the YUV format is a YUV 4:2:0 format.

13. 1. An apparatus for processing video data, comprising: At least one memory; and one or more processors coupled to the at least one memory, the one or more processors comprising: obtaining a latent representation of at least one luminance component of a current frame using a machine learning system, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; determining motion information for the at least one luminance component of the current frame using a latent representation of the at least one luminance component of the current frame using the machine learning system; and and determining motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame using the machine learning system.

14. or at least one camera configured to capture one or more frames; or The apparatus of claim 13 comprising a mobile device.

15. When executed by one or more processors, the one or more processors are obtaining a latent representation of at least one luminance component of a current frame using a machine learning system, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; determining motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame using the machine learning system; and and determining, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame.

Images