A Front-End Architecture for Neural Network-Based Video Coding

Abstract

Techniques for processing video data using a neural network system are described herein. For example, the process may include generating, by a first convolutional layer of an encoder sub-network of the neural network system, output values ​​associated with a luminance channel of the frame. The process may include generating, by a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame. The process may include generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame. The process may further include generating encoded video data based on the combined representation of the frame.

Description

[Technical Field]

[0001]

[0001] The present disclosure generally relates to image and video coding, including encoding (or compression) and decoding (decompression) of images and / or videos. For example, aspects of the present disclosure relate to techniques for processing luminance-chrominance (YUV) input formats (e.g., 4:2:0 YUV input format, 4:4:4 YUV input format, 4:2:2 YUV input format, etc.) and / or other input formats using end-to-end machine learning (e.g., neural network)-based image and video coding systems. [Background technology]

[0002]

[0002] Many devices and systems enable video data to be processed and output for consumption. Digital video data comprises a large amount of data to meet the demands of consumers and video providers. For example, consumers of video data desire high-quality video, including high fidelity, resolution, frame rates, etc. As a result, the large amount of video data required to meet these demands places a strain on communication networks and devices that process and store the video data.

[0003]

[0003] Video coding techniques can be used to compress video data. The goal of video coding is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation to video quality. As ever-evolving video services become available, encoding techniques with better coding efficiency are needed. Summary of the Invention

[0004]

[0004] Systems and techniques are described for coding (e.g., encoding and / or decoding) image and / or video content using one or more machine learning systems. For example, an end-to-end machine learning (e.g., neural network)-based image and video coding (E2E-NNVC) system is provided that can process a YUV (digital domain YCbCr) input format (and possibly other input formats), particularly a 4:2:0 YUV input format. The E2E-NNVC system can process video data including standalone frames (also called images or pictures) and / or multiple frames. The YUV format includes one luminance channel (Y) and a pair of chrominance channels (U and V). The U and V channels can be subsampled relative to the Y channel without a significant or noticeable impact on visual quality. Correlation between channels is reduced in the YUV format, which may not be true for other color formats (e.g., red-green-blue (RGB) formats). Aspects of the systems and techniques described herein provide a front-end architecture (e.g., a new sub-network) for accommodating a YUV 4:2:0 input format in an E2E-NNVC designed for an RGB input format (and possibly an E2E-NNVC designed for other input formats). The front-end architecture is applicable to many E2E-NNVC architectures.

[0005] In one illustrative example, a method for processing video data is provided, the method including generating, by a first convolutional layer of an encoder sub-network of a neural network system, output values ​​associated with a luminance channel of a frame, generating, by a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame, generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame, and generating encoded video data based on the combined representation of the frame.

[0006] In another example, an apparatus for processing video data is provided, the apparatus including a memory and a processor coupled to the memory (e.g., implemented in a circuit). In some examples, two or more processors may be coupled to the memory and used to perform one or more of the operations. The processors are configured to: use a first convolutional layer of an encoder sub-network of a neural network system to generate output values ​​associated with a luminance channel of the frame; use a second convolutional layer of the encoder sub-network to generate output values ​​associated with at least one chrominance channel of the frame; use a third convolutional layer to generate a joint representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; and generate encoded video data based on the joint representation of the frame.

[0007]

[0007] In another example, a non-transitory computer-readable medium for encoding video data is provided, storing instructions that, when executed by one or more processors, cause the one or more processors to use a first convolutional layer of an encoder sub-network of a neural network system to generate output values ​​associated with a luminance channel of the frame, use a second convolutional layer of the encoder sub-network to generate output values ​​associated with at least one chrominance channel of the frame, use a third convolutional layer to generate a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with at least one chrominance channel of the frame, and generate encoded video data based on the combined representation of the frame.

[0008] In another example, an apparatus for processing video data includes means for generating, by a first convolutional layer of an encoder sub-network of a neural network system, output values ​​associated with a luminance channel of a frame, means for generating, by a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame, means for generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame, and means for generating encoded video data based on the combined representation of the frame.

[0009] In some aspects, the third convolutional layer includes a 1×1 convolutional layer including one or more 1×1 convolution filters.

[0010] In some aspects, the above-described method, apparatus, and computer-readable medium for processing video data further comprise: processing output values ​​associated with a luminance channel of the frame using a first nonlinear layer of the encoder sub-network; and processing output values ​​associated with at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network. In such aspects, the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer.

[0011] In some aspects, the joint representation is generated by a third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0012] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise quantizing the encoded video data.

[0013] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise entropy coding the encoded video data.

[0014] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise storing the encoded video data in a memory.

[0015]

[0015] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise transmitting the encoded video data to at least one device via a transmission medium.

[0016]

[0016] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further include obtaining an encoded frame, generating, by a first convolutional layer of a decoder sub-network of the neural network system, a reconstructed output value associated with a luminance channel of the encoded frame, and generating, by a second convolutional layer of the decoder sub-network, a reconstructed output value associated with at least one chrominance channel of the encoded frame.

[0017]

[0017] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise separating a luminance channel of an encoded frame from at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network.

[0018] In some aspects, the third convolutional layer of the decoder sub-network includes a 1×1 convolutional layer that includes one or more 1×1 convolution filters.

[0019] In some aspects, the frame comprises a video frame. In some aspects, the at least one chrominance channel comprises a chrominance blue channel and a chrominance red channel. In some aspects, the frame has a luminance-chrominance (YUV) format.

[0020] In one illustrative example, a method for processing video data is provided, the method including obtaining an encoded frame, separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame, generating, by a second convolutional layer of the decoder sub-network of a neural network system, reconstructed output values ​​associated with the luminance channel of the encoded frame, generating, by a third convolutional layer of the decoder sub-network, reconstructed output values ​​associated with the at least one chrominance channel of the encoded frame, and generating an output frame including the reconstructed output values ​​associated with the luminance channel and the at least one chrominance channel.

[0021] In another example, an apparatus for processing video data is provided, the apparatus including a memory and a processor coupled to the memory (e.g., implemented in a circuit). In some examples, two or more processors may be coupled to the memory and used to perform one or more of the operations. The processor is configured to obtain an encoded frame, separate a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a first convolutional layer of a decoder sub-network, generate reconstructed output values ​​associated with the luminance channel of the encoded frame using a second convolutional layer of the decoder sub-network of a neural network system, generate reconstructed output values ​​associated with the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network, and generate an output frame including the reconstructed output values ​​associated with the luminance channel and the at least one chrominance channel.

[0022]

[0022] In another example, a non-transitory computer-readable medium for encoding video data is provided, storing instructions that, when executed by one or more processors, cause one or more processors to obtain an encoded frame; separate a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a first convolutional layer of a decoder sub-network; generate reconstructed output values ​​associated with the luminance channel of the encoded frame using a second convolutional layer of the decoder sub-network of a neural network system; generate reconstructed output values ​​associated with at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network; and generate an output frame including reconstructed output values ​​associated with the luminance channel and reconstructed output values ​​associated with at least one chrominance channel.

[0023] In another example, an apparatus for processing video data is provided, the apparatus including: means for obtaining an encoded frame; means for separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame; means for generating, by a second convolutional layer of the decoder sub-network of a neural network system, reconstructed output values ​​associated with the luminance channel of the encoded frame; means for generating, by a third convolutional layer of the decoder sub-network, reconstructed output values ​​associated with the at least one chrominance channel of the encoded frame; and means for generating an output frame including the reconstructed output values ​​associated with the luminance channel and the at least one chrominance channel.

[0024] In some aspects, the first convolutional layer of the decoder sub-network includes a 1×1 convolutional layer that includes one or more 1×1 convolution filters.

[0025]

[0025] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise: processing values ​​associated with a luminance channel of an encoded frame using a first nonlinear layer of a decoder sub-network, wherein a reconstructed output value associated with the luminance channel is generated based on the output of the first nonlinear layer; and processing values ​​associated with at least one chrominance channel of an encoded frame using a second nonlinear layer of the decoder sub-network, wherein a reconstructed output value associated with at least one chrominance channel is generated based on the output of the second nonlinear layer.

[0026] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise dequantizing samples of the encoded frames.

[0027] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise entropy decoding samples of the encoded frames.

[0028] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise storing the output frames in a memory.

[0029] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise displaying the output frames.

[0030]

[0030] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further include generating, by a first convolutional layer of an encoder sub-network of the neural network system, output values ​​associated with a luminance channel of the frame; generating, by a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame; generating, by a third convolutional layer of the encoder sub-network, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with at least one chrominance channel of the frame; and generating an encoded frame based on the combined representation of the frame.

[0031] In some aspects, the third convolutional layer of the encoder sub-network includes a 1×1 convolutional layer that includes one or more 1×1 convolution filters.

[0032]

[0032] In some aspects, the above-described methods, apparatus, and computer-readable media for processing video data further comprise processing output values ​​associated with a luminance channel of the frame using a first nonlinear layer of the encoder sub-network, and processing output values ​​associated with at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network, wherein a combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer.

[0033]

[0033] In some aspects, the combined representation of the frame is generated by a third convolutional layer of the encoder sub-network using the output of the first non-linear layer and the output of the second non-linear layer as inputs.

[0034] In some aspects, the encoded frames include encoded video frames.

[0035] In some aspects, the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel.

[0036] In some aspects, the encoded frames have a luminance-chrominance (YUV) format.

[0037] In some aspects, the device may be, or may be part of, a mobile device (e.g., a mobile phone or so-called “smartphone,” a tablet computer, or other type of mobile device), a network-connected wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer (e.g., a video server or other server device), a television, a vehicle (or a vehicular computing device or system), a camera (e.g., a digital camera, an Internet Protocol (IP) camera, etc.), a multi-camera system, a robotics device or system, an aerial device or system, or other device. In some aspects, the device includes at least one camera for capturing one or more images or video frames (or pictures). For example, the device may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more videos including one or more images and / or video frames. In some aspects, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to transmit one or more video frames and / or syntax data to at least one device over a transmission medium. In some aspects, the apparatus described above may include one or more sensors. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0038] This Summary is not intended to identify key or essential features of the claimed subject matter, nor should it be used in isolation to determine the scope of the claimed subject matter, which subject matter should be understood with reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0039]

[0039] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0040]

[0040] Exemplary embodiments of the present application are described in detail below with reference to the following figures: [Brief explanation of the drawings]

[0041] [Figure 1]

[0041] FIG. 1 illustrates an exemplary implementation of a system on a chip (SOC). [Figure 2A]

[0042] FIG. 1 is a diagram showing an example of a fully connected neural network. [Figure 2B]

[0043] FIG. 1 is a diagram showing an example of a locally coupled neural network. [Figure 2C]

[0044] FIG. 1 is a diagram showing an example of a convolutional neural network. [Figure 2D]

[0045] A detailed example of a deep convolutional network (DCN) designed to recognize visual features from images. [Figure 3]

[0046] Block diagram showing a deep convolutional network (DCN). [Figure 4]

[0047] FIG. 1 illustrates an example of a system including a device operable to perform image and / or video coding (encoding and decoding) using a neural network-based system, according to some examples. [Figure 5]

[0048] FIG. 1 illustrates an example of an end-to-end neural network-based image and video coding system for inputs having a red-green-blue (RGB) format, according to some examples. [Figure 6A]

[0049] FIG. 1 illustrates an example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples. [Figure 6B]

[0050] FIG. 1 illustrates an exemplary operation of a 1×1 convolutional layer, according to some examples. [Figure 6C]

[0051] FIG. 1 illustrates another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples. [Figure 6D]

[0052] FIG. 1 illustrates another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples. [Figure 6E]

[0053] FIG. 1 illustrates another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples. [Figure 6F]

[0054] FIG. 1 illustrates another example of a front-end neural network architecture that may be part of an end-to-end neural network-based image and video coding system, according to some examples. [Figure 7]

[0055] 1 is a flowchart illustrating an example of a process for processing video data, according to some examples. [Figure 8]

[0056] 10 is a flowchart illustrating another example of a process for processing video data, according to some examples. [Figure 9]

[0057] FIG. 1 illustrates an example computing device architecture for an example computing device capable of implementing various techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0042]

[0058] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not limiting.

[0043]

[0059] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0044]

[0060] Digital video data can contain large amounts of data, especially as the demand for high-quality video data continues to grow. For example, consumers of video data typically desire increasingly higher quality video having high fidelity, resolution, frame rates, etc. However, the large amounts of video data required to meet such demand can place a significant strain on communication networks, as well as devices that process and store the video data.

[0045]

[0061] Various techniques may be used to code video data. Video coding may be performed according to a particular video coding standard. Exemplary video coding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding, and Versatile Video Coding (VVC). Video coding often uses prediction methods such as inter-prediction or intra-prediction that exploit redundancy present within a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. As demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0046]

[0062] Described herein are systems, apparatuses, processes (also called methods), and computer-readable media (collectively referred to as “systems and techniques”) for performing image and / or video coding using one or more machine learning (ML) systems. Generally, ML is a subset of artificial intelligence (AI). ML systems may include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inference without explicit instructions. One example of an ML system is a neural network (also called an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks may be used in a variety of applications and / or devices, such as image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, among others.

[0047]

[0063] Individual nodes in a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, each node's input data may be multiplied by its corresponding weight value, and the products may be summed. The sum of the products may be adjusted by an optional bias, and an activation function is applied to the result, resulting in the node's output signal or "output activation" (sometimes called an activation map or feature map). Weight values ​​may initially be determined by an iterative flow of training data through the network (e.g., weight values ​​are established during a training phase in which the network learns how to distinguish between specific classes given their typical input data characteristics).

[0048]

[0064] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and multi-layer perceptron (MLP) neural networks, among others. For example, a convolutional neural network (CNN) is a type of feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) that collectively tile the input space. RNNs operate on the principle of saving the output of a layer and feeding this output back to the input to help predict the outcome of that layer. A GAN is a form of generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that can reasonably be from the original dataset. A GAN can include two neural networks working together, including a generative neural network that generates the synthesized output and a discriminative neural network that evaluates the output for reliability. In an MLP neural network, data may be fed into an input layer, and one or more hidden layers provide a level of abstraction for the data. Predictions may then be made on the output layer based on the abstracted data.

[0049]

[0065] In a hierarchical neural network architecture (called a deep neural network when there are multiple hidden layers), the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to a third layer of artificial neurons, and so on. A CNN, for example, may be trained to recognize a hierarchy of features. Computation in a CNN architecture may be distributed across a population of processing nodes, which may be organized in one or more computational chains. These multi-layered architectures may be trained one layer at a time and may be fine-tuned using backpropagation.

[0050]

[0066] In some aspects, the systems and techniques described herein include an end-to-end ML-based image and video coding (E2E-NNVC) system (e.g., using a neural network architecture) designed to process input data having a luminance-chrominance (YUV) input format. The YUV format includes one luminance channel (Y) and a pair of chrominance channels (U and V). The U channel can be referred to as the chrominance (or chroma) blue channel, and the V channel can be referred to as the chrominance (or chroma) red channel. In some cases, the luminance (Y) channel or component can also be referred to as the luma channel or component. In some cases, the chrominance (U and V) channels or components can also be referred to as chroma channels or components. YUV input formats can include YUV4:2:0, YUV4:4:4, and YUV4:2:2, among others. In some cases, the systems and techniques described herein may be designed to process other input formats, such as Y-chroma blue (Cb)-chroma red (Cr) (YCbCr) format, red-green-blue (RGB) format, and / or other formats. The E2E-NNVC systems described herein can encode and / or decode video data including standalone frames and / or multiple frames (also called images or pictures).

[0051]

[0067] An E2E-NNVC system is often designed as a combination of an autoencoder subnetwork (encoder subnetwork) and a second subnetwork (decoder subnetwork) (sometimes called a hyperprior network) responsible for learning a probability model on the quantization latency used for entropy coding. In some cases, there may be other subnetworks in the decoder. Such an E2E-NNVC system architecture can be viewed as a combination of a transform-plus-quantization module (or encoder subnetwork) and an entropy modeling subnetwork module.

[0052]

[0068] Most E2E-NNVC system architectures are designed to operate with non-subsampled input formats, such as RGB, YUV4:4:4, or other non-subsampled input formats. However, video coding standards such as HEVC and VVC are designed to support the YUV4:2:0 color format within their respective Main Profiles. To support the 4:2:0 YUV format, E2E-NNVC architectures designed to operate with non-subsampled input formats need to be modified.

[0053]

[0069] The systems and techniques described herein provide a front-end architecture (e.g., a sub-network) for processing one or more specific color formats (e.g., YUV4:2:0 color format) applicable to existing E2E-NNVC architectures. The systems and techniques consider different characteristics of the Y and UV channels as well as differences in resolution. For example, the Y and UV channels of a frame or portion of a frame may be input to two separate neural network layers of an encoder sub-network of a neural network system. In some examples, the two neural network layers include convolutional layers. In some aspects, the outputs of the two separate neural network layers are processed by a pair of nonlinear layers or operators of the encoder sub-network. The pair of nonlinear layers or operators may include a generalized neural activity normalization (GDN) layer or operator, a parametric rectified linear unit (PReLU) layer or operator, and / or other nonlinear layers or operators. The outputs of the two separate neural network layers (or the outputs of the nonlinear layers or operators) are combined using an additional neural network layer of the encoder sub-network.

[0054]

[0070] In some examples, the additional neural network layer is a 1x1 convolutional layer. The 1x1 convolutional layer performs per-pixel or per-value cross-channel mixing of Y and UV components (e.g., by generating a linear combination) to result in a prediction of a cross-component (e.g., cross-luminance and chrominance components) that improves coding performance. For example, cross-channel mixing of Y and UV components decorrelates the Y component from the U and V components, thereby improving coding performance (e.g., improving coding efficiency). In some cases, the 1x1 convolutional layer may include N 1x1 convolutional filters (where N is equal to an integer value corresponding to the number of channels input to the 1x1 convolutional layer). Each 1x1 convolutional filter has a respective scaling factor applied to the corresponding Nth channel of the Y component and the corresponding Nth channel of the UV component.

[0055]

[0071] The output of the additional neural network layer (e.g., a 1×1 convolutional layer) may be processed by one or more nonlinear layers of the encoder sub-network and / or one or more further neural network layers (e.g., convolutional layers). A quantization engine may perform quantization on the features output by the last neural network layer of the encoder sub-network to generate a quantized output. An entropy coding engine may entropy code the quantized output from the quantization engine to generate a bitstream. The neural network system may output the bitstream for storage, transmission to another device, a server device, or a server system, etc.

[0056]

[0072] A decoder sub-network of the neural network system or a decoder sub-network of another neural network system (in another device) can decode the bitstream. For example, an entropy decoding engine of the decoder sub-network can entropy decode the bitstream and output the entropy-decoded data to an inverse quantization engine. The inverse quantization engine can inverse quantize the data. The inverse quantized data can be processed by one or more neural network layers (e.g., convolutional layers) and / or one or more inverse nonlinear layers of the decoder sub-network. For example, after being processed by one or more convolutional layers and one or more inverse nonlinear layers, a 1×1 convolutional layer can process the data. The 1×1 convolutional layer can split the data into Y channel features and combined UV channel features. The Y channel features and combined UV channel features can be processed by two final neural network layers (e.g., two convolutional layers) and possibly two final inverse nonlinear layers. For example, the first and last neural network layer may process the Y channel features and output a reconstructed Y channel (e.g., luminance sample or pixel) for each pixel or sample of the reconstructed frame. The second and last neural network layer may process the combined UV channel features and output a reconstructed U channel (e.g., chrominance blue sample or pixel) for each pixel or sample of the reconstructed frame and a reconstructed V channel (e.g., chrominance red sample or pixel) for each pixel or sample of the reconstructed frame.

[0057]

[0073] Further details regarding the systems and techniques are described with reference to the figures.

[0058]

[0074] 1 illustrates an example implementation of a system-on-chip (SOC) 100 that may include a central processing unit (CPU) 102 or multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, among other information, may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, or a memory block 118, and / or may be distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from the memory block 118.

[0059]

[0075] The SOC 100 may also include additional processing blocks adapted for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 that may include fifth generation (5G) connectivity, fourth generation long-term evolution (4G LTE®) connectivity, Wi-Fi® connectivity, USB connectivity, Bluetooth® connectivity, etc., and a multimedia processor 112 that may, for example, detect and recognize gestures. In one implementation, the NPU is implemented within the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120 that may include a global positioning system.

[0060]

[0076] The SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, instructions loaded into the CPU 102 may include code for searching a stored multiplication result in a look-up table (LUT) corresponding to the multiplication product of the input value and the filter weight. The instructions loaded into the CPU 102 may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a look-up table hit for the multiplication product is detected. Furthermore, the instructions loaded into the CPU 102 may include code for storing the calculated multiplication product of the input value and the filter weight when a look-up table miss for the multiplication product is detected.

[0061]

[0077] SOC 100 and / or its components may be configured to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video coding) using machine learning techniques in accordance with aspects of the present disclosure described herein. By using a deep learning architecture to perform video compression and / or decompression, aspects of the present disclosure can increase the efficiency of video compression and / or decompression on a device. For example, a device using the described video coding techniques can more efficiently compress video using machine learning-based techniques and transmit the compressed video to another device, which can more efficiently decompress the video compressed using the machine learning-based techniques described herein.

[0062]

[0078] As described above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processing is performed by hidden nodes in one or more hidden layers, and output is generated via output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, deep learning networks may have a series of many hidden layers, with early layers used to determine simple, low-level characteristics of the input and later layers building a hierarchy of more complex, abstract characteristics.

[0063]

[0079] Deep learning architectures can learn hierarchies of features. For example, when presented with visual data, a first layer can be trained to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer can be trained to recognize the spectral power of specific frequencies. A second layer, taking the output of the first layer as input, can be trained to recognize combinations of features, such as simple shapes in the case of visual data or combinations of sounds in the case of auditory data. For example, higher layers can be trained to represent complex shapes in visual data or words in auditory data. Even higher layers can be trained to recognize common visual objects or spoken phrases.

[0064]

[0080] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, motor vehicle classification can benefit from first learning to recognize wheels, windshields, and other features. These features may be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0065]

[0081] Neural networks may be designed with various connection patterns. In feedforward networks, information is passed from lower layers to higher layers, with each neuron in a given layer informing neurons in higher layers. As described above, hierarchical representations may be constructed in successive layers of a feedforward network. Neural networks may also have recurrent connections or feedback connections (also called top-down). In recurrent connections, the output from a neuron in a given layer may be passed to another neuron in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks that are sequentially delivered to the neural network. The connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognizing high-level concepts may help distinguish certain low-level features of the input.

[0066]

[0082] Connections between layers of a neural network can be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, a neuron in a first layer can communicate its output to every neuron in a second layer, such that each neuron in the second layer receives input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of a locally connected neural network 204 may be configured so that each neuron in the layer has the same or similar connection pattern, but with connection strengths that may have different values ​​(e.g., 210, 212, 214, and 216). The locally connected connection pattern may result in spatially distinct receptive fields in the upper layers, as upper layer neurons in a given region receive inputs that are tuned through training to the properties of a limited portion of the total inputs to the network.

[0067]

[0083] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g., 208). The convolutional neural network may be suitable for problems in which the spatial location of the inputs is meaningful. The convolutional neural network 206 may be used to perform one or more aspects of video compression and / or decompression according to aspects of the present disclosure.

[0068]

[0084] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN 200 designed to recognize visual features from images 226 input from an image capture device 230, such as an in-car camera. The DCN 200 in this example may be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or identifying traffic signals.

[0069]

[0085] The DCN 200 may be trained using supervised learning. During training, the DCN 200 may be presented with an image, such as the speed limit sign image 226, and then a forward pass may be computed to generate the output 222. The DCN 200 may include a feature extraction section and a classification section. Upon receiving the image 226, the convolution layer 232 may apply a convolution kernel (not shown) to the image 226 to generate the first set of feature maps 218. As an example, the convolution kernel for the convolution layer 232 may be a 5×5 kernel that generates a 28×28 feature map. In this example, four different feature maps are generated in the first set of feature maps 218, so four different convolution kernels were applied to the image 226 in the convolution layer 232. A convolution kernel may also be referred to as a filter or a convolution filter.

[0070]

[0086] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0071]

[0087] 2D , the second set of feature maps 220 are convolved to generate a first feature vector 224. The first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign," "60," and "100." A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 contains one or more features.

[0072]

[0088] In this example, the probability in output 222 for "sign" and "60" is higher than the probability of other outputs 222 such as "30," "40," "50," "70," "80," "90," and "100." Prior to training, the output 222 generated by DCN 200 may be inaccurate. Therefore, an error may be calculated between output 222 and a target output. The target output is the ground truth of image 226 (e.g., "sign" and "60"). The weights of DCN 200 may then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0073]

[0089] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. In the top layer, the gradient may correspond directly to the value of the weights connecting the activating neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradient of the upper layer. The weights may then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward pass" through the neural network.

[0074]

[0090] In practice, the error gradient of the weights may be calculated over a small number of examples, such that the calculated gradient approximates the true error gradient. This approximation method may be called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the overall system no longer decreases, or until the error rate reaches a target level. After training, the DCN may be presented with new images, and a forward pass through the network may result in an output 222, which may be considered the DCN's inference or prediction.

[0075]

[0091] A deep belief network (DBN) is a probabilistic model with multiple layers of hidden nodes. DBNs may be used to extract hierarchical representations of a training dataset. DBNs may be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions in the absence of information about the class to which each input should be classified, RBMs are often used in unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the lower RBM of the DBN may be trained in an unsupervised manner and can function as a feature extractor, and the upper RBM may be trained in a supervised manner (on the joint distribution of inputs and target classes from previous layers) and can function as a classifier.

[0076]

[0092] A deep convolutional network (DCN) is a network of convolutional networks composed of additional pooling and nonlinear (e.g., regularization) layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the network weights using gradient descent.

[0077]

[0093] The DCN may be a feedforward network. In addition, as described above, connections from neurons in a first layer of the DCN to groups of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of the DCN may be exploited for high-speed processing. The computational burden of the DCN may be much less than that of a similarly sized neural network with, for example, recurrent or feedback connections.

[0078]

[0094] The processing of each layer of a convolutional network may be considered a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input may be considered three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The outputs of the convolutional connections may be considered to form feature maps in subsequent layers, with each element of the feature map (e.g., 220) receiving input from various neurons in the previous layer (e.g., feature map 218) and from each of multiple channels. The values ​​in the feature map may be further processed using nonlinearities such as rectification, max(0,x), etc. Values ​​from neighboring neurons may be further pooled, which corresponds to downsampling and can provide further local invariance and dimensionality reduction.

[0079]

[0095] 3 is a block diagram illustrating an example of a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connections and weight sharing. As shown in FIG. 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may be composed of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0080]

[0096] The convolutional layer 356 may include one or more convolutional filters, which may be applied to the input data 352 to generate feature maps. While only two convolutional blocks 354A, 354B are shown, the disclosure is not so limited; instead, any number of convolutional blocks (e.g., blocks 354A, 354B) may be included in the deep convolutional network 350, depending on design preference. The normalization layer 358 may normalize the outputs of the convolutional filters. For example, the normalization layer 358 may provide whitening or lateral inhibition. The max-pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0081]

[0097] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. Additionally, the deep convolutional network 350 may have access to other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0082]

[0098] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362A, 362B, 364 of the deep convolutional network 350, there are weights (not shown) that must be updated. The output of each of the layers (e.g., 356, 358, 360, 362A, 362B, 364) can serve as an input to a subsequent one of the layers (e.g., 356, 358, 360, 362A, 362B, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) provided in the initial convolution block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0083]

[0099] As mentioned above, digital video data can contain large amounts of data, which can place a significant strain on communication networks and devices that process and store the video data. For example, recording uncompressed video content generally results in large file sizes that increase significantly as the resolution of the recorded video content increases. In one illustrative example, uncompressed, 16-bit-per-channel video recorded at 1080p / 24 (e.g., a resolution of 1920 pixels wide by 1080 pixels high captured at 24 frames per second) may occupy 12.4 megabytes per frame or 297.6 megabytes per second. Uncompressed, 16-bit-per-channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame or 1195.2 megabytes per second.

[0084]

[0100] Network bandwidth is another constraint that can become problematic for large video files. For example, video content is often delivered over wireless networks (e.g., over LTE, LTE Advanced, New Radio (NR), WiFi, Bluetooth, or other wireless networks) and can constitute a large portion of a consumer's Internet traffic. Despite advances in the amount of available bandwidth in wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content in these networks.

[0085]

[0101] Because uncompressed video content can result in large files that may require significant memory for physical storage and significant bandwidth for transmission, video coding techniques may be utilized to compress and then decompress such video content.

[0086]

[0102] To reduce the size of video content, and therefore the amount of storage required to store it and the amount of bandwidth required to deliver it, various video coding techniques may be implemented according to specific video coding standards such as HEVC, AVC, MPEG, and VVC, among others. Video coding often uses prediction methods such as inter-prediction or intra-prediction that exploit redundancy present within a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. As demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0087]

[0103] Generally, an encoding device encodes video data according to a video coding standard to generate an encoded video bitstream. In some examples, an encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more coded video sequences. An encoding device can generate a coded representation of a picture by partitioning each picture into multiple slices. Slices are independent of other slices such that information within a slice is coded without dependency on data from other slices within the same picture. A slice includes one or more slice segments, including independent slice segments, and one or more dependent slice segments, if any, that depend on previous slice segments. In HEVC, slices are partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, along with syntax for the samples, are called a coding tree unit (CTU). A CTU is sometimes called a "treeblock" or a "largest coding unit" (LCU). A CTU is the basic processing unit for HEVC encoding. A CTU may be divided into multiple coding units (CUs) of various sizes. A CU contains an array of luma and chroma samples called a coding block (CB).

[0088]

[0104] The luma CB and the chroma CB may be further divided into prediction blocks (PBs). A PB is a block of luma or chroma component samples that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). A luma PB and one or more chroma PBs, together with associated syntax, form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) is signaled in the bitstream for each PU and used for inter prediction of the luma PB and one or more chroma PBs. The motion parameters are sometimes referred to as motion information. A CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of color component samples to which a residual transform (e.g., possibly the same two-dimensional transform) is applied to code the prediction residual signal. A transform unit (TU) represents a TB of luma samples and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

[0089]

[0105] According to the HEVC standard, transforms may be performed using TUs. TUs may be sized based on the size of the PUs within a given CU. TUs may be the same size as or smaller than the PUs. In some examples, residual samples corresponding to a CU may be subdivided into smaller units using a quadtree structure known as a residual quadtree (RQT). Leaf nodes of the RQT may correspond to TUs. Pixel difference values ​​associated with the TUs may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoding device.

[0090]

[0106] Once a picture of video data is partitioned into CUs, the encoding device predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, a prediction mode may be signaled in the bitstream using syntax data. The prediction mode may include intra prediction (or intra-picture prediction) or inter prediction (or inter-picture prediction). Intra prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra prediction, each PU is predicted from neighboring image data within the same picture using, for example, DC prediction to find the average value for the PU, planar prediction to fit a flat surface to the PU, directional prediction to extrapolate from neighboring data, or any other suitable type of prediction. Inter prediction uses the temporal correlation between pictures to derive a motion-compensated prediction for a block of image samples. For example, using inter prediction, each PU is predicted using motion-compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision as to whether to code a picture area using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0091]

[0107] After performing prediction using intra prediction and / or inter prediction, the encoding device may perform transform and quantization. For example, after prediction, the encoding device may calculate a residual value corresponding to the PU. The residual value may comprise pixel difference values ​​between a current block of pixels being coded (PU) and a predictive block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a predictive block (e.g., issuing inter prediction or intra prediction), the encoding device may generate a residual block by subtracting the predictive block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantify differences between pixel values ​​of the current block and pixel values ​​of the predictive block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0092]

[0108] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32x32, 16x16, 8x8, 4x4, or other suitable size) may be applied to the residual data in each CU. In some embodiments, TUs may be used in the transform and quantization processes implemented by the encoding device. A given CU having one or more PUs may also include one or more TUs. As described in more detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TUs to generate serialized transform coefficients for entropy coding.

[0093]

[0109] The encoding device may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0094]

[0110] Once quantization is performed, the coded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partitioning information, and any other suitable data, such as other syntax data. Different elements of the coded video bitstream may then be entropy coded by an encoding device. In some examples, the encoding device may utilize a predefined scan order to scan the quantized transform coefficients to generate serialized vectors that can be entropy coded. In some examples, the encoding device may perform adaptive scanning. After scanning the quantized transform coefficients to form vectors (e.g., one-dimensional vectors), the encoding device may entropy code the vectors. For example, the encoding device may use context-adaptive variable length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partitioned entropy coding, or another suitable entropy coding technique.

[0095]

[0111] The encoding device may send the encoded video bitstream data via a communication link to a receiving device, which may store the encoded video bitstream and / or may include a decoding device. The encoding device may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) to extract elements of one or more coded video sequences that make up the encoded video data. The decoding device may then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to a prediction stage of the decoding device. The decoding device then predicts blocks of pixels (e.g., PUs) using intra-prediction, inter-prediction, IBC, and / or other types of prediction. In some examples, the prediction is added to the output of the inverse transform (the residual data). The decoding device may output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a content consumer.

[0096]

[0112] Video coding systems and techniques defined by various video coding standards (e.g., the HEVC video coding techniques described above) may be able to preserve most of the information in raw video content and may be defined a priori based on concepts from signal processing and information theory. However, in some cases, machine learning (ML)-based image and / or video systems may provide benefits over non-ML-based image and video coding systems, such as end-to-end neural network-based image and video coding (E2E-NNVC) systems. As described above, many E2E-NNVC systems are designed as a combination of an autoencoder subnetwork (encoder subnetwork) and a second subnetwork responsible for learning a probability model on the quantization latent used for entropy coding. Such an architecture can be viewed as a combination of a transform-plus-quantization module (encoder subnetwork) and an entropy modeling subnetwork module.

[0097]

[0113] 4 depicts a system 400 including a device 402 configured to perform video encoding and decoding using an E2E-NNVC system 410. The device 402 is coupled to a camera 407 and a storage medium 414 (e.g., a data storage device). In some implementations, the camera 407 is configured to provide image data 408 (e.g., a video data stream) to a processor 404 for encoding by the E2E-NNVC system 410. In some implementations, the device 402 may be coupled to and / or include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 402 may be coupled to a microphone and / or other input device (e.g., a keyboard, a mouse, a touch input device such as a touchscreen and / or touchpad, and / or other input device). In some examples, the camera 407, the storage medium 414, the microphone, and / or other input device may be part of the device 402.

[0098]

[0114] The device 402 is also coupled to the second device 490 via a transmission medium 418, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 418 may include channels provided by a wireless network, a wired network, or a combination of wired and wireless networks. The transmission medium 418 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 418 may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from a source device to a receiving device. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi, radio frequency (RF), UWB, WiFi-Direct, cellular, Long Term Evolution (LTE), WiMax, etc.). A wired network may include any wired interface (e.g., fiber, Ethernet, powerline Ethernet, Ethernet over coaxial cable, Digital Signal Line (DSL), etc.). Wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0099]

[0115] The device 402 includes one or more processors 404 (referred to herein as “processors”) coupled to a memory 406, a first interface (“I / F1”) 412, and a second interface (“I / F2”) 416. The processor 404 is configured to receive image data 408 from a camera 407, from the memory 406, and / or from a storage medium 414. The processor 404 is coupled to the storage medium 414 via the first interface 412 (e.g., via a memory bus) and to a transmission medium 418 via a second interface 416 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0100]

[0116] The processor 404 includes an E2E-NNVC system 410. The E2E-NNVC system 410 includes an encoder portion 462 and a decoder portion 466. In some implementations, the E2E-NNVC system 410 may include one or more autoencoders. The encoder portion 462 is configured to receive input data 470 and process the input data 470 to generate output data 474 based at least in part on the input data 470.

[0101]

[0117] In some implementations, the encoder portion 462 of the E2E-NNVC system 410 is configured to perform lossy compression of the input data 470 to generate output data 474, such that the output data 474 has fewer bits than the input data 470. The encoder portion 462 may be trained to compress the input data 470 (e.g., an image or video frame) based on any previous representation (e.g., one or more previously reconstructed frames) without using motion compensation. For example, the encoder portion 462 may compress a video frame using only video data from that video frame and without using any data from previously reconstructed frames. The video frames processed by the encoder portion 462 may be referred to herein as intra-predicted frames (I-frames). In some examples, the I-frames may be generated using traditional video coding techniques (e.g., according to HEVC, VVC, MPEG-4, or other video coding standards). In such examples, the processor 404 may include or be coupled to a video coding device (e.g., an encoding device) configured to perform block-based intra prediction, such as that described above with respect to the HEVC standard. In such examples, the E2E-NNVC system 410 may be excluded from the processor 404.

[0102]

[0118] In some implementations, the encoder portion 462 of the E2E-NNVC system 410 may be trained to compress input data 470 (e.g., video frames) using motion compensation based on a previous representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 462 may compress a video frame using video data from that video frame and using data from a previously reconstructed frame. The video frames processed by the encoder portion 462 may be referred to herein as intra-predicted frames (P-frames). Motion compensation may be used to determine data for a current frame by describing how pixels from a previously reconstructed frame move to new positions within the current frame along with residual information.

[0103]

[0119] As shown, the encoder portion 462 of the E2E-NNVC system 410 may include a neural network 463 and a quantizer 464. The neural network 463 may include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures that generate intermediate data 472. The intermediate data 472 is input to the quantizer 464. Examples of components that may be included in the encoder portion 462 are shown in FIGS. 6A-6E.

[0104]

[0120] The quantizer 464 is configured to perform quantization and possibly entropy coding of the intermediate data 472 to generate output data 474. The output data 474 may include quantized (and possibly entropy coded) data. The quantization operation performed by the quantizer 464 may result in the generation of a quantized code (or data representing the quantized code generated by the E2E-NNVC system 410) from the intermediate data 472. The quantized code (or data representing the quantized code) may also be referred to as a latent code (denoted as z) or a latent. The entropy model applied to the latent may be referred to herein as a “prior.” In some examples, the quantization and / or entropy coding operations may be performed using existing quantization and entropy coding operations performed when encoding and / or decoding video data according to existing video coding standards. In some examples, the quantization and / or entropy coding operations may be performed by the E2E-NNVC coding system 410. In one illustrative example, the E2E-NNVC system 410 may be trained using supervised training, during which residual data is used as input and quantization codes and entropy codes are used as known outputs (labels).

[0105]

[0121] The decoder portion 466 of the E2E-NNVC system 410 is configured to receive output data 474 (e.g., directly from the quantizer 464 and / or from the storage medium 414). The decoder portion 466 can process the output data 474 to generate a representation 476 of the input data 470 based at least in part on the output data 474. In some examples, the decoder portion 466 of the E2E-NNVC system 410 includes a neural network 468, which may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures. Examples of components that may be included in the decoder portion 466 are shown in FIGS. 6A-6E.

[0106]

[0122] The processor 404 is configured to send the output data 474 to at least one of the transmission medium 418 or the storage medium 414. For example, the output data 474 may be stored in the storage medium 414 for later retrieval and decoding (or reconstruction) by the decoder portion 466 to generate a representation 476 of the input data 470 as reconstructed data. The reconstructed data may be used for various purposes, such as for playback of the video data encoded / compressed to generate the output data 474. In some implementations, the output data 474 may be decoded in another decoder device matching the decoder portion 466 (e.g., in the device 402, in the second device 490, or in another device) to generate the representation 476 of the input data 470 as reconstructed data. For example, the second device 490 may include a decoder matching (or substantially matching) the decoder portion 466, and the output data 474 may be transmitted to the second device 490 via the transmission medium 418. The second device 490 can process the output data 474 to generate a representation 476 of the input data 470 as recovered data.

[0107]

[0123] The components of system 400 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0108]

[0124] Although system 400 is shown to include several components, one skilled in the art will appreciate that system 400 can include more or fewer components than those shown in Figure 4. For example, system 400 can also include, or be part of, a computing device including input and output devices (not shown). In some implementations, system 400 may also include, or be part of, a computing device that includes one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) in communication with and / or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and a baseband processor per wireless interface) for performing wireless communications, one or more wired interfaces (e.g., serial interfaces such as Universal Serial Bus (USB) inputs, Lightning connectors, and / or other wired interfaces) for performing communications via one or more hardware connections, and / or other components not shown in FIG. 4 .

[0109]

[0125] In some implementations, system 400 may be implemented locally by and / or included in a computing device. For example, the computing device may include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head-mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses, or other AR device), a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), a television, and / or any other computing device having the resource capabilities to perform the techniques described herein.

[0110]

[0126] In one example, the E2E-NNVC system 410 may be incorporated into a portable electronic device including a memory 406 coupled to a processor 404 and configured to store instructions executable by the processor 404, and a wireless transceiver coupled to an antenna and the processor 404 and operable to transmit output data 474 to a remote device.

[0111]

[0127] E2E-NNVC systems are typically designed to process RGB inputs. Examples of image and video coding schemes targeting RGB inputs are described in J. Balle, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, "Variational image compression with a scale hyperprior," ICLR, 2018 (referred to as the "J. Balle Paper") and D. Minnen, J. Balle, and G. Toderici, "Joint Autoregressive and Hierarchical Priors for Learned Image Compression," CVPR, 2018 (referred to as the "D. Minnen Paper"), which are incorporated herein by reference in their entireties for all purposes.

[0112]

[0128] FIG. 5 is a diagram showing an example of the E2E-NNVC system described in the J. Balle Paper. a Subnetwork and g s The sub-networks correspond to the encoder sub-network (e.g., encoder portion 462) and the decoder sub-network (e.g., decoder portion 466), respectively. a Subnetwork and g s The subnetwork is designed for a three-channel RGB input, and all three R, G, and B input channels pass through and are processed by the same neural network layers (convolutional layers and generalized neural activity normalization (GDN) layers). The neural network layers can include convolutional layers that perform convolution operations and inverse GDN (IGDN) nonlinear layers that implement local neural activity normalization. Local neural activity normalization is a type of transformation that has been shown to be particularly suitable for image density modeling and compression. E2E-NNVC systems (such as those shown in Figure 5) target input channels with similar statistical properties, such as RGB data (where the statistical properties of the different R, G, and B channels are similar).

[0113]

[0129] Although E2E-NNVC systems are typically designed to process RGB inputs, most image and video coding systems use YUV input formats (e.g., often the YUV420 format). The chrominance (U and V) channels in a YUV format may be subsampled relative to the luminance (Y) channel. The subsampling has minimal impact on visual quality (e.g., no significant or noticeable impact on visual quality). Subsampled formats include the YUV420 format, the YUV422 format, and / or other YUV formats. Correlation between channels is reduced in YUV formats, which may not be true for other color formats (e.g., RGB formats). Furthermore, the statistics of the luminance (Y) channel and the chrominance (U and V) channels are different. For example, the U and V channels have less variance compared to the luminance channel, whereas in RGB formats, the statistical characteristics of the R, G, and B channels are similar. A video coder-decoder (or codec) is designed according to the input characteristics of the data (e.g., the codec can encode and / or decode data according to the input format of the data). For example, if the chrominance channels of a frame are subsampled (e.g., the chrominance channels are at half the resolution compared to the luma channels), when the codec predicts blocks of the frame for motion compensation, the luma blocks are twice as large in both width and height compared to the chrominance blocks. In another example, the codec can determine, among other things, how many pixels are to be coded or decoded for chrominance and luma.

[0114]

[0130] If RGB input data (which, as described above, most E2E-NNVC systems are designed to process) were replaced with YUV4:4:4 input data (all channels having the same dimensions), the performance of the E2E-NNVC system processing the input data would degrade due to the different statistical characteristics of the luminance (Y) and chrominance (U and V) channels. As described above, the chrominance (U and V) channels are subsampled in some YUV formats, such as YUV420. For example, for content having a YUV4:2:0 format, the resolution of the U and V channels is half that of the Y channel (the U and V channels have one-quarter the size of the Y channel due to their half width and half height). Such subsampling may make the input data incompatible with the input of the E2E-NNVC system. The input data is information that the E2E-NNVC system is attempting to encode and / or decode (e.g., a YUV frame including three channels, including the luminance (Y) and chrominance (U and V) channels). Many neural network-based systems assume that all channel dimensions of the input data are the same, and therefore feed all of the input channels into the same network. In such cases, the outputs of some operations may be added (e.g., using matrix addition), in which case the channel dimensions must be the same.

[0115]

[0131] In some examples, to address such issues, the Y channel may be subsampled to four half-resolution Y channels. The four half-resolution Y channels may be combined with two chrominance channels to result in six input channels. The six input channels may be input or fed into an E2E-NNVC system designed for RGB input. Such an approach may address issues related to differences in resolution of the luminance (Y) channel and the chrominance (U and V) channels. However, inherent differences between the luminance (Y) channel and the chrominance (U and V) channels still exist, resulting in poor coding (e.g., encoding and / or decoding) performance.

[0116]

[0132] As described above, systems and techniques for performing image and / or video coding using one or more ML-based systems are described herein. The systems and techniques described herein provide a front-end architecture (e.g., a new sub-network such as an end-to-end neural network-based image and video coding (E2E-NNVC) system) designed to process input data having a luminance-chrominance (YUV) input format (e.g., YUV420, YUV444, YUV422, among others). In some examples, the front-end architecture is configured to accommodate a YUV4:2:0 input format in an E2E-NNVC system designed for an RGB input format. As described above, the front-end architecture is applicable to many E2E-NNVC architectures (e.g., including those described in the J. Balle and D. Minnen papers). The systems and techniques described herein take into account the different characteristics of the luminance (Y) channel and the chrominance (U and V) channels, as well as differences in the resolution of the luminance (Y) channel and the chrominance (U and V) channels. The E2E-NNVC system can encode and / or decode video data including standalone frames (or images) and / or multiple frames.

[0117]

[0133] In some examples, the systems and techniques described herein may first input or feed the Y and UV channels into two separate layers. The E2E-NNVC system may then combine data associated with the Y and UV channels after a certain number of layers (e.g., after the first pair of convolutional and nonlinear layers, as shown in Figures 6A-6E described below, or after other layers). Because the U and V chrominance components are subsampled relative to the luminance (Y) channel, subsampling in the first convolutional layer may be skipped, and a convolutional (e.g., CNN) kernel of a specific size (e.g., having a size of (N / 2+1) × (N / 2+1)) may be used for the subsampled input to the chrominance (U and V) channels. Then, a CNN kernel (e.g., an N × N CNN kernel) having a different size compared to the kernel used for the chrominance (U and V) channels may be used for the luminance (Y) channel. The two branches of the front-end architecture (carrying luma and chroma channels or component information separately) may be combined using a convolutional layer (e.g., a 1×1 convolutional layer) that combines values ​​across channels. The use of a 1×1 convolutional layer can provide various benefits described herein, including improved coding efficiency.

[0118]

[0134] 6A-6F illustrate illustrative examples of front-end architectures of neural network systems. In some examples, the front-end architectures of FIGS. 6A-6F may be part of an E2E-NNVC system designed to process (encode and / or decode) data having a YUV 4:2:0 format. For example, the front-end architectures may be configured to process input data having a YUV 4:2:0 format. The front-end architectures of FIGS. 6A, 6C, 6D, and 6E have two different nonlinear operators applied after the 1×1 convolution layer. For example, a generalized neural activity normalization (GDN) operator is used in the architecture of FIG. 6A, and a parametric rectified linear unit (PReLU) nonlinear operator is applied in the architectures of FIGS. 6C-6E. In some examples, neural network architectures similar to those shown in Figures 6A and 6C-6F may be used to encode and / or decode other types of YUV content (e.g., content having a YUV 4:4:4 format, a YUV 4:2:2 format, etc.) and / or content having other input formats.

[0119]

[0135] For example, FIG. 6A illustrates an example of a front-end neural network system or architecture that can be configured to work directly with 4:2:0 input (Y, U, and V) data. As shown in FIG. 6A, in the encoder sub-network of the neural network system, the separated luma and chroma channels (luma (Y) channel 602 and chroma (U and V) channels 604) are combined using a 1×1 convolution layer 606, and then a GDN nonlinear operator 608 is applied. Similar operations are performed on the decoder sub-network of the neural network system, but in the reverse order. For example, as shown in FIG. 6A, an inverse GDN (IGDN) operator 609 is applied, where the Y channel and the U, V channels are separated using a 1×1 convolution layer 613, and the separate Y channel and the U, V channels are processed using respective IGDN 615, 616 and convolution layers 617, 618.

[0120]

[0136] For example, the first two neural network layers in the encoder sub-network of the neural network system of Figure 6A include a first convolutional layer 611 (denoted as Nconv|3x3|↓1), a second convolutional layer 610 (denoted as Nconv|5x5|↓2), a first GDN layer 614, and a second GDN layer 612. The last two neural network layers in the decoder sub-network of the front-end neural network architecture of Figure 6A include a first inverse GDN (IGDN) layer 616 for generating the reconstructed chrominance (U and V) components of the frame, a second inverse GDN (IGDN) layer 615, a first convolutional layer 618 (denoted as 2conv|3x3|↑1), and a second convolutional layer 617 (denoted as 1conv|5x5|↑2) for generating the reconstructed luminance (Y) component of the frame. The notation "Nconv" refers to the number of output channels (N) (corresponding to the number of output features) of a given convolutional layer (with the value of N defining the number of output channels). The notations 3x3 and 5x5 refer to the size of the respective convolutional kernels (e.g., 3x3 kernel and 5x5 kernel). The notations "↓1" and "↓2" refer to stride values, where ↓1 refers to a stride of 1 (for downsampling, indicated by "↓") and ↓2 refers to a stride of 1 (for downsampling). The notations "↑1" and "↑2" refer to stride values, where ↑1 refers to a stride of 1 (for upsampling, indicated by "↑") and ↑2 refers to a stride of 1 (for upsampling).

[0121]

[0137] For example, the convolution layer 610 downsamples the input luma channel by a factor of four by applying a 5×5 convolution filter in the horizontal and vertical dimensions with a stride value of 2. The resulting output of the convolution layer 610 is N arrays (corresponding to the N channels) of feature values. The convolution layer 611 processes the input chroma (U and V) channels 604 by applying a 3×3 convolution filter in the horizontal and vertical dimensions with a stride value of 1. The resulting output of the convolution layer 611 is N arrays (corresponding to the N channels) of feature values. The arrays of feature values ​​output by the convolution layer 610 have the same dimensions as the arrays of feature values ​​output by the convolution layer 611. The GDN layer 612 can then process the feature values ​​output by the convolution layer 610, and the GDN layer 614 can process the feature values ​​output by the convolution layer 611.

[0122]

[0138] The 1×1 convolution layer 606 can then process the feature values ​​output by the GDN layers 612, 614. The 1×1 convolution layer 606 can generate a linear combination of features associated with the luma channel 602 and the chroma channels 604. The linear combination operation acts as a cross-channel mixing per Y and UV component value, resulting in a prediction of the cross-component (e.g., cross-luminance and chrominance components) that improves coding performance. Each 1×1 convolution filter of the 1×1 convolution layer 606 can include a respective scaling factor applied to the corresponding Nth channel of the luma channel 602 and the corresponding Nth channel of the chroma channel 604.

[0123]

[0139] FIG. 6B illustrates an example operation of the 1×1 convolution layer 638. As discussed above, N represents the number of output channels. As shown in FIG. 6B, 2N channels are provided as inputs to the 1×1 convolution layer 638, including an N channel chroma (combined U and V) output 632 and an N channel luma (Y) output 634. In the example of FIG. 6B, the value of N is equal to 2, indicating two channels of values ​​for the N channel chroma output 632 and two channels of values ​​for the N channel luma output 634. Referring to FIG. 6A, the N channel chroma output 632 may be an output from the GDN layer 614, and the N channel luma output 634 may be an output from the GDN layer 612. However, in other examples, the N channel chroma output 632 and the N channel luma output 634 may be outputs from other nonlinear layers (e.g., from pReLU layers 652 and 654, respectively, in FIG. 6D and from pReLU layers 662 and 664, respectively, in FIG. 6E) or directly from convolutional layers (e.g., from convolutional layers 670 and 671, respectively, in FIG. 6F).

[0124]

[0140] The 1×1 convolution layer 638 processes 2N channels, performs feature-specific linear combinations of the 2N channels, and then outputs N channel sets of features or coefficients. The 1×1 convolution layer 638 includes two 1×1 convolution filters (based on N=2). The first 1×1 convolution filter is denoted by a value of S1, and the second 1×1 convolution filter is denoted by a value of S2. The value of S1 represents a first scaling factor, and the value of S2 represents a second scaling factor. In one illustrative example, the value of S1 is equal to 3, and the value of S2 is equal to 4. Each of the 1×1 convolution filters of the 1×1 convolution layer 638 has a stride value of 1, indicating that scaling factors S1 and S2 are applied to each value at the UV output 632 and the Y output 634.

[0125]

[0141] For example, a scaling factor S1 of a first 1×1 convolution filter is applied to each value in the first channel (C1) of the UV output 632 and each value in the first channel (C1) of the Y output 634. After each value in the first channel (C1) of the UV output 632 and each value in the first channel (C1) of the Y output 634 is scaled by the scaling factor S1 of the first 1×1 convolution filter, the scaled value is combined into the first channel (C1) of the output values ​​639. A scaling factor S2 of a second 1×1 convolution filter is applied to each value in the second channel (C2) of the UV output 632 and each value in the second channel (C2) of the Y output 634. After each value in the second channel (C2) of the UV output 632 and each value in the second channel (C2) of the Y output 634 is scaled by the scaling factor S2 of the second 1×1 convolution filter, the scaled value is combined into the second channel (C2) of the output values ​​639. As a result, the four Y and UV channels (two Y channels and two combined UV channels) are mixed and combined into two output channels C1 and C2.

[0126]

[0142] Returning to FIG. 6A , the output of the 1×1 convolutional layer 606 is processed by additional GDN layers and additional convolutional layers of the encoder sub-network. A quantization engine 620 may perform quantization on the features output by the final neural network layer 619 of the encoder sub-network to generate a quantized output. An entropy coding engine 621 may entropy code the quantized output from the quantization engine 620 to generate a bitstream. As shown in FIG. 6A , the entropy coding engine 621 may use priors generated by a hyperprior network to perform the entropy coding. The neural network system may output the bitstream for storage, transmission to another device, a server device, or a server system, and / or otherwise.

[0127]

[0143] A decoder sub-network of the neural network system or a decoder sub-network of another neural network system (in another device) can decode the bitstream. For example, as shown in FIG. 6A, an entropy decoding engine 622 of the decoder sub-network can entropy decode the bitstream and output the entropy-decoded data to an inverse quantization engine 623. The entropy decoding engine 622 can use priors generated by a hyper-prior network to perform the entropy decoding, as shown in FIG. 6A. The inverse quantization engine 623 can inverse quantize the data. The inverse quantized data can be processed by several convolutional layers and several inverse GDNs (IGDNs) of the decoder sub-network.

[0128]

[0144] After being processed by the IGDN layer 609, the data can be processed by a 1x1 convolution layer 613. The 1x1 convolution layer 613 can include 2N convolution filters that can split the data into Y channel features and combined UV channel features. For example, each of the N channels output by the IGDN layer 609 can be processed using 2N 1x1 convolutions (which result in scaling) in the 1x1 convolution layer 613. The scaling factor n corresponding to the output channel (out of a total of 2N output channels) applied to the N input channels can be i For each scaling factor n, the decoder sub-network can perform summations across the N input channels, resulting in 2N outputs. In one illustrative example, for a scaling factor n, the decoder sub-network can apply the scaling factor n to the N input channels and sum the results, resulting in one output channel. The decoder sub-network can apply 2N different scaling factors (e.g., scaling factor n, scaling factor n, scaling factor n). 2N This operation can be performed for up to

[0129]

[0145] The Y channel features output by the 1×1 convolution layer 613 may be processed by the IGDN 615. The combined UV channel features output by the 1×1 convolution layer 613 may be processed by the IGDN 616. The convolution layer 617 may process the Y channel features and output a reconstructed Y channel or reconstructed frame sample per pixel (e.g., luminance sample or pixel), shown as a reconstructed Y component 624. The convolution layer 618 may process the combined UV channel features and output a reconstructed U channel or reconstructed frame sample per pixel (e.g., chrominance blue sample or pixel) and a reconstructed V channel or reconstructed frame sample per pixel (e.g., chrominance red sample or pixel), shown as a reconstructed U and V component 625.

[0130]

[0146] 6C illustrates another example of a front-end neural network system or architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in FIG. 6C , in the encoder sub-network of the neural network system, the separated luma and chroma channels (luma channel 642 and chroma channel 644) are combined using a 1×1 convolutional layer 648 (similar to that described above with respect to the 1×1 convolutional layer 606 of FIG. 6A ), and then a pReLU nonlinear operator 649 is applied. In other examples, an operator other than a pReLU nonlinear operator may be applied. Similar operations are performed by the decoder sub-network of the neural network system of FIG. 6C (similar to that described above with respect to FIG. 6A ), but in the reverse order (e.g., the pReLU operator is applied, the Y channel and the U, V channels are separated using a 1×1 convolutional layer, and the separate Y channel and the U, V channels are processed using an inverse IGDN layer and a convolutional layer, respectively).

[0131]

[0147] Compared to the E2E-NNVC system (neural network-based codec) described in Figure 5, the input processing of the front-end architecture in Figures 6A and 6C is a and the corresponding g (on the decoder side) s 5. The first convolutional layer (e.g., convolutional layer 610 of FIG. 6A and convolutional layer 646 of FIG. 6C) used to process the Y component, denoted Nconv|5x5|↓2, can be the same as or similar to the first convolutional layer 510 of FIG. 5. Similarly, the second convolutional layer (denoted 1conv|5x5|↑2) of the decoder sub-networks of FIGS. 6A and 6C used to generate the restored luminance (Y) component is modified by separate processing of the Y and UV channels in the first two network layers in FIG. 5. s 5 , the U and V chrominance channels are processed by the architectures of FIGS. 6A and 6C using a separate convolutional layer denoted Nconv|3x3|↓1 (e.g., a separate CNN such as convolutional layer 611 of FIG. 6A or convolutional layer 647 of FIG. 6C ) with a kernel that has half the size of the Y kernel of the Nconv|5x5|↓2 convolutional layer 610 of FIG. 6A or the Nconv|5x5|↓2 convolutional layer 646 of FIG. 6C (there is no downsampling corresponding to a stride equal to 1), followed by specific GDN layers (one GDN layer for luminance Y and one GDN layer for chrominance U and V).

[0132]

[0148] After the convolutional layers (the first pair of Nconv|5x5|↓2 and Nconv|3x3|↓1 layers of the CNN) and the GDN layer in FIGS. 6A and 6C, the representations and features of the luminance (Y) and chrominance (U and V) channels (e.g., transformed or filtered versions of the input channels) have the same dimensions and are then combined using the 1× convolutional layer 606 in FIG. 6A or the 1× convolutional layer 648 in FIG. 6C. For example, the luminance (Y) channel is twice the size of the chrominance (U and V) channels in each dimension in the YUV4:2:0 format. When the chrominance (U and V) channels are subsampled by 2, the output generated based on processing those channels has the same dimensions as the conv2d output of the luminance channel (because the luminance channel is not subsampled). Separate normalization of the channels addresses differences in variability between the luminance and chrominance channels. As mentioned above, a nonlinear operator is then applied (e.g., using GDN608 or pReLU649) before using three or more convolutional layers until a quantization step is reached.

[0133]

[0149] In the decoder sub-networks of the architectures of Figures 6A and 6C, separate IGDN and convolutional layers are used to separately generate the reconstructed luminance (Y) and chrominance (U and V) components. For example, convolutional layer 618 in Figure 6A (a 2conv|3x3|↑1 layer in the decoder sub-network) used to generate reconstructed chrominance (U and V) components 625 has a kernel size that is approximately half the size (there is no upsampling corresponding to a stride equal to 1) of the kernel used in convolutional layer 617 (a 1conv|5x5|↑2 layer in the decoder sub-network) used to generate reconstructed luminance (Y) component 624.

[0134]

[0150] 6D is a diagram illustrating another example of a front-end neural network architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in FIG. 6D, at the encoder side, the branched luma and chroma channels are combined using a 1×1 convolutional layer, and then a pReLU nonlinear operator is applied. Compared to the architectures shown in FIG. 6A and FIG. 6C, the GDN layers in the luma and chroma branches are replaced with pReLU operators.

[0135]

[0151] 6E is a diagram illustrating another example of a front-end neural network architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in FIG. 6E, at the encoder side, the split luma and chroma channels are combined using a 1×1 convolutional layer, and then a pReLU nonlinear operator is applied. Compared to the architectures shown in FIGS. 6A, 6C, and 6D, all GDN layers in the architecture of FIG. 6E are replaced with pReLU operators.

[0152] FIG. 6F illustrates another example of a front-end neural network architecture that can be configured to operate directly with 4:2:0 input (Y, U, and V) input data. As shown in FIG. 6F, at the encoder side, the split luma and chroma channels are combined using a 1×1 convolutional layer. Compared to the architectures shown in FIGS. 6A-6E, all GDN layers are completely removed, and no nonlinear activation operators are used between the convolutional layers.

[0136]

[0153] The neural network architecture designs shown in Figures 6C-6F can be used to reduce the GDN layer (e.g., as shown in the architecture of Figure 6C) or to completely remove the GDN layer (e.g., as shown in the architectures of Figures 6E and 6F).

[0137]

[0154] In some examples, the systems and techniques described herein may be used in conjunction with other encoder-decoder sub-networks that use convolution (e.g., CNN) and normalization stages at the input of neural network-based coding systems.

[0138]

[0155] 7 is a flow chart illustrating an example of a process 700 for processing video using one or more of the machine learning techniques described herein. At block 702, the process 700 includes generating, by a first convolutional layer of an encoder sub-network of a neural network system, an output value associated with a luminance channel of a frame.

[0139]

[0156] At block 704, the process 700 includes generating, by a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame. At block 706, the process 700 includes generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame. In some cases, the third convolutional layer includes a 1×1 convolutional layer including one or more 1×1 convolution filters (e.g., the 1×1 convolutional layer of the encoder sub-networks of FIGS. 6A-6F). At block 708, the process 700 includes generating encoded video data based on the combined representation of the frame.

[0140]

[0157] In some examples, the process 700 includes processing output values ​​associated with a luminance channel of the frame using a first nonlinear layer of the encoder sub-network. The process 700 may include processing output values ​​associated with at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network. In such examples, the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer. In some cases, the combined representation of the frame is generated by a third convolutional layer (e.g., the 1×1 convolutional layer of the encoder sub-networks of FIGS. 6A-6F) using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0141]

[0158] In some examples, the process 700 includes quantizing the encoded video data (e.g., using the quantization engine 620). In some examples, the process 700 includes entropy coding the encoded video data (e.g., using the entropy coding engine 621). In some examples, the process 700 includes storing the encoded video data in a memory. In some examples, the process 700 includes transmitting the encoded video data over a transmission medium to at least one device.

[0142]

[0159] In some examples, the process 700 includes obtaining an encoded frame. The process 700 may include generating, by a first convolutional layer of a decoder sub-network of a neural network system, a reconstructed output value associated with a luminance channel of the encoded frame. The process 700 may further include generating, by a second convolutional layer of the decoder sub-network, a reconstructed output value associated with at least one chrominance channel of the encoded frame. In some examples, the process 700 includes separating, using a third convolutional layer of the decoder sub-network, the luminance channel of the encoded frame from at least one chrominance channel of the encoded frame. In some cases, the third convolutional layer of the decoder sub-network includes a 1×1 convolutional layer including one or more 1×1 convolution filters (e.g., the 1×1 convolutional layer of the decoder sub-networks of FIGS. 6A-6F ).

[0143]

[0160] In some examples, the frame comprises a video frame. In some examples, the at least one chrominance channel comprises a chrominance blue channel and a chrominance red channel. In some examples, the frame has a luminance-chrominance (YUV) format.

[0144]

[0161] FIG. 8 is a flowchart illustrating an example of a process 800 for processing video using one or more of the machine learning techniques described herein. At block 802, the process 800 includes obtaining an encoded frame. At block 804, the process 800 includes separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame. In some cases, the first convolutional layer of the decoder sub-network includes a 1×1 convolutional layer including one or more 1×1 convolution filters (e.g., the 1×1 convolutional layer of the decoder sub-networks of FIGS. 6A-6F ). At block 806, the process 800 includes generating, by a second convolutional layer of a decoder sub-network of a neural network system, a reconstructed output value associated with the luminance channel of the encoded frame. At block 808, the process 800 includes generating, by a third convolutional layer of the decoder sub-network, a reconstructed output value associated with at least one chrominance channel of the encoded frame. At block 810, the process 800 includes generating an output frame that includes a reconstructed output value associated with the luminance channel and a reconstructed output value associated with at least one chrominance channel.

[0145]

[0162] In some examples, the process 800 includes processing values ​​associated with a luminance channel of the encoded frame using a first nonlinear layer of a decoder sub-network. The reconstructed output value associated with the luminance channel is generated based on an output of the first nonlinear layer. The process 800 may include processing values ​​associated with at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder sub-network. The reconstructed output value associated with the at least one chrominance channel is generated based on an output of the second nonlinear layer.

[0146]

[0163] In some examples, the process 800 includes inverse quantizing samples of the encoded frame (e.g., by the inverse quantization engine 623). In some examples, the process 800 includes entropy decoding samples of the encoded frame (e.g., by the entropy decoding engine 622). In some examples, the process 800 includes storing the output frame in memory. In some examples, the process 800 includes displaying the output frame.

[0147]

[0164] In some examples, the process 800 includes generating, by a first convolutional layer of an encoder subnetwork of a neural network system, output values ​​associated with a luminance channel of the frame. The process 800 may include generating, by a second convolutional layer of the encoder subnetwork, output values ​​associated with at least one chrominance channel of the frame. The process 800 may further include generating, by a third convolutional layer of the encoder subnetwork, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with at least one chrominance channel of the frame. The process 800 may include generating an encoded frame based on the combined representation of the frame. In some cases, the third convolutional layer of the encoder subnetwork includes a 1×1 convolutional layer including one or more 1×1 convolutional filters (e.g., the 1×1 convolutional layer of the encoder subnetworks of FIGS. 6A-6F ).

[0148]

[0165] In some examples, the process 800 includes processing output values ​​associated with a luminance channel of the frame using a first nonlinear layer of the encoder sub-network and processing output values ​​associated with at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network. In such examples, the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer. In some examples, the combined representation of the frame is generated by a third convolutional layer of the encoder sub-network using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0149]

[0166] In some examples, the encoded frame comprises an encoded video frame. In some examples, the at least one chrominance channel comprises a chrominance blue channel and a chrominance red channel. In some examples, the encoded frame has a luminance-chrominance (YUV) format.

[0150]

[0167] In some examples, the processes described herein (e.g., process 700, process 800, and / or other processes described herein) may be performed by a computing device or apparatus, such as a computing device having computing device architecture 900 shown in FIG. 9. In one example, process 700 and / or process 800 may be performed by a computing device having computing device architecture 900 that implements one of the neural network architectures shown in FIGS. 6A-6F. In some examples, the computing device may include a mobile device (e.g., a mobile phone, a tablet computing device, etc.), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device in a vehicle), a robotic device, and / or any other computing device having the resource capabilities to perform the processes described herein, including process 700 and / or process 800.

[0151]

[0168] In some cases, a computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers, or combined transmitter-receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other components configured to perform the steps of processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0152]

[0169] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0153]

[0170] Processes 700 and 800 are illustrated as logical flow diagrams, whose operations represent sequences of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be limiting, and any number of the described actions may be combined in any order and / or in parallel to implement a process.

[0154]

[0171] Additionally, the processes described herein (including process 700, process 800, and / or other processes described herein) may be executed under the control of one or more computer systems configured of executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As discussed above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0155]

[0172] 9 shows an exemplary computing device architecture 900 of an exemplary computing device that may implement various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other device. For example, the computing device architecture 900 may implement the system of FIG. 6. The components of the computing device architecture 900 are shown in electrical communication with each other using connections 905, such as a bus. The exemplary computing device architecture 900 includes a processing unit (CPU or processor) 910 and computing device connections 905 that couple various computing device components, including computing device memory 915, such as read-only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910.

[0156]

[0173] The computing device architecture 900 may include a cache of high-speed memory directly connected to the processor 910, in close proximity to the processor 910, or integrated as part of the processor 910. The computing device architecture 900 may copy data from the memory 915 and / or the storage device 930 to the cache 912 for quick access by the processor 910. In this manner, the cache may provide performance improvements that avoid delays for the processor 910 while waiting for data. These and other modules may control or be configured to control the processor 910 to perform various actions. Other computing device memory 915 may also be available for use. The memory 915 may include multiple different types of memory with different performance characteristics. The processor 910 may include any general-purpose processor, hardware or software services, such as service 1 932, service 2 934, and service 3 936 stored in the storage device 930, configured to control the processor 910, as well as special-purpose processors in which software instructions are incorporated into the processor design. Processor 910 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0157]

[0174] To enable user interaction with computing device architecture 900, input device(s) 945 can represent any number of input mechanisms, such as a microphone for audio, a touchscreen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Output device(s) 935 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with computing device architecture 900. Communications interface 940 can generally orchestrate and manage user input and computing device output. There is no restriction to operating on any particular hardware configuration, and thus, basic features herein may be readily substituted for improved hardware or firmware configurations as they are developed.

[0158]

[0175] The storage device 930 is non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 925, a read-only memory (ROM) 920, and hybrids thereof. The storage device 930 may include services 932, 934, 936 for controlling the processor 910. Other hardware or software modules are contemplated. The storage device 930 may be connected to the computing device connections 905. In one aspect, a hardware module that performs a specific function may include software components stored on a computer-readable medium along with the necessary hardware components, such as the processor 910, connections 905, output devices 935, etc., to perform that function.

[0159]

[0176] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop computer, vehicle, drone, or other device) that includes or is coupled to one or more active depth-sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0160]

[0177] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, or a processing system). As used herein, a device may be any electronic device having one or more parts capable of implementing at least some portions of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a specific configuration, type, or number of objects. Furthermore, the term "system" is not limited to multiple components or a specific embodiment. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. The following description and examples use the term "system" to describe various aspects of the present disclosure, but the term "system" is not limited to a specific configuration, type, or number of objects.

[0161]

[0178] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a software-implemented method, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0162]

[0179] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process terminates when the operations of a process are completed, but may have additional steps not included in the diagrams. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0163]

[0180] The processes and methods according to the above-described examples may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0164]

[0181] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions and / or data. Computer-readable media may include non-transitory media on which data may be stored and which do not include carrier waves and / or transitory electronic signals propagating wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as flash memory, memories or memory devices, magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, compact discs (CDs) or digital versatile discs (DVDs), among others. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0165]

[0182] In some embodiments, computer-readable storage devices, media, and memories can include cables or wireless signals containing bitstreams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0166]

[0183] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) to perform the necessary tasks may be stored on a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing within a single device, as further examples.

[0167]

[0184] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0168]

[0185] While the foregoing description describes aspects of the present application with reference to specific embodiments thereof, those skilled in the art will recognize that the present application is not limited thereto. Accordingly, while exemplary embodiments of the present application have been described in detail herein, it should be understood that, except as limited by the prior art, the inventive concepts may in some cases be embodied and employed in various ways, and the appended claims are intended to be construed to include such variations. Various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings should be regarded as illustrative and not restrictive. For purposes of explanation, methods have been described in a particular order. It should be appreciated that in alternative embodiments, methods may be performed in an order different from that described.

[0169]

[0186] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.

[0170]

[0187] Where a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0171]

[0188] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection, and / or other suitable communication interface).

[0172]

[0189] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0173]

[0190] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0174]

[0191] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a nonvolatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio waves, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0175]

[0192] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0176]

[0193] Illustrative examples of the present disclosure include the following:

[0177]

[0194] Aspect 1: A method for processing video data, comprising: generating, by a first convolutional layer of an encoder sub-network of a neural network system, output values ​​associated with a luminance channel of a frame; generating, by a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame; generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; and generating encoded video data based on the combined representation of the frame.

[0178]

[0195] Aspect 2: The method of aspect 1, wherein the third convolutional layer includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

[0179]

[0196] Aspect 3: The method of any one of aspects 1 or 2, further comprising: using a first nonlinear layer of the encoder sub-network to process output values ​​associated with a luminance channel of the frame; and using a second nonlinear layer of the encoder sub-network to process output values ​​associated with at least one chrominance channel of the frame; and wherein the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer.

[0180]

[0197] Aspect 4: The method of aspect 3, wherein the combined representation of the frame is generated by a third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0181]

[0198] Aspect 5: The method of any one of aspects 1 to 4, further comprising quantizing the encoded video data.

[0182]

[0199] Aspect 6: The method of any one of aspects 1 to 5, further comprising entropy coding the encoded video data.

[0183]

[0200] Example 7: The method of any one of Examples 1 to 6, further comprising storing the encoded video data in a memory.

[0184]

[0201] Aspect 8: The method of any one of aspects 1 to 7, further comprising transmitting the encoded video data over a transmission medium to at least one device.

[0185]

[0202] Aspect 9: A method as described in any one of aspects 1 to 8, further comprising obtaining an encoded frame, generating, by a first convolutional layer of a decoder sub-network of the neural network system, a reconstructed output value associated with a luminance channel of the encoded frame, and generating, by a second convolutional layer of the decoder sub-network, a reconstructed output value associated with at least one chrominance channel of the encoded frame.

[0186]

[0203] Aspect 10: The method of aspect 9, further comprising separating a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network.

[0187]

[0204] Aspect 11: The method of aspect 10, wherein the third convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

[0188]

[0205] Embodiment 12: The method of any one of embodiments 1 to 11, wherein the frame comprises a video frame.

[0189]

[0206] Embodiment 13: The method of any one of embodiments 1 to 12, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel.

[0190]

[0207] Example 14: The method of any one of examples 1 to 13, wherein the frame has a luminance-chrominance (YUV) format.

[0191]

[0208] Aspect 15: An apparatus for processing video data. The apparatus comprises: a memory; and a processor coupled to the memory and configured to: use a first convolutional layer of an encoder sub-network of a neural network system to generate output values ​​associated with a luminance channel of a frame, use a second convolutional layer of the encoder sub-network to generate output values ​​associated with at least one chrominance channel of the frame, use a third convolutional layer of the encoder sub-network to generate a joint representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame, and generate encoded video data based on the joint representation of the frame.

[0192]

[0209] Example 16: The apparatus of example 15, wherein the third convolutional layer includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

[0193]

[0210] Aspect 17: The apparatus of any one of aspects 15 or 16, wherein the processor is configured to process output values ​​associated with a luminance channel of the frame using a first nonlinear layer of the encoder sub-network and to process output values ​​associated with at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network, and wherein the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer.

[0194]

[0211] Aspect 18: The apparatus of aspect 17, wherein the combined representation of the frame is generated by a third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0195]

[0212] Aspect 19: The apparatus of any one of aspects 15 to 18, wherein the processor is configured to quantize the encoded video data.

[0196]

[0213] Example 20: The apparatus of any one of examples 15 to 19, wherein the processor is configured to entropy code the encoded video data.

[0197]

[0214] Aspect 21: The apparatus of any one of aspects 15 to 20, wherein the processor is configured to store the encoded video data in the memory.

[0198]

[0215] Aspect 22: The apparatus of any one of aspects 15 to 21, wherein the processor is configured to transmit the encoded video data over a transmission medium to at least one device.

[0199]

[0216] Aspect 23: The apparatus of any one of aspects 15 to 22, wherein the processor is configured to obtain an encoded frame, use a first convolutional layer of a decoder sub-network of the neural network system to generate a reconstructed output value associated with a luminance channel of the encoded frame, and use a second convolutional layer of the decoder sub-network to generate a reconstructed output value associated with at least one chrominance channel of the encoded frame.

[0200]

[0217] Aspect 24: A processor, Separating the luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network. 24. The apparatus of embodiment 23, configured to:

[0201]

[0218] Example 25: The apparatus of example 24, wherein the third convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer including one or more 1x1 convolutional filters.

[0202]

[0219] Embodiment 26: The apparatus of any one of embodiments 15 to 25, wherein the frame comprises a video frame.

[0203]

[0220] Embodiment 27: The apparatus of any one of embodiments 15 to 26, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel.

[0204]

[0221] Embodiment 28: The apparatus of any one of embodiments 15 to 27, wherein the frame has a luminance-chrominance (YUV) format.

[0205]

[0222] Embodiment 29: The apparatus of any one of embodiments 15 to 28, wherein the processor includes a neural processing unit (NPU).

[0206]

[0223] Embodiment 30: The apparatus of any one of embodiments 15 to 29, wherein the apparatus comprises a mobile device.

[0207]

[0224] Aspect 31: An apparatus described in any one of aspects 15 to 30, wherein the apparatus comprises an extended reality device.

[0208]

[0225] Embodiment 32: The device of any one of embodiments 15 to 31, further comprising a display.

[0209]

[0226] Embodiment 33: The apparatus of any one of embodiments 15 to 29, wherein the apparatus comprises a television.

[0210]

[0227] Embodiment 34: An apparatus described in any one of embodiments 15 to 33, wherein the apparatus comprises a camera configured to capture one or more video frames.

[0211]

[0228] Aspect 35: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations recited in aspects 1-14.

[0212]

[0229] Embodiment 36: An apparatus comprising means for performing any of the operations recited in embodiments 1 to 14.

[0213]

[0230] Aspect 37: A method of processing video data, comprising: obtaining an encoded frame; separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame; generating, by a second convolutional layer of the decoder sub-network of a neural network system, a reconstructed output value associated with the luminance channel of the encoded frame; and generating, by a third convolutional layer of the decoder sub-network, a reconstructed output value associated with the at least one chrominance channel of the encoded frame. generating an output frame including a reconstructed output value associated with a luminance channel and a reconstructed output value associated with at least one chrominance channel.

[0214]

[0231] Aspect 38: The method of aspect 37, wherein the first convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

[0215]

[0232] Aspect 39: A method described in any one of aspects 37 or 38, further comprising: processing values ​​associated with a luminance channel of an encoded frame using a first nonlinear layer of a decoder sub-network, wherein a reconstructed output value associated with the luminance channel is generated based on the output of the first nonlinear layer; and processing values ​​associated with at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder sub-network, wherein a reconstructed output value associated with at least one chrominance channel is generated based on the output of the second nonlinear layer.

[0216]

[0233] Example 40: The method of any one of examples 37 to 39, further comprising dequantizing samples of the encoded frame.

[0217]

[0234] Example 41: The method of any one of examples 37 to 40, further comprising entropy decoding samples of the encoded frame.

[0218]

[0235] Embodiment 42: The method of any one of embodiments 37 to 41, further comprising storing the output frame in a memory.

[0219]

[0236] Embodiment 43: The method of any one of embodiments 37 to 42, further comprising displaying the output frame.

[0220]

[0237] Aspect 44: A method described in any one of aspects 37 to 43, further comprising: generating, by a first convolutional layer of an encoder subnetwork of the neural network system, output values ​​associated with a luminance channel of the frame; generating, by a second convolutional layer of the encoder subnetwork, output values ​​associated with at least one chrominance channel of the frame; generating, by a third convolutional layer of the encoder subnetwork, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; and generating an encoded frame based on the combined representation of the frame.

[0221]

[0238] Aspect 45: The method of aspect 44, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

[0222]

[0239] Aspect 46: A method described in any one of aspects 44 or 45, further comprising using a first nonlinear layer of the encoder sub-network to process output values ​​associated with a luminance channel of the frame, and using a second nonlinear layer of the encoder sub-network to process output values ​​associated with at least one chrominance channel of the frame, wherein the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer.

[0223]

[0240] Aspect 47: The method described in aspect 46, wherein the combined representation of the frame is generated by a third convolutional layer of the encoder sub-network using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0224]

[0241] Embodiment 48: The method of any one of embodiments 37 to 47, wherein the encoded frame comprises an encoded video frame.

[0225]

[0242] Embodiment 49: The method of any one of embodiments 37 to 48, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel.

[0226]

[0243] Embodiment 50: The method of any one of embodiments 37 to 49, wherein the encoded frame has a luminance-chrominance (YUV) format.

[0227]

[0244] Aspect 49: An apparatus for processing video data. The apparatus comprises: a memory; and a processor coupled to the memory and configured to obtain an encoded frame, separate a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a first convolutional layer of a decoder sub-network, generate reconstructed output values ​​associated with the luminance channel of the encoded frame using a second convolutional layer of the decoder sub-network of a neural network system, generate reconstructed output values ​​associated with the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network, and generate an output frame including the reconstructed output values ​​associated with the luminance channel and the at least one chrominance channel.

[0228]

[0245] Example 50: The apparatus of example 49, wherein the first convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer including one or more 1x1 convolutional filters.

[0229]

[0246] Aspect 51: The apparatus described in any one of aspects 49 or 50, wherein the processor is configured to process values ​​associated with a luminance channel of an encoded frame using a first nonlinear layer of a decoder sub-network, wherein a reconstructed output value associated with the luminance channel is generated based on the output of the first nonlinear layer, and to process values ​​associated with at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder sub-network, wherein a reconstructed output value associated with the at least one chrominance channel is generated based on the output of the second nonlinear layer.

[0230]

[0247] Embodiment 52: The apparatus of any one of embodiments 49 to 51, wherein the processor is configured to inverse quantize samples of the encoded frame.

[0231]

[0248] Aspect 53: The apparatus of any one of aspects 49 to 52, wherein the processor is configured to entropy decode samples of the encoded frame.

[0232]

[0249] Aspect 54: The apparatus of any one of aspects 49 to 53, wherein the processor is configured to store the output frame in a memory.

[0233]

[0250] Embodiment 55: The apparatus of any one of embodiments 49 to 54, wherein the processor is configured to display the output frame.

[0234]

[0251] Aspect 56: The apparatus of any one of aspects 49 to 55, wherein the processor is configured to generate, by a first convolutional layer of an encoder subnetwork of the neural network system, output values ​​associated with a luminance channel of the frame, by a second convolutional layer of the encoder subnetwork, generate output values ​​associated with at least one chrominance channel of the frame, by a third convolutional layer of the encoder subnetwork, generate a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame, and generate an encoded frame based on the combined representation of the frame.

[0235]

[0252] Aspect 57: The apparatus described in aspect 56, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, and the 1x1 convolutional layer includes one or more 1x1 convolutional filters.

[0236]

[0253] Aspect 58: The apparatus described in any one of aspects 44 or 57, wherein the processor is configured to process output values ​​associated with a luminance channel of the frame using a first nonlinear layer of the encoder subnetwork and to process output values ​​associated with at least one chrominance channel of the frame using a second nonlinear layer of the encoder subnetwork, and wherein the combined representation is generated based on the output of the first nonlinear layer and the output of the second nonlinear layer.

[0237]

[0254] Aspect 59: The apparatus described in aspect 58, wherein the combined representation of the frame is generated by a third convolutional layer of the encoder sub-network using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs.

[0238]

[0255] Embodiment 60: The apparatus of any one of embodiments 49 to 59, wherein the encoded frame comprises an encoded video frame.

[0239]

[0256] Embodiment 61: The device described in any one of embodiments 49 to 60, wherein at least one chrominance channel includes a chrominance blue channel and a chrominance red channel.

[0240]

[0257] Aspect 62: The apparatus described in any one of aspects 49 to 61, wherein the encoded frame has a luminance-chrominance (YUV) format.

[0241]

[0258] Embodiment 63: The device of any one of embodiments 49 to 62, wherein the processor includes a neural processing unit (NPU).

[0242]

[0259] Embodiment 64: The apparatus of any one of embodiments 49 to 63, wherein the apparatus comprises a mobile device.

[0243]

[0260] Aspect 65: An apparatus described in any one of aspects 49 to 64, wherein the apparatus comprises an extended reality device.

[0244]

[0261] Embodiment 66: The device of any one of embodiments 49 to 65, further comprising a display.

[0245]

[0262] Embodiment 67: The apparatus of any one of embodiments 49 to 63, wherein the apparatus comprises a television.

[0246]

[0263] Aspect 68: An apparatus described in any one of aspects 49 to 67, wherein the apparatus comprises a camera configured to capture one or more video frames.

[0247]

[0264] Aspect 69: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations recited in aspects 37 to 48.

[0248]

[0265] Embodiment 70: An apparatus comprising means for performing any of the operations recited in embodiments 37 to 48. The inventions described in the claims of the present application as originally filed are set forth below. [C1] 1. A method for processing video data, comprising: generating, by a first convolutional layer of an encoder sub-network of a neural network system, an output value associated with a luminance channel of a frame; and generating, by a second convolutional layer of the encoder sub-network, an output value associated with at least one chrominance channel of the frame. generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; generating encoded video data based on the combined representation of the frame; A method comprising: [C2] 3. The method of claim 1, wherein the third convolutional layer comprises a 1×1 convolutional layer, the 1×1 convolutional layer comprising one or more 1×1 convolutional filters. [C3] processing the output values ​​associated with the luminance channel of the frame using a first nonlinear layer of the encoder sub-network; processing the output values ​​associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network; Furthermore, the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The method described in C1. [C4] 4. The method of claim 3, wherein the combined representation of the frame is generated by the third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs. [C5] quantizing the encoded video data; The method of C1, further comprising: [C6] entropy coding the encoded video data; The method of C1, further comprising: [C7] storing the encoded video data in a memory; The method of C1, further comprising: [C8] transmitting the encoded video data over a transmission medium to at least one device; The method of C1, further comprising: [C9] obtaining an encoded frame; generating, by a first convolutional layer of a decoder sub-network of the neural network system, a reconstructed output value associated with a luminance channel of the encoded frame; The method of C1, further comprising: generating, by a second convolutional layer of the decoder sub-network, a reconstructed output value associated with at least one chrominance channel of the coded frame. [C10] separating the luminance channel of the encoded frame from the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network; The method of C9, further comprising: [C11] The method of C10, wherein the third convolutional layer of the decoder sub-network includes a 1×1 convolutional layer, the 1×1 convolutional layer including one or more 1×1 convolutional filters. [C12] The method of C1, wherein the frames include video frames. [C13] The method of C1, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel. [C14] The method of C1, wherein the frames have a luminance-chrominance (YUV) format. [C15] 1. An apparatus for processing video data, comprising: Memory and a processor coupled to the memory, the processor comprising: generating an output value associated with a luminance channel of the frame using a first convolutional layer of an encoder sub-network of the neural network system; generating, using a second convolutional layer of the encoder sub-network, output values ​​associated with at least one chrominance channel of the frame; and generating, using a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame. generating encoded video data based on the combined representation of the frame; An apparatus configured to: [C16] The apparatus of C15, wherein the third convolutional layer includes a 1×1 convolutional layer, the 1×1 convolutional layer including one or more 1×1 convolution filters. [C17] The processor: processing the output values ​​associated with the luminance channel of the frame using a first nonlinear layer of the encoder sub-network; processing the output values ​​associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network; configured to: the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The apparatus described in C15. [C18] The apparatus of C17, wherein the combined representation of the frame is generated by the third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs. [C19] The processor: The apparatus of C15, configured to quantize the encoded video data. [C20] The processor: The apparatus of C15, configured to entropy code the encoded video data. [C21] The processor: The apparatus of C15, configured to store the encoded video data in a memory. [C22] The processor: The apparatus of C15, configured to transmit the encoded video data over a transmission medium to at least one device. [C23] The processor: obtaining an encoded frame; generating a reconstructed output value associated with a luminance channel of the encoded frame using a first convolutional layer of a decoder sub-network of the neural network system; generating a reconstructed output value associated with at least one chrominance channel of the encoded frame using a second convolutional layer of the decoder sub-network; and The apparatus of C15, configured to perform the following. [C24] The processor: The apparatus of C23, configured to separate the luminance channel of the encoded frame from the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network. [C25] The apparatus of C24, wherein the third convolutional layer of the decoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer including one or more 1x1 convolution filters. [C26] The apparatus of C15, wherein the frames include video frames. [C27] The apparatus of C15, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel. [C28] The apparatus of C15, wherein the frame has a luminance-chrominance (YUV) format. [C29] The apparatus of C15, wherein the processor includes a neural processing unit (NPU). [C30] The apparatus of C15, wherein the apparatus comprises a mobile device. [C31] The apparatus of C15, further comprising at least one of a display and a camera configured to capture one or more frames. [C32] 1. A method for processing video data, comprising: obtaining an encoded frame; separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame; generating, by a second convolutional layer of the decoder sub-network of a neural network system, a reconstructed output value associated with the luminance channel of the encoded frame; generating, by a third convolutional layer of the decoder sub-network, a reconstructed output value associated with the at least one chrominance channel of the encoded frame; generating an output frame including the reconstructed output values ​​associated with the luminance channel and the reconstructed output values ​​associated with the at least one chrominance channel; A method comprising: [C33] The method of C32, wherein the first convolutional layer of the decoder sub-network includes a 1×1 convolutional layer, the 1×1 convolutional layer including one or more 1×1 convolution filters. [C34] processing values ​​associated with the luminance channel of the encoded frame using a first non-linear layer of the decoder sub-network, wherein the reconstructed output values ​​associated with the luminance channel are generated based on an output of the first non-linear layer. processing values ​​associated with the at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder sub-network, wherein the reconstructed output values ​​associated with the at least one chrominance channel are generated based on an output of the second nonlinear layer. The method of C32, further comprising: [C35] dequantizing the samples of the encoded frame; The method of C32, further comprising: [C36] entropy decoding the samples of the encoded frame; The method of C32, further comprising: [C37] storing the output frame in a memory; The method of C32, further comprising: [C38] displaying the output frame; The method of C32, further comprising: [C39] generating, by a first convolutional layer of an encoder sub-network of the neural network system, an output value associated with a luminance channel of the frame; generating, by a second convolutional layer of the encoder sub-network, an output value associated with at least one chrominance channel of the frame; generating, by a third convolutional layer of the encoder sub-network, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; generating the encoded frame based on the combined representation of the frame; The method of C32, further comprising: [C40] 30. The method of claim 39, wherein the third convolutional layer of the encoder sub-network comprises a 1×1 convolutional layer, the 1×1 convolutional layer comprising one or more 1×1 convolution filters. [C41] processing the output values ​​associated with the luminance channel of the frame using a first nonlinear layer of the encoder sub-network; processing the output values ​​associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network; Furthermore, the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The method described in C39. [C42] The method of C41, wherein the combined representation of the frame is generated by the third convolutional layer of the encoder sub-network using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs. [C43] The method of C32, wherein the coded frames comprise coded video frames. [C44] The method of C32, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel. [C45] The method of C32, wherein the coded frames have a luminance-chrominance (YUV) format. [C46] 1. An apparatus for processing video data, comprising: Memory and a processor coupled to the memory, the processor comprising: obtaining an encoded frame; separating a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a first convolutional layer of a decoder sub-network; generating a reconstructed output value associated with the luminance channel of the encoded frame using a second convolutional layer of the decoder sub-network of a neural network system; generating a reconstructed output value associated with the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network; and generating an output frame including the reconstructed output values ​​associated with the luminance channel and the reconstructed output values ​​associated with the at least one chrominance channel; An apparatus configured to: [C47] The apparatus of C46, ​​wherein the first convolutional layer of the decoder sub-network includes a 1×1 convolutional layer, the 1×1 convolutional layer including one or more 1×1 convolution filters. [C48] The processor: processing values ​​associated with the luminance channel of the encoded frame using a first non-linear layer of the decoder sub-network, wherein the reconstructed output values ​​associated with the luminance channel are generated based on an output of the first non-linear layer. processing values ​​associated with the at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder sub-network, wherein the reconstructed output values ​​associated with the at least one chrominance channel are generated based on an output of the second nonlinear layer. The apparatus of C46, ​​configured to perform the following: [C49] The processor: The apparatus of C46, ​​configured to inverse quantize samples of the encoded frame. [C50] The processor: The apparatus of C46, ​​configured to entropy decode samples of the encoded frame. [C51] The processor: The apparatus of C46, ​​configured to store the output frame in a memory. [C52] The processor: The apparatus of C46 configured to display the output frame. [C53] The processor: generating, by a first convolutional layer of an encoder sub-network of the neural network system, an output value associated with a luminance channel of the frame; generating, by a second convolutional layer of the encoder sub-network, an output value associated with at least one chrominance channel of the frame; generating, by a third convolutional layer of the encoder sub-network, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; generating the encoded frame based on the combined representation of the frame; The apparatus of C46, ​​configured to perform the following: [C54] The apparatus of C53, wherein the third convolutional layer of the encoder sub-network includes a 1x1 convolutional layer, the 1x1 convolutional layer including one or more 1x1 convolution filters. [C55] The processor: processing the output values ​​associated with the luminance channel of the frame using a first nonlinear layer of the encoder sub-network; processing the output values ​​associated with the at least one chrominance channel of the frame using a second nonlinear layer of the encoder sub-network; configured to: the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer. The device described in C53. [C56] The apparatus of C55, wherein the combined representation of the frame is generated by the third convolutional layer of the encoder sub-network using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs. [C57] The apparatus of C46, ​​wherein the coded frames comprise coded video frames. [C58] The apparatus of C57, wherein the at least one chrominance channel includes a chrominance blue channel and a chrominance red channel. [C59] The apparatus of C46, ​​wherein the coded frames have a luminance-chrominance (YUV) format. [C60] The apparatus of C46, ​​further comprising at least one of a display and a camera configured to capture one or more video frames.

Claims

1. 1. A method for processing video data, comprising: generating, by a first convolutional layer of an encoder sub-network of the neural network system, an output value associated with a luminance channel of the frame; generating, by a second convolutional layer of the encoder sub-network, an output value associated with at least one chrominance channel of the frame; generating, by a third convolutional layer, a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame; generating encoded video data based on the combined representation of the frame; Equipped with the third convolutional layer includes a 1×1 convolutional layer, the 1×1 convolutional layer including one or more 1×1 convolutional filters; method.

2. processing the output values ​​associated with the luminance channel of the frame using a first nonlinear layer of the encoder sub-network; processing the output values ​​associated with the at least one chrominance channel of the frame using a second non-linear layer of the encoder sub-network; Furthermore, the combined representation is generated based on an output of the first nonlinear layer and an output of the second nonlinear layer; the combined representation of the frame is generated by the third convolutional layer using the output of the first nonlinear layer and the output of the second nonlinear layer as inputs. The method of claim 1.

3. quantizing the encoded video data; entropy coding the encoded video data; storing the encoded video data in a memory; transmitting the encoded video data over a transmission medium to at least one device; The method of claim 1 , further comprising one or more of:

4. obtaining an encoded frame; generating, by a first convolutional layer of a decoder sub-network of the neural network system, a reconstructed output value associated with a luminance channel of the encoded frame; generating, by a second convolutional layer of the decoder sub-network, a reconstructed output value associated with at least one chrominance channel of the encoded frame; The method of claim 1 further comprising:

5. separating the luminance channel of the encoded frame from the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network; 5. The method of claim 4, further comprising:

6. 2. The method of claim 1, wherein the frame comprises a video frame, the frame comprises a luminance-chrominance (YUV) format, and / or the at least one chrominance channel comprises a chrominance blue channel and a chrominance red channel.

7. 1. An apparatus for processing video data, comprising: Memory and a processor coupled to the memory, the processor comprising: generating an output value associated with a luminance channel of the frame using a first convolutional layer of an encoder sub-network of the neural network system; generating an output value associated with at least one chrominance channel of the frame using a second convolutional layer of the encoder sub-network; generating a combined representation of the frame based on the output values ​​associated with the luminance channel of the frame and the output values ​​associated with the at least one chrominance channel of the frame using a third convolutional layer; generating encoded video data based on the combined representation of the frame; configured to: the third convolutional layer includes a 1×1 convolutional layer, the 1×1 convolutional layer including one or more 1×1 convolutional filters; Device.

8. The apparatus of claim 7 , wherein the processor is further configured to perform the method of any one of claims 2 to 6 .

9. 10. The apparatus of claim 7, wherein the processor includes a neural processing unit (NPU), the apparatus comprises a mobile device, and / or the apparatus further comprises at least one of a display and a camera configured to capture one or more frames.

10. 1. A method for processing video data, comprising: obtaining an encoded frame; separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame; generating, by a second convolutional layer of the decoder sub-network of a neural network system, a reconstructed output value associated with the luminance channel of the encoded frame; generating, by a third convolutional layer of the decoder sub-network, a reconstructed output value associated with the at least one chrominance channel of the encoded frame; generating an output frame including the reconstructed output values ​​associated with the luminance channel and the reconstructed output values ​​associated with the at least one chrominance channel; Equipped with the first convolutional layer of the decoder sub-network comprises a 1×1 convolutional layer, the 1×1 convolutional layer comprising one or more 1×1 convolutional filters; method.

11. processing values ​​associated with the luminance channel of the coded frame using a first non-linear layer of the decoder sub-network, wherein the reconstructed output values ​​associated with the luminance channel are generated based on an output of the first non-linear layer. processing values ​​associated with the at least one chrominance channel of the encoded frame using a second nonlinear layer of the decoder sub-network, wherein the reconstructed output value associated with the at least one chrominance channel is generated based on an output of the second nonlinear layer. The method of claim 10 further comprising:

12. 1. An apparatus for processing video data, comprising: Memory and a processor coupled to the memory, the processor comprising: obtaining an encoded frame; separating a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame using a first convolutional layer of a decoder sub-network; generating a reconstructed output value associated with the luminance channel of the encoded frame using a second convolutional layer of the decoder sub-network of a neural network system; generating a reconstructed output value associated with the at least one chrominance channel of the encoded frame using a third convolutional layer of the decoder sub-network; and generating an output frame including the reconstructed output values ​​associated with the luminance channel and the reconstructed output values ​​associated with the at least one chrominance channel; configured to: the first convolutional layer of the decoder sub-network comprises a 1×1 convolutional layer, the 1×1 convolutional layer comprising one or more 1×1 convolutional filters; Device.

13. 12. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations recited in aspects 1-6 or 10-11.

Images