Bitrate estimation for video coding using machine learning enhancements

Abstract

[0009] Techniques for processing video data are described herein. For example, the techniques can include encoding one or more frames of video data using a video encoder, where the video encoder includes at least a quantization process, determining an actual bitrate of the encoded one or more frames, predicting an estimated bitrate using an encoder proxy, where the encoder proxy includes a statistical model for estimating the bitrate of the encoded one or more frames, determining a gradient of the estimated bitrate using the encoder proxy, and training the encoder proxy to predict the estimated bitrate based on the actual bitrate, the estimated bitrate, and the gradient.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE

[0001] This disclosure relates generally to video coding (e.g., encoding and / or decoding video data). For example, aspects of this disclosure relate to systems and techniques for performing bitrate estimation to enhance video coding using machine learning. [Background technology]

[0002]

[0002] Many devices and systems enable video data to be processed and output for consumption. Digital video data comprises a large amount of data to meet the demands of consumers and video providers. For example, video data consumers desire high quality video, including high fidelity, resolution, frame rate, etc. As a result, the large amount of video data required to meet these demands places a strain on communication networks and devices that process and store the video data.

[0003]

[0003] Video coding techniques can be used to compress video data. The goal of video coding is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation to video quality. As ever-evolving video services become available, encoding techniques with better coding efficiency are needed. Summary of the Invention

[0004]

[0004] The following presents a simplified summary of one or more aspects disclosed herein. As such, the following summary should not be considered an extensive overview of all contemplated aspects, nor should it be considered as identifying key or critical elements of all contemplated aspects or as delimiting the scope of any particular aspect. As such, the following summary presents certain concepts of one or more aspects of the mechanisms disclosed herein in a simplified form, prior to the Detailed Description presented below.

[0005]

[0005] Systems and techniques for coding (e.g., encoding and / or decoding) image and / or video content are described. In one illustrative example, a method for processing video data is provided. The method includes: encoding one or more frames of video data using a video encoder, where the video encoder includes at least a quantization process; determining an actual bit rate for the encoded one or more frames; predicting an estimated bit rate using an encoder proxy, where the encoder proxy includes a statistical model for estimating the bit rate for the encoded one or more frames; determining a gradient of the estimated bit rate using the encoder proxy; and training the encoder proxy to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0006] In another example, an apparatus for processing video data is provided, the apparatus including: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to: encode one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determine an actual bitrate for the encoded one or more frames; predict an estimated bitrate using an encoder proxy, the encoder proxy including a statistical model for estimating a bitrate for the encoded one or more frames; determine a gradient of the estimated bitrate using the encoder proxy; and train the encoder proxy to predict the estimated bitrate based on the actual bitrate, the estimated bitrate, and the gradient.

[0007]

[0007] In another example, a non-transitory computer-readable storage medium is provided that stores instructions that, when executed by one or more processors, cause the one or more processors to: receive one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predict an estimated bitrate for the one or more frames after encoding by the video encoder using an encoder proxy, the encoder proxy including a statistical model for estimating an estimated bitrate, the statistical model being trained based on a gradient of the estimated bitrate; adjust one or more quality parameters based on the predicted estimated bitrate; and encode the one or more frames of video data using the video encoder, the video encoder including at least a quantization process.

[0008] In another example, an apparatus is provided that includes: means for receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; means for predicting an estimated bit rate for the one or more frames after being encoded by the video encoder using the encoder proxy, the encoder proxy including a statistical model for estimating the estimated bit rate, the statistical model being trained based on a gradient of the estimated bit rate; means for adjusting one or more quality parameters based on the predicted estimated bit rate; and means for encoding the one or more frames of video data using the video encoder, the video encoder including at least the quantization process.

[0009] In some aspects, the apparatus comprises a mobile device (e.g., a mobile phone or so-called “smartphone,” a tablet computer, or other type of mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television), a vehicle (or a vehicle's computing device), or other device. In some aspects, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include one camera (e.g., an RGB camera) or multiple cameras for capturing one or more videos including one or more images and / or video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to transmit one or more video frames and / or syntax data to the at least one device over a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0010]

[0010] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0011]

[0011] The foregoing, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief explanation of the drawings]

[0012]

[0012] With reference to the following figures, exemplary embodiments of the present application are described in detail below. [Figure 1]

[0013] FIG. 1 illustrates an exemplary implementation of a system-on-chip (SOC), according to some examples. [Figure 2]

[0014] 1 is a block diagram illustrating an encoding device and a decoding device, according to some examples. [Figure 3]

[0015] FIG. 1 illustrates an example of a system including a device operable to perform image and / or video coding (encoding and decoding) using a machine learning coding system, according to some examples. [Figure 4A]

[0016] FIG. 1 illustrates an example of a neural network according to some examples. [Figure 4B] FIG. 1 illustrates an example of a neural network, according to some examples. [Figure 4C] FIG. 1 illustrates an example of a neural network, according to some examples. [Figure 4D] FIG. 1 illustrates an example of a neural network according to some examples. [Figure 5]

[0017] FIG. 1 illustrates an example of a deep convolutional network, according to some examples. [Figure 6A]

[0018] FIG. 1 is a block diagram illustrating an example implementation of a video codec neural boosting system, according to some examples. [Figure 6B] FIG. 1 is a block diagram illustrating an example implementation of a video codec neural boosting system, according to some examples. [Figure 7]

[0019] FIG. 2 illustrates encoder elements of an exemplary video encoder, according to some examples. [Figure 8]

[0020] FIG. 2 is a block diagram illustrating elements of an exemplary differentiable encoder proxy, in accordance with some examples. [Figure 9]

[0021] 1 is a graph plotting an example adjustment function and a derivative of the adjustment function, in accordance with some examples. [Figure 10]

[0022] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 11] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 12] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 13] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 14] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 15] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 16] 10 is a histogram of exemplary ratios between frame bit rates, according to some examples. [Figure 17]

[0023] 1 is a flow diagram illustrating a technique for performing bit rate estimation according to an aspect of the disclosure. [Figure 18]

[0024] 1 is a flow diagram illustrating a technique for performing bit rate estimation according to an aspect of the disclosure. [Figure 19]

[0025] 1 is an exemplary computing device capable of implementing various techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0026] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to one skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and description are not intended to be limiting.

[0014]

[0027] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for practicing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0015]

[0028] Digital video data can include large amounts of data, especially as the demand for high-quality video data continues to grow. For example, consumers of video data typically desire increasingly higher quality video, having high fidelity, resolution, frame rates, etc. However, the large amounts of video data required to meet such demands can place a significant strain on communication networks and devices that process and store the video data.

[0016]

[0029] Various techniques may be used to code video data. Video coding may be performed according to a specific video coding standard or may be performed using one or more machine learning systems or algorithms. Exemplary video coding standards include Generic Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding (e.g., MPEG-5 Essential Video Coding (EVC) or other MPEG-based coding), and AOMedia Video 1 (AV1), among others. Video coding often uses prediction methods such as inter-prediction or intra-prediction that exploit redundancy present within a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. As demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0017]

[0030] Video coding devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame or intra-prediction), temporal prediction (e.g., inter-frame or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques to reduce or remove redundancy inherent in video sequences. A video encoder may partition each picture of an original video sequence into rectangular regions called video blocks or coding units (described in more detail below). These video blocks may be coded using particular prediction modes.

[0018]

[0031] A video block may be divided into one or more groups of smaller blocks in one or more ways. A block may include a coding tree block, a prediction block, a transform block, and / or other suitable block. In general, references to a "block" may refer to such a video block (e.g., a coding tree block, a coding block, a prediction block, a transform block, or other suitable block or sub-block, as would be understood by one of ordinary skill in the art) unless otherwise specified. Furthermore, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., a coding tree unit (CTU), a coding unit, a prediction unit (PU), a transform unit (TU), etc.). In some cases, a unit may refer to a coding logical unit that is coded in a bitstream, and a block may refer to a portion of a video frame buffer that a process targets.

[0019]

[0032] In inter-prediction modes, a video encoder may search for a block similar to a block being coded in a frame (or picture) located at another temporal location, called a reference frame or picture. The video encoder may limit this search to a certain spatial displacement from the block to be coded. A two-dimensional (2D) motion vector, including a horizontal displacement component and a vertical displacement component, may be used to locate the best match. In intra-prediction modes, the video encoder may use spatial prediction techniques to form a predicted block based on data from previously coded neighboring blocks in the same picture.

[0020]

[0033] The video encoder may determine a prediction error. For example, the prediction may be determined as the difference between pixel values ​​in the block being coded and the predicted block. The prediction error is sometimes referred to as a residual. The video encoder may also apply a transform to the prediction error using transform coding (e.g., using a form of discrete cosine transform (DCT), a form of discrete sine transform (DST), or other suitable transform) to generate transform coefficients. After the transform, the video encoder may quantize the transform coefficients. The quantized transform coefficients and motion vectors may be represented using syntax elements and, together with control information, form a coded representation of the video sequence. In some instances, the video encoder may entropy code the syntax elements, thereby further reducing the number of bits required for their representation.

[0021]

[0034] A video decoder may use the syntax elements and control information described above to construct prediction data (e.g., a prediction block) for decoding a current frame. For example, the video decoder may add the prediction block and a compressed prediction error. The video decoder may determine the compressed prediction error by weighting the transform basis functions using quantized coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0022]

[0035] As described in more detail below, this specification describes systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to as "systems and techniques") for performing bitrate estimation to enhance video coding using machine learning. In some devices, video codecs may be implemented using custom hardware such as application-specific integrated circuits (ASICs). While the aid of custom hardware allows for high efficiency in terms of processing speed and power usage, it significantly reduces flexibility as additional features require designing and deploying new hardware, which may be slow and expensive.

[0023]

[0036] Techniques have been proposed to improve performance by adaptively modifying video before encoding and after decoding, while leveraging the ubiquity of codec hardware. Recent proposals have demonstrated the benefits of using machine learning and neural networks for this purpose, as they can use advanced algorithms and extensive training to better identify how video compression can be improved. This approach is sometimes referred to as standard video codec neural boosting.

[0024]

[0037] A fundamental problem with many such techniques is that the design and optimization of neural-based solutions is much more effective when derivatives of bitrate and distortion performance measures can be used. However, these derivatives may not be available from complex standard video codecs, and therefore they cannot be effectively integrated in end-to-end system designs, resulting in suboptimal performance.

[0025]

[0038] The present disclosure addresses at least this problem using a new type of codec proxy. For example, a codec proxy according to the systems and techniques described herein can efficiently and accurately estimate codec performance, along with the partial derivatives (e.g., all partial derivatives) required for optimization. The codec proxy can enable integration with backpropagation techniques (e.g., gradient backpropagation) used by neural network training tools. Experimental results described below demonstrate the accuracy of the estimation using a comparison with the bitrate of the HEVC / H.265 video standard.

[0026]

[0039] Various aspects of the present disclosure are described with reference to the figures.

[0027]

[0040] 1 illustrates an example implementation of a system-on-chip (SOC) 100 that may include a central processing unit (CPU) 102 or multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, among other information, may be stored in and / or distributed across memory blocks associated with a neural processing unit (NPU) 108, memory blocks associated with the CPU 102, memory blocks associated with a graphics processing unit (GPU) 104, memory blocks associated with a digital signal processor (DSP) 106, and memory blocks 118. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from memory blocks 118.

[0028]

[0041] SOC 100 may also include a connectivity block 110 that may include a GPU 104, a DSP 106, fifth generation (5G) connectivity, fourth generation long-term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., as well as additional processing blocks adapted to specific functions, such as a multimedia processor 112 that may detect and recognize gestures. In one implementation, the NPU is implemented within CPU 102, DSP 106, and / or GPU 104. SOC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, and / or a navigation module 120 that may include a global positioning system.

[0029]

[0042] The SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, instructions loaded into the CPU 102 may include code for searching a stored multiplication result in a lookup table (LUT) corresponding to the multiplication product of the input value and the filter weight. The instructions loaded into the CPU 102 may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit for the multiplication product is detected. Additionally, the instructions loaded into the CPU 102 may include code for storing the calculated multiplication product of the input value and the filter weight when a lookup table miss for the multiplication product is detected.

[0030]

[0043] SOC 100 and / or its components may be configured to perform video compression and / or decompression (also called video encoding and / or decoding, collectively referred to as video coding) using standards-based video coding and / or using machine learning techniques. Examples of standards-based and machine learning-based video coding systems are described with respect to FIGS. 2 and 3.

[0031]

[0044] 2 is a block diagram illustrating an example of a system 200 including an encoding device 204 and a decoding device 212, which can encode and decode video data, respectively, according to examples described herein. In some examples, the encoding device 204 and / or the decoding device 212 can include the SOC 100 of FIG. 1. The encoding device 204 can be part of a source device, and the decoding device 212 can be part of a receiving device (also referred to as a client device). In some examples, the source device can also include a decoding device similar to the decoding device 212. In some examples, the receiving device can also include an encoding device similar to the encoding device 204. The source device and / or receiving device may include an electronic device such as a mobile or landline telephone handset (e.g., a smartphone, a cellular telephone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, an Internet Protocol (IP) camera, a server device in a server system including one or more server devices (e.g., a video streaming server system or other suitable server system), a head-mounted display (HMD), a head-up display (HUD), smart glasses (e.g., virtual reality (VR) glasses, augmented reality (AR) glasses, or other smart glasses), or any other suitable electronic device.

[0032]

[0045] The system components 200 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include SOC 100 and / or one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0033]

[0046] Although system 200 is shown as including certain components, one skilled in the art will understand that system 200 can include more or fewer components than those shown in Figure 2. For example, system 200, in some examples, can also include one or more memory devices other than storage 208 and storage 218 (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, NPUs, and / or other processing devices) in communication with and / or electrically connected to the one or more memory devices, one or more wireless interfaces for implementing wireless communications (e.g., including one or more transceivers and a baseband processor for each wireless interface), one or more wired interfaces for implementing communications via one or more wired connections (e.g., a serial interface such as a universal serial bus (USB) input, a lightning connector, and / or other wired interfaces), and / or other components not shown in Figure 2.

[0034]

[0047] The coding techniques described herein are applicable to video coding in various multimedia applications, including streaming video transmission (e.g., over the Internet), television broadcasting or transmission, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. In some examples, system 200 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0035]

[0048] In some examples, encoding device 204 (or encoder) may be used to encode video data using a video coding standard or protocol to generate an encoded video bitstream. Examples of video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) including its Scalable Video Coding (SVC) extension and Multiview Video Coding (MVC) extension, High Efficiency Video Coding (HEVC) or ITU-T H.265, Versatile Video Coding (VVC) or ITU-T H.266, and / or other video coding standards. One or more of the video coding standards have extensions related to other aspects of video coding. For example, various extensions to HEVC address multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multiview extensions (MV-HEVC) and scalable extensions (SHVC).

[0036]

[0049] Many embodiments described herein may be implemented using video codecs such as VVC, HEVC, AVC, and / or extensions thereof. However, the techniques and systems described herein may also be applicable to other coding standards, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, extensions thereof, or other suitable coding standards that may already be available or that are not yet available or developed, such as the machine learning-based video coding described below. Thus, those skilled in the art will understand that while the techniques and systems described herein may be described with respect to a particular video coding standard, this description should not be construed as applying exclusively to that particular standard.

[0037]

[0050] 2, a video source 202 may provide video data to an encoding device 204. The video source 202 may be part of a source device or part of a device other than the source device. The video source 202 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0038]

[0051] The video data from the video source 202 may include one or more input pictures. A picture is sometimes referred to as a "frame." A picture or frame is a still image that is sometimes part of a video. In some examples, the data from the video source 202 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chrominance samples, and SCr is a two-dimensional array of Cr chrominance samples. The chrominance samples are sometimes referred to herein as "chroma" samples. In other cases, a picture may be monochrome and include only an array of luma samples.

[0039]

[0052] The encoder engine 206 (or encoder) of the encoding device 204 encodes video data to generate a coded video bitstream. In some examples, a coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more coded video sequences. According to HEVC, a coded video sequence (CVS) includes a series of AUs starting with an access unit (AU) in a base layer that has a random access point picture with some property (e.g., a RASL flag (e.g., NoRaslOutputFlag) equal to 1) up to, but not including, the next AU in the base layer that has a random access point picture with some property. An AU includes one or more coded pictures and control information corresponding to coded pictures that share the same output time. At the bitstream level, coded slices of a picture are encapsulated in data units called network abstraction layer (NAL) units. For example, an HEVC video bitstream may include one or more CVSs that include NAL units. Each NAL unit has an NAL unit header. Syntax elements in the NAL unit header take designated bits and are therefore recognizable by all kinds of systems and transport layers, such as transport streams, real-time transport protocols (RTP), file formats, among others.

[0040]

[0053] Two classes of NAL units exist in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units contain coded picture data that form a coded video bitstream. For example, a sequence of bits that form a coded video bitstream resides in a VCL NAL unit. A VCL NAL unit may contain one slice or slice segment (described below) of coded picture data, while a non-VCL NAL unit contains control information about one or more coded pictures. In some cases, NAL units are referred to as packets. An HEVC AU contains VCL NAL units that contain coded picture data and non-VCL NAL units that correspond to the coded picture data (if any). A non-VCL NAL unit may contain, in addition to other information, a parameter set that has high-level information about the coded video bitstream. For example, the parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of the bitstream may reference a single active PPS, SPS, and / or VPS to allow the decoding device 212 to access information that can be used to decode the slice or other portion of the bitstream.

[0041]

[0054] An NAL unit may include a sequence of bits that form a coded representation of video data (e.g., an encoded video bitstream, a CVS of a bitstream, etc.), such as a coded representation of a picture in a video. The encoder engine 206 generates coded representations of pictures by partitioning each picture into multiple slices. A slice is independent of other slices such that information in the slice is coded without dependency on data from other slices in the same picture. A slice includes one or more slice segments, including an independent slice segment and, if present, one or more dependent slice segments that depend on a previous slice segment.

[0042]

[0055] In HEVC, a slice is partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, together with the syntax for the samples, are called a coding tree unit (CTU). A CTU is sometimes called a "treeblock" or "largest coding unit" (LCU). A CTU is the basic processing unit for HEVC encoding. A CTU can be split into multiple coding units (CUs) of various sizes. A CU contains a luma sample array and a chroma sample array, called coding blocks (CBs).

[0043]

[0056] The luma CB and the chroma CB may be further split into prediction blocks (PBs). A PB is a block of luma or chroma component samples that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). A luma PB and one or more chroma PBs, together with associated syntax, form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) is signaled in the bitstream for each PU and is used for inter prediction of the luma PB and one or more chroma PBs. The motion parameters are sometimes referred to as motion information. A CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of color component samples to which a residual transform (e.g., possibly the same two-dimensional transform) is applied to code the prediction residual signal. A transform unit (TU) represents a TB of luma samples and chroma samples, as well as corresponding syntax elements. Transform coding is described in more detail below.

[0044]

[0057] The size of a CU corresponds to the size of a coding mode and may be square in shape. For example, the size of a CU may be 8x8 samples, 16x16 samples, 32x32 samples, 64x64 samples, or any other suitable size up to the size of the corresponding CTU. The phrase "NxN" is used herein to refer to the pixel dimensions of a video block in terms of vertical and horizontal dimensions (e.g., 8 pixels x 8 pixels). The pixels in a block may be arranged in rows and columns. In some embodiments, a block may not have the same number of pixels in the horizontal direction as in the vertical direction. Syntax data associated with a CU may, for example, describe the partitioning of the CU into one or more PUs. The partitioning mode may differ between whether the CU is coded in an intra-prediction mode or an inter-prediction mode. A PU may be partitioned to be non-square in shape. Syntax data associated with a CU may also, for example, describe the partitioning of a CU into one or more TUs according to a CTU. A TU may be square or non-square in shape.

[0045]

[0058] According to HEVC, transforms may be performed using transform units (TUs). TUs may be different for different CUs. TUs may be sized based on the size of the PUs within a given CU. TUs may be the same size or smaller than the PUs. In some examples, residual samples corresponding to a CU may be subdivided into smaller units using a quadtree structure called a residual quadtree (RQT). Leaf nodes of the RQT may correspond to TUs. Pixel difference values ​​associated with the TUs may be transformed to generate transform coefficients. The transform coefficients may be quantized by the encoder engine 206.

[0046]

[0059] Once a picture of video data is partitioned into CUs, the encoder engine 206 predicts each PU using a prediction mode. The prediction unit or prediction block is subtracted from the original video data to obtain a residual (described below). For each CU, a prediction mode may be signaled within the bitstream using syntax data. The prediction mode may include intra-prediction (or intra-picture prediction) or inter-prediction (or inter-picture prediction). Intra-prediction exploits the correlation between spatially neighboring samples within a picture. For example, using intra-prediction, each PU is predicted from neighboring image data within the same picture using, for example, DC prediction to find the mean value for the PU, planar prediction to fit a flat surface to the PU, directional prediction to extrapolate from neighboring data, or any other suitable type of prediction. Inter-prediction uses temporal correlation between pictures to derive motion-compensated predictions for blocks of image samples. For example, using inter-prediction, each PU is predicted using motion-compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision as to whether a picture area should be coded using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0047]

[0060] As mentioned above, in some cases, the encoder engine 206 and the decoder engine 216 (described in more detail below) may be configured to operate according to VVC. According to VVC, a video coder (such as the encoder engine 206 and / or the decoder engine 216) partitions a picture into multiple coding tree units (CTUs) (where a CTB for luma samples and one or more CTBs for chroma samples, together with syntax for the samples, are referred to as a CTU). The video coder may partition the CTUs according to a tree structure, such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels, including a first level partitioned according to a quadtree partition and a second level partitioned according to a binary tree partition. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0048]

[0061] In the MTT partitioning structure, blocks may be partitioned using quadtree partitioning, binary tree partitioning, and one or more types of triple tree partitioning. Triple tree partitioning is a partition in which a block is split into three sub-blocks. In some examples, triple tree partitioning splits a block into three sub-blocks without splitting the original block through the center. Partition types in MTT (e.g., quadtree, binary tree, and triple tree) can be symmetric or asymmetric.

[0049]

[0062] In some examples, a video coder may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other examples, a video coder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luminance component and another QTBT or MTT structure for both chrominance components (or two QTBT and / or MTT structures for each chrominance component).

[0050]

[0063] The video coder may be configured to use quadtree partitioning per HEVC, QTBT partitioning, MTT partitioning, or other partition structures. For illustrative purposes, the description herein may refer to QTBT partitioning. However, it should be understood that the techniques of this disclosure may also be applied to video coders configured to use quadtree partitioning, or other types of partitioning as well.

[0051]

[0064] As mentioned above, intra-picture prediction exploits the correlation between spatially neighboring samples within a picture. There are multiple intra-prediction modes (also referred to as "intra modes"). In some examples, intra-prediction of luma blocks includes 35 modes, including planar mode, DC mode, and 33 angular modes (e.g., diagonal intra-prediction mode and angular modes adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction are indexed as shown in Table 1 below. In other examples, more intra-modes may be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may differ from those used in HEVC.

[0052] [Table 1]

[0053]

[0065] Inter-picture prediction uses temporal correlation between pictures to derive motion-compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block and Δy specifies its vertical displacement. In some cases, the motion vector (Δx, Δy) can be integer sample precision (also called integer precision), in which case the motion vector points to an integer pel grid (or integer pixel sampling grid) of the reference frame. In some cases, the motion vector (Δx, Δy) can be fractional sample precision (also called fractional pel precision or non-integer precision) to more accurately capture the motion of the underlying object without being limited to the integer pel grid of the reference frame. The precision of the motion vector is represented by the quantization level of the motion vector. For example, the quantization level can be integer precision (e.g., 1 pixel) or fractional pel precision (e.g., 1 / 4 pixel, 1 / 2 pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference picture to derive a prediction signal. For example, available samples at integer positions may be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. A previously decoded reference picture is indicated by a reference index (refIdx) to a reference picture list. The motion vector and the reference index may be referred to as motion parameters. Two types of inter-picture prediction may be performed, including uni-prediction and bi-prediction.

[0054]

[0066] With bi-prediction, two sets of motion parameters (Δx0, y0, refldx0 and Δx1, y 1,refldx1) are used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, with bi-prediction, each prediction block uses two motion compensated prediction signals to generate B prediction units. The two motion compensated predictions are combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions may be combined by averaging. In another example, weighted prediction may be used, in which case a different weight may be applied to each motion compensated prediction. Reference pictures that may be used in bi-prediction are stored in two separate lists, denoted as list 0 and list 1. Motion parameters may be derived in the encoder using a motion estimation process.

[0055]

[0067] When using inter prediction using uni prediction, one set of motion parameters (Δx0, y0, refldx0) is used to generate a motion-compensated prediction from a reference picture. For example, when using uni prediction, each prediction block uses at most one motion-compensated prediction signal to generate P prediction units.

[0056]

[0068] A PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when a PU is encoded using intra prediction, the PU may include data describing the intra prediction mode of the PU. As another example, when a PU is encoded using inter prediction, the PU may include data specifying a motion vector for the PU. The data defining the motion vector of a PU may describe, for example, the horizontal component (Δx) of the motion vector, the vertical component (Δy) of the motion vector, the resolution of the motion vector (e.g., integer precision, ¼-pixel precision, or ⅛-pixel precision), the reference picture to which the motion vector points, the reference index, the reference picture list (e.g., list 0, list 1, or list C) of the motion vector, or any combination thereof.

[0057]

[0069] After performing prediction using intra prediction and / or inter prediction, the encoding device 204 may perform transform and quantization. For example, following prediction, the encoder engine 206 may calculate residual values ​​corresponding to the PU. The residual values ​​may include pixel difference values ​​between the current block of pixels being coded (PU) and a predictive block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a predictive block (e.g., using inter prediction or intra prediction), the encoder engine 206 may generate a residual block by subtracting the predictive block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantify differences between pixel values ​​of the current block and pixel values ​​of the predictive block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0058]

[0070] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform (DCT), a discrete sine transform (DST), an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., kernels of size 32x32, 16x16, 8x8, 4x4, or other suitable sizes) may be applied to the residual data in each CU. In some examples, TUs may be used for the transform and quantization processes implemented by the encoder engine 206. A given CU having one or more PUs may also include one or more TUs. As described in more detail below, residual values ​​may be transformed into transform coefficients using a block transform, and may be quantized and scanned using the TUs to generate serialized transform coefficients for entropy coding.

[0059]

[0071] In some embodiments, following intra-predictive coding or inter-predictive coding using a PU of a CU, the encoder engine 206 may calculate residual data for the TUs of the CU. The PU may include pixel data in the spatial domain (or pixel domain). As mentioned above, the residual data may correspond to pixel difference values ​​between pixels of the uncoded picture and predicted values ​​corresponding to the PU. The encoder engine 206 may form one or more TUs including the residual data for the CU (including the PU) and may transform the TUs to generate transform coefficients for the CU. The TUs may include coefficients in the transform domain after applying a block transform.

[0060]

[0072] The encoder engine 206 may perform quantization of the transform coefficients. Quantization provides further compression by quantifying the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0061]

[0073] Once quantization is performed, the coded video bitstream includes any other suitable data, such as quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partition information, and other syntax data. Different elements of the coded video bitstream may be entropy coded by the encoder engine 206. In some examples, the encoder engine 206 may scan the quantized transform coefficients using a predefined scan order to generate serialized vectors that can be entropy coded. In some examples, the encoder engine 206 may perform adaptive scanning. After scanning the quantized transform coefficients to form vectors (e.g., one-dimensional vectors), the encoder engine 206 may entropy code the vectors. For example, the encoder engine 206 may use context-adaptive variable length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partition entropy coding, or another suitable entropy coding technique.

[0062]

[0074] The output 210 of the encoding device 204 may transmit the NAL units constituting the encoded video bitstream data to a decoding device 212 of a receiving device via a communication link 220. An input 214 of the decoding device 212 may receive the NAL units. The communication link 220 may include channels provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The wireless network may include any wireless interface or a combination of multiple wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi, radio frequency (RF), UWB, WiFi-Direct, cellular, Long Term Evolution (LTE), WiMax, or the like). The wired network may include any wired interface (e.g., fiber, Ethernet, powerline Ethernet, Ethernet over coaxial cable, digital signal line (DSL), etc.). The wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0063]

[0075] In some examples, encoding device 204 may store the encoded video bitstream data in storage 208. Output unit 210 may retrieve the encoded video bitstream data from encoder engine 206 or from storage 208. Storage 208 may include any of a variety of distributed or locally accessed data storage media. For example, storage 208 may include a hard drive, a storage disk, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. Storage 208 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-prediction. In further examples, storage 208 may correspond to a file server or another intermediate storage device that may store encoded video generated by a source device. In such cases, a receiving device, including decoding device 212, can access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to a receiving device. Exemplary file servers include web servers (e.g., for websites), FTP servers, network-attached storage (NAS) devices, or local disk drives. Receiving devices may access the encoded video data through any standard data connection, including an Internet connection. Access may include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both, that are suitable for accessing the encoded video data stored on the file server. Transmission of the encoded video data from storage 208 may be a streaming transmission, a download transmission, or a combination thereof.

[0064]

[0076] The input 214 of the decoding device 212 may receive the encoded video bitstream data and provide the video bitstream data to the decoder engine 216 or to the storage 218 for later use by the decoder engine 216. For example, the storage 218 may include a DPB for storing reference pictures for use in inter-prediction. A receiving device including the decoding device 212 may receive the encoded video data to be decoded via the storage 208. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the receiving device. The communication medium for the transmitted encoded video data may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from a source device to a receiving device.

[0065]

[0077] The decoder engine 216 may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more coded video sequences that make up the encoded video data. The decoder engine 216 may rescale and perform an inverse transform on the encoded video bitstream data. The residual data is passed to a prediction stage of the decoder engine 216. The decoder engine 216 predicts blocks of pixels (e.g., PUs). In some examples, the prediction is added to the output of the inverse transform (the residual data).

[0066]

[0078] Video decoding device 212 may output the decoded video to video destination device 222, which may include a display or other output device for displaying the decoded video data to a content consumer. In some aspects, video destination device 222 may be part of a receiving device that includes decoding device 212. In some aspects, video destination device 222 may be part of a separate device other than the receiving device.

[0067]

[0079] In some embodiments, the video encoding device 204 and / or the video decoding device 212 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 204 and / or the video decoding device 212 may also include other hardware or software necessary to implement the coding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 204 and the video decoding device 212 may be integrated as part of a combined encoder / decoder (codec) in their respective devices.

[0068]

[0080] The exemplary system shown in FIG. 2 is one illustrative example that may be used herein. Techniques for processing video data using the techniques described herein may be performed by any digital video encoding and / or decoding device. Generally, the techniques of this disclosure are implemented by a video encoding device or a video decoding device, although the techniques may also be implemented by a composite video encoder / decoder, commonly referred to as a "codec." Additionally, the techniques of this disclosure may also be implemented by a video preprocessor. The source device and receiving device are merely examples of such coding devices, in which the source device generates coded video data for transmission to the receiving device. In some examples, the source device and receiving device may operate substantially symmetrically, such that each device includes video encoding and decoding components. Thus, the exemplary system may support one-way or two-way video transmission between video devices, e.g., for video streaming, video playback, video broadcasting, or video telephony.

[0069]

[0081] As mentioned above, in some examples, SOC 100 and / or components thereof may be configured to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video coding) using machine learning techniques. For example, encoding device 204 (or encoder) may be used to encode video data using a machine learning system with a deep learning architecture (e.g., by utilizing NPU 108 of SOC 100 of FIG. 1). In some cases, using a deep learning architecture to perform video compression and / or decompression can increase the efficiency of video compression and / or decompression on the device. For example, encoding device 204 may use machine learning-based video coding techniques to more efficiently compress video and transmit the compressed video to decoding device 212, which may decompress the compressed video using machine learning-based techniques.

[0070]

[0082] A neural network is an example of a machine learning system, and may include an input layer, one or more hidden layers, and an output layer. Data is provided by input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is generated through output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, deep learning networks may have a series of many hidden layers, with early layers used to determine simple, low-level characteristics of the input and later layers building a hierarchy of more complex and abstract characteristics.

[0071]

[0083] Deep learning architectures may learn a hierarchy of features. When presented with visual data, for example, a first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0072]

[0084] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, classification of electric vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features may be combined in different ways at higher layers to recognize cars, trucks, and airplanes.

[0073]

[0085] Neural networks may be designed with various connectivity patterns. In feedforward networks, each neuron in a given layer communicates with neurons in a higher layer, and information is passed from lower layers to higher layers. As described above, hierarchical representations may be constructed within successive layers of a feedforward network. Neural networks may also have recurrent or feedback (also called top-down) connections. In recurrent connections, the output from a neuron in a given layer may be transmitted to another neuron in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks delivered to the neural network in sequence. Connections from neurons in a given layer to neurons in a lower layer are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognizing high-level concepts can help distinguish certain low-level features of the input. Connections between layers of a neural network may be fully connected or locally connected. Various examples of neural network architectures are described below with reference to FIGS. 4A-5.

[0074]

[0086] 3 illustrates a system 300 including a device 302 configured to perform video encoding using a machine learning coding system 310. The device 302 is coupled to a camera 307 and storage 314 (e.g., a data storage device). In some implementations, the camera 307 is configured to provide image data 308 (e.g., a video data stream) to a processor 304 for encoding by the machine learning coding system 310. In some implementations, the device 302 can be coupled to and / or include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 302 can be coupled to a microphone and / or other input devices (e.g., a keyboard, a mouse, a touch input device such as a touchscreen and / or touchpad, and / or other input devices). In some examples, the camera 307, storage 314, microphone, and / or other input devices may be part of the device 302.

[0075]

[0087] The device 302 is also coupled to a second device 390 via a transmission medium 318, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 318 may include channels provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The transmission medium 318 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 318 may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from a source device to a receiving device. The wireless network may include any wireless interface or a combination of multiple wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi, radio frequency (RF), UWB, WiFi-Direct, cellular, Long Term Evolution (LTE), WiMax, or the like). The wired network may include any wired interface (e.g., fiber, Ethernet, powerline Ethernet, Ethernet over coaxial cable, Digital Signal Line (DSL), etc.). Wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0076]

[0088] Device 302 includes one or more processors 304 (referred to herein as "processors") coupled to a memory 306, a first interface ("I / F1") 312, and a second interface ("I / F2") 316. Processor 304 is configured to receive image data 308 from a camera 307, from memory 306, and / or from a storage medium 314. Processor 304 is coupled to storage 314 via first interface 312 (e.g., via a memory bus) and to transmission medium 318 via second interface 316 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0077]

[0089] Device 390 is similar to device 302 and includes one or more processors 394 (referred to herein as “processors”) coupled to a memory 392, a first interface (“I / F1”) 396, and a second interface (“I / F2”) 398. Processor 392 is configured to receive data from a transmission medium 318, from memory 306, and / or from storage 314 via second interface 396. Processor 394 is coupled to storage 399 via first interface 398 (e.g., via a memory bus) and to transmission medium 318 via second interface 396 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0078]

[0090] The processor 304 includes a machine learning coding system 310. The machine learning coding system 310 includes an encoder portion 362. The encoder portion 362 is configured to receive input data 370 and process the input data 370 to generate encoded data 374 based at least in part on the input data 370. In some cases, the machine learning coding system 310 may include both the encoder portion 362 and a decoder portion, such as the decoder portion 366 shown here to be included in the processor 394 of the device 390. In some implementations, the machine learning coding system 310 may include one or more autoencoders.

[0079]

[0091] In some implementations, the encoder portion 362 of the machine learning coding system 310 is configured to perform lossy compression of the input data 370 to generate encoded data 374, such that the encoded data 374 has fewer bits than the input data 370.

[0080]

[0092] As shown, encoder portion 362 of machine learning coding system 310 may include machine learning-based preprocessor 363 and encoder 364. As indicated above, some implementations of processor 304 include multiple processors, and elements of encoder portion 362 may be included on (e.g., executed by) different ones of the multiple processors. For example, encoder 364 may be included on a custom processor, such as an application-specific integrated circuit (ASIC).

[0081]

[0093] In some cases, video consumption is growing very rapidly, while technical requirements are becoming much more stringent with the adoption of increased resolution, frame rate, and dynamic range. To provide the desired quality of service with the required data processing speed, high throughput, and low power consumption, current consumer video applications are best supported using custom hardware, such as ASICs, that implement video compression standards such as H.264 / AVC and H.265 / HEVC.

[0082]

[0094] In some cases, updating an ASIC can be difficult and relatively slow, which can limit opportunities to make improvements to existing ASICs as new technologies and features become available. Machine learning and neural network implementations can be used to help improve the performance of existing ASICs using a technique referred to herein as video codec neural boosting. A machine learning-based preprocessor 363 can be used to perform video codec neural boosting.

[0083]

[0095] In some implementations, the machine learning-based preprocessor 363 may be included on a machine learning accelerator or other processor optimized for performing machine learning / artificial intelligence processing. The machine learning-based preprocessor 363 may include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures. The machine learning-based preprocessor 363 may receive as input encoder parameters 366 for configuring the encoder 364. The encoder parameters 366 may include parameters for configuring the operation of the encoder 364, such as quality parameters (e.g., quality settings, parameters for enabling / disabling optimizations, or other parameters for controlling how the encoder 364 operates). In some cases, the machine learning-based pre-processor 363 may also receive codec internal data 368 from the encoder 364 as input. The codec internal data 368 may include information or settings used to perform one or more steps of the encoding process, such as scaled transform coefficients, quantizer step sizes, etc. Based on the input (such as the input data 370, the encoder parameters 366, and / or the codec internal data 368), the machine learning-based pre-processor 363 may generate codec boosting information 372, such as control side information and / or pixel side information, to improve the performance of the encoder 364. The machine learning-based pre-processor 363 may pass the codec boosting information 372 along with the encoded data 374 to the decoder portion 366 for decoding. The machine learning-based pre-processor 363 may pass the encoder parameters 366 along with the input data 370 to the encoder 364 for encoding.In some cases, the encoder 364 may encode the input data 370 based on the encoder parameters 366 using existing video compression standards such as H.265 / AVC and H.264 / HEVC to generate encoded data 374. The encoded data 374 may be transmitted to the device 390 via the second interface 316 over the transmission medium 318.

[0084]

[0096] The processor 394 of the device 390 includes a machine learning decoding system 350. The machine learning decoding system 350 includes a decoder portion 366. The device 390 may receive encoded data 374 along with codec boost information 372 via a second interface 396 and pass the encoded data 374 and codec boosting information 372 to the decoder portion 366. In this example, the decoder portion 366 includes a decoder 352 and a machine learning-based post-processor 354. The decoder portion 366 is configured to receive the encoded data 376 and process the encoded data 376 to generate a representation 378 based on the input data 370, which may be displayed to a user, for example, via a display (not shown). In some cases, the decoder 352 may decode the encoded data 376 using existing video compression standards, such as H.265 / AVC and H.264 / HEVC, to generate decoded data 356. The decoded data 356 may be passed to the machine learning-based post-processor 354. In some cases, the machine learning decoding system 350 may include both an encoder portion, such as the encoder portion 362, and a decoder portion 366.

[0085]

[0097] The machine learning-based post-processor 354 may receive the decoded data 356 along with the codec boosting information 372. The machine learning-based post-processor 354 may enhance the decoded data 356 using one or more trained machine learning algorithms based on the input codec boosting information 372 to generate a representation 378. In some examples, the decoder portion's machine learning-based post-processor 354 includes a neural network, which may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures. The processor 394 may be configured to transmit the representation 378 to a storage medium 399 or output the representation 378 for display on a display (not shown).

[0086]

[0098] The components of system 300 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuitry (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0087]

[0099] Although system 300 is shown as including several components, one skilled in the art will understand that system 300 can include more or fewer components than those shown in Figure 3. For example, system 300 can also include or be part of a computing device including input and output devices (not shown). In some implementations, system 300 may also include, or be part of, a computing device that includes one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) in communication with and / or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and baseband processors for each wireless interface) for performing wireless communications, one or more wired interfaces (e.g., serial interfaces such as universal serial bus (USB) inputs, lightning connectors, and / or other wired interfaces) for performing communications over one or more hardwired connections, and / or other components not shown in FIG. 3 .

[0088]

[0100] In some implementations, system 300 may be implemented locally by and / or included in a computing device, such as a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head-mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses, or other AR device), a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), a television, and / or any other computing device having the resource capabilities to perform the techniques described herein.

[0089]

[0101] As mentioned above, some video coding systems utilize neural networks or other machine learning systems to compress video and / or image data. Neural networks may be designed with various connectivity patterns. In feedforward networks, each neuron in a given layer communicates with neurons in higher layers, and information is passed from lower layers to higher layers. As mentioned above, hierarchical representations may be built within successive layers of a feedforward network. Neural networks may also have recurrent or feedback (also called top-down) connections. In recurrent connections, outputs from neurons in a given layer may be transmitted to other neurons in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks delivered to the neural network in a sequence. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with numerous feedback connections may be useful when recognizing high-level concepts can help distinguish certain low-level features of the input.

[0090]

[0102] Connections between layers of a neural network may be fully connected or locally connected. FIG. 4A shows an example of a fully connected neural network 402. In a fully connected neural network 402, a neuron in a first layer may transmit its output to every neuron in a second layer, resulting in each neuron in the second layer receiving input from every neuron in the first layer. FIG. 4B shows an example of a locally connected neural network 404. In a locally connected neural network 404, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of a locally connected neural network 404 can be configured so that each neuron in a layer has the same or similar connectivity pattern, but with connection strengths that may have different values ​​(e.g., 410, 412, 414, and 416). Because higher layer neurons in a given region may receive inputs that, through training, are tuned to the properties of a constrained subset of all inputs to the network, the connectivity patterns of local connections may give rise to spatially distinct receptive fields within the higher layers.

[0091]

[0103] An example of a locally connected neural network is a convolutional neural network. FIG. 4C shows an example of a convolutional neural network 406. The convolutional neural network 406 may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g., 408). Convolutional neural networks may be suitable for problems in which the spatial location of the inputs is meaningful. The convolutional neural network 406 may be used to perform one or more aspects of video compression and / or decompression according to aspects of the present disclosure.

[0092]

[0104] One type of convolutional neural network is the deep convolutional network (DCN). Figure 4D shows a detailed example of a DCN 400 designed to recognize visual features from an image 426 input from an image capture device 430, such as an in-car camera. The DCN 400 in this example can be trained to identify traffic signs and numbers on traffic signs. Of course, the DCN 400 can be trained for other tasks, such as recognizing lane markings or identifying traffic signals.

[0093]

[0105] The DCN 400 can be trained using supervised learning. During training, the DCN 400 may be presented with an image, such as a speed limit sign image 426, and can then compute a forward pass to generate the output 422. The DCN 400 may include a feature extraction section and a classification section. Upon receiving the image 426, the convolutional layer 432 may apply a convolutional kernel (not shown) to the image 426 to generate the first set of feature maps 418. As an example, the convolutional kernel for the convolutional layer 432 may be a 5×5 kernel that generates 28×28 feature maps. In this example, four different feature maps are generated in the first set of feature maps 418, and therefore four different convolutional kernels were applied to the image 426 in the convolutional layer 432. A convolutional kernel may also be referred to as a filter or a convolutional filter.

[0094]

[0106] The first set of feature maps 418 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 420. The max pooling layer reduces the size of the first set of feature maps 418. That is, the size of the second set of feature maps 420, such as 14x14, is smaller than the size of the first set of feature maps 418, such as 28x28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 420 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0095]

[0107] 4D , a second set of feature maps 420 is convolved to generate a first feature vector 424. Furthermore, the first feature vector 424 is further convolved to generate a second feature vector 428. Each feature in the second feature vector 428 may include a number corresponding to a possible feature of the image 426, such as "sign," "60," and "100." A softmax function (not shown) may convert the numbers in the second feature vector 428 into probabilities. Thus, the output 422 of the DCN 400 is the probability that the image 426 contains one or more features.

[0096]

[0108] In this example, the probability in output 422 for "sign" and "60" is higher than the probability for other outputs 422, such as "30," "40," "50," "70," "80," "90," and "100." Prior to training, the output 422 produced by DCN 400 may be inaccurate. Therefore, an error may be calculated between output 422 and a target output. The target output is the ground truth (e.g., "sign" and "60") for image 426. The weights of DCN 400 may then be adjusted so that output 422 of DCN 400 more closely aligns with the target output.

[0097]

[0109] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient can indicate the amount by which the error will increase or decrease if the weights are adjusted. In the top layer, the gradient can correspond directly to the values ​​of the weights connecting activated neurons in the penultimate layer to neurons in the output layer. In lower layers, the gradient can depend on the values ​​of the weights and the calculated error gradients of the upper layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward pass" through the neural network.

[0098]

[0110] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the overall system stops decreasing or reaches a target level. After training, the DCN may be presented with a new image, and a forward pass through the network may provide an output 422, which may be considered the DCN's estimate or prediction.

[0099]

[0111] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs may be used to extract hierarchical representations of a training dataset. DBNs may be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions without information about the class into which each input should be categorized, RBMs are frequently used in unsupervised learning. Using a hybrid unsupervised-supervised paradigm, the lower RBM of a DBN can be trained in an unsupervised manner and function as a feature extractor, and the upper RBM can be trained in a supervised manner (on the joint distribution of inputs from previous layers and the target class) and function as a classifier.

[0100]

[0112] Deep convolutional networks (DCNs) are networks of convolutional networks constructed with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for a large number of examples and are used to modify the network weights using gradient descent.

[0101]

[0113] A DCN may be a feedforward network. Additionally, as described above, connections from neurons in a first layer of a DCN to groups of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be exploited for high-speed processing. The computational burden of a DCN may be much smaller than that of a similarly sized neural network that includes, for example, recurrent or feedback connections.

[0102]

[0114] The processing in each layer of a convolutional network may be viewed as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input may be viewed as three-dimensional, with two spatial dimensions along the image axes and a third dimension capturing color information. The output of the convolutional connections may be viewed as forming a feature map in subsequent layers, with each element of the feature map (e.g., 420) receiving input from a range of neurons in the previous layer (e.g., feature map 418) and from each of multiple channels. Values ​​in the feature map may be further processed using nonlinearities such as rectification, max(0,x), etc. Values ​​from neighboring neurons may also be pooled, corresponding to downsampling, to provide additional local invariance and dimensionality reduction.

[0103]

[0115] FIG. 5 is a block diagram illustrating an example of a deep convolutional network 550. The deep convolutional network 550 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 5, the deep convolutional network 550 includes convolution blocks 554A and 554B. Each of the convolution blocks 554A and 554B may be configured using a convolution layer (CONV) 556, a normalization layer (LNorm) 558, and a max pooling layer (MAX POOL) 560.

[0104]

[0116] The convolutional layer 556 may include one or more convolutional filters that may be applied to the input data 552 to generate feature maps. While only two convolutional blocks 554A, 554B are shown, the present disclosure is not so limited; instead, any number of convolutional blocks (e.g., blocks 554A, 554B) may be included in the deep convolutional network 550 according to design preference. The normalization layer 558 may normalize the outputs of the convolutional filters. For example, the normalization layer 558 may provide whitening or lateral inhibition. The max-pooling layer 560 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0105]

[0117] For example, a parallel filter bank of a deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 550 may access other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0106]

[0118] The deep convolutional network 550 may also include one or more fully connected layers, such as layer 562A (labeled "FC1") and layer 562B (labeled "FC2"). The deep convolutional network 550 may further include a logistic regression (LR) layer 564. Between each layer 556, 558, 560, 562A, 562B, 564 of the deep convolutional network 550, there are weights (not shown) that must be updated. The output of each of the layers (e.g., 556, 558, 560, 562A, 562B, 564) can serve as an input to a subsequent one of the layers (e.g., 556, 558, 560, 562A, 562B, 564) in the deep convolutional network 550 to learn a hierarchical feature representation from the input data 552 (e.g., image, audio, video, sensor data, and / or other input data) provided in the initial convolution block 554A. The output of the deep convolutional network 550 is a classification score 566 for the input data 552. The classification score 566 can be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0107]

[0119] While traditional video compression techniques are sometimes empirically designed using human knowledge and intuition, neural networks and other machine learning tools rely on large amounts of training data and efficient learning algorithms. Generally, there are two main classes of learning algorithms: (1) techniques based on discrete trial-and-error testing with on-the-fly generation of optimization strategies, and (2) techniques that calculate the gradient of an objective (or loss) function and use automatic differentiation in the form of gradient backpropagation to optimize system parameters. While both techniques have been successfully used to solve practical problems, the availability of derivatives of performance measures typically makes the second technique much more efficient and, as a result, much more commonly used. For example, the second technique is the technique used in well-known development tools such as TensorFlow and PyTorch.

[0108]

[0120] One problem with video coding systems is that some operations implemented by standard video codecs are not differentiable. In addition, video coding systems may include several stages, such as adaptive deblocking filters, adaptive loop filters, etc., which may be linear and differentiable, but are very complex and difficult to integrate into training programs.

[0109]

[0121] In some cases, when a neural boosting system is designed (e.g., during the learning (or training) phase), the compressed data and decoded video of the codec are not used directly, since only a measure of the codec performance, along with its derivative, is needed for optimization. Thus, if sufficiently good estimates of the measure and its derivative can be obtained, the full codec is not needed. Systems designed for this purpose are referred to herein as differentiable codec proxies.

[0110]

[0122] As explained above, entropy coding is one of the final stages (and in some cases the final stage) of encoding (compression) and defines the value and number of bits to be added to the compressed data bitstream. Modern standards-based video encoding methods (e.g., VVC, HEVC, AV1, etc.) employ adaptive arithmetic coding to enable high-quality compression performance. The bitstream generated by adaptive arithmetic coding can only be coded and decoded sequentially. For example, a data element can only be restored by first decoding all previous elements, because the decoder needs to reach the same state the encoder had when it coded that element.

[0111]

[0123] FIG. 6A is a block diagram 600 illustrating an example implementation of a video codec neural boosting system according to aspects of the present disclosure. In the example video codec neural boosting system, a standard codec (e.g., VVC, HEVC, AVC, MPEG, AV1, or other similar codec) may be used when a codec with neural boosting is deployed within the video codec neural boosting system. Video data 604 may be input to an ML pre-processing engine 606. The ML pre-processing engine 606 may include one or more ML models. The one or more ML models may perform various pre-processing tasks. Examples of pre-processing tasks may include spatial upsampling, temporal upsampling, quality optimization, compression artifact removal, selective noise removal, dynamic frame resizing, etc. Generally, these pre-processing tasks are directed to improving the quality of the video and / or video compression without modifying the video encoder 608, which may be an existing standards-compliant video encoder (e.g., VVC, HEVC, AVC, MPEG, AV1, or other similar encoder). For example, one pre-processing task may be to apply an ML model to perform selective noise reduction to remove types of noise that the encoder may be less able to process to improve compression of the video. As another example, the ML model may dynamically adjust encoder parameters 614 based on input video data 604 to improve compression or quality of the video. The compressed video 610 may be transmitted 612 to a playback device (which may be any device, including an encoding device) for decoding by a video decoder 616. In some cases, the video decoder 616 may be a standard decoder (e.g., VVCV, HEVC, AVC, MPEG, AV1, or other similar decoder). The video data 604 output by the video decoder 616 may be input to an ML post-processing engine 618. The ML post-processing engine 618 may perform various post-processing tasks.For example, the ML post-processing engine 618 may apply one or more ML models to dynamically add noise to the output video similar to the noise removed by the ML pre-processing engine 606.

[0112]

[0124] FIG. 6B is a block diagram 650 illustrating training of a video codec neural boosting system according to an aspect of the present disclosure. In some cases, during the network learning (training) phase, the codec is replaced by a differentiable codec proxy 652 that enables gradient backpropagation 654. The video data 654 may be input to an ML pre-processing engine 656. In some cases, the ML pre-processing engine 656 may be the ML pre-processing engine 606 being trained. The output from the ML pre-processing engine 656 is input to the differentiable codec proxy 652. The differentiable codec proxy 652 may estimate a bitrate 670 of the compressed video that a video encoder, such as the video encoder 608, may output. The differentiable codec proxy 652 may pass the estimated bitrate 670 to a loss measurement engine 668. The differentiable codec proxy 652 may also pass the video data 654 to the ML post-processing engine 668. In some cases, the ML post-processing engine 668 may be the ML post-processing engine 618 being trained. The output from the ML post-processing engine 668 may be passed to a loss measurement engine 668, which may compare the output from the ML post-processing engine 668 and the estimated bitrate 670 to a ground truth reference to calculate a loss function and gradient 672. This gradient 672 may be back-propagated to the ML pre-processing engine 656, the differentiable codec proxy 652, and the ML post-processing engine 668 for training.

[0113]

[0125] In some cases, the effectiveness of using a differentiable codec proxy 652 depends on the accuracy of the proxy estimate, which is defined by two factors controlled by a selected quality parameter, referred to herein as QP. The bitrate R (related to the QP) corresponds to the number of bits used by the encoder to compress a video frame or block within a frame, divided by the number of pixels to normalize to bits per pixel. The bitrate decreases with increasing QP. The distortion D (also related to the QP) is a measure of the difference between the original and decoded video pixels, also normalized per pixel. The distortion can correspond, for example, to mean squared error or a more complex measure that approximates human subjective preference. The distortion increases with increasing QP.

[0114]

[0126] The bit rate and distortion can be combined into a single loss function, such as the loss function defined in Equation 1 below: L(QP) = D(QP) + λ(QP)R(QP), Formula (1)

[0127] where λ(QP) is the Lagrange multiplier defined together with the quantization step according to the definition of QP in the video standard.

[0115]

[0128] One observation is that the distortion measure strongly depends on the quantization step, predefined by the encoder parameter QP, and is therefore more predictable and easier to estimate, whereas the bitrate is more difficult to estimate, since it is affected by complex statistical dependencies between many pixels.

[0116]

[0129] Some video encoders, such as some HEVC / H.265 standard video encoders, implement what is known as a hybrid coding scheme, which combines predictive coding and transform coding. The components that perform the prediction and transform are linear and therefore differentiable, making them easier to integrate directly into the training process.

[0117]

[0130] In some cases, some encoder elements are nonlinear and non-differentiable, such as those defined by quantization and entropy coding processes. As used herein, non-differentiable may refer to a process that is not effectively differentiable for gradient backpropagation and / or loss measurement. For example, a decimal rounding function may be a basic form of quantization in which input decimal numbers such as 3.14 and 2.56 are rounded to 3. The derivative of the function's output over a range of inputs, such as the set of decimal numbers from 2.51 to 3.50 (assuming two decimal places), is simply 0 from 2.51 to 3.49 and infinity for 3.50. Such a differential output may not be useful for gradient backpropagation because no actual gradient is present.

[0118]

[0131] 7 is a block diagram illustrating encoder elements of a video encoder 700, such as an HEVC encoder or an H.265 encoder, according to aspects of this disclosure. In some cases, certain encoder elements may be nonlinear and non-differentiable, such as the quantization 702 and entropy coding 704 processes. Given a block of dimensions M×N pixels, a linear transform (typically a discrete cosine transform) is applied to the residuals (e.g., an array of differences between predicted and actual pixel values). The resulting array of transform coefficients d_(m,n) is then scaled (divided by a positive quantizer step size s 706) to obtain

[0119]

number

[0120] It is quantized according to the following equation 2.

[0121]

number

[0122]

[0132] where ξ is an offset that defines the type of dead-zone quantization.

[0123]

[0133] an array q of quantized transform coefficients m,n is entropy coded and defines the number of bits used to code the block. Since the quantization is not differentiable, any proxy estimate of q m,n Instead of the array of scaled transform coefficients c m,n You must use the value from

[0124]

[0134] In some cases, the image coding differentiable proxy can estimate the number of bits as a function such as Equation 3 below.

[0125]

number

[0126]

[0135] where μ is a normalization constant.

[0127]

[0136] However, current video coding standards employ sophisticated binary adaptive arithmetic coding, which improves compression by processing the array in multiple passes, using coding contexts to exploit statistical dependencies between the elements of the array. m,n Therefore, estimators such as the above formula, which do not take into account inter-coefficient dependencies, have very limited accuracy.

[0128]

[0137] As described above, according to aspects of the present disclosure, a joint statistical model is defined for all values ​​in the array, and c m,n Systems and techniques are described herein that address at least this problem by computing maximum likelihood estimates of the model parameters and then using the model to estimate the number of bits. Such a process is differentiable.

[0129]

[0138] 8 is a block diagram illustrating elements of a differentiable encoder proxy 800 according to an aspect of the present disclosure. In the differentiable encoder proxy 800, the non-differentiable quantization 702 and entropy coding 704 elements of an encoder (e.g., video encoder 700) may be replaced by a differentiable rate estimation process for the encoder proxy 800. The differentiable rate estimation process includes a noise and adjustment process 802, a parameter estimation process 804, and a block mode 806. The proxy implementation is assumed to approximate the main encoder decisions, such as selecting the appropriate prediction 808, block dimension (M, N), and transform type 810.

[0130]

[0139] Next, we discuss regularization and coefficient adjustment. For example, in some cases, quantization may alter the scaled transform coefficients in a process similar to adding a random quantization noise η with an approximately uniform distribution in the interval [−0.5, 0.5]. To that end, we may define two arrays of uniformly distributed random numbers as follows:

[0131]

number

[0132] In some cases, for two arrays of random values, one may be selected to be non-zero and the other may be selected to be zero.

[0133]

[0140] In some cases, this technique of adding random numbers is used when training end-to-end neural codecs to account for the quantization of latent variables. However, in this example, it is included because it has been empirically observed to improve estimation accuracy and can also be used as a form of regularization, avoiding numerical instability during estimation when all transform coefficients have very small or zero magnitude.

[0134]

[0141] Another factor is that, even though not required by the standard, video codecs often use dead-zone quantizers such as Equation 2, which increase the probability of quantizing to a value of 0. To approximate this feature, an "adjust" function is defined as Equation 4 below:

[0135]

number

[0136]

[0142] where x is the argument of the function, and A and the integer K are positive constants (parameters). In some cases, the adjustment function may be updated, for example, during the training process for the encoder proxy. For example, the values ​​of A and K, or x, may be adjusted based on a loss function.

[0137]

[0143] An exemplary derivative of Equation 4 is shown below in Equation 5:

[0138]

number

[0139]

[0144] 9 is a graph plotting an example adjustment function and the derivative of the adjustment function 900, according to an embodiment of the present disclosure. In this example, FIG. 9 shows a first line 902 corresponding to given Equation 4.

[0140]

number

[0141] From these definitions, the random variables can be defined in the statistical model used for bitrate estimation as shown in Equation 6 below.

[0142]

number

[0145] A second line 904 shown in FIG. 9 corresponds to the derivative of Equation 4 (eg, Equation 5).

[0143]

[0146] Next, we will describe the definition of the statistical model and parameter estimation. For example, in some cases, the statistical model may be defined by a vector x containing three parameters of the model and may be based on one or more assumptions. The first assumption under which the statistical model may be defined is the array t m,n All elements of are independent, have mean 0 and scale s m,n (x) has a Laplace distribution. For example, the array t m,n The probability distribution function for the elements of can be described by Equation 7 below:

[0144]

number

[0145]

[0147] For simplicity of notation, the Laplace distribution is

[0146]

number

[0147] Alternatively, it may be parameterized by a scale s.

[0148]

[0148] A second assumption that allows the statistical model to be defined is that the array of Laplacian distribution scales is defined by Equation 8 below:

[0149]

number

[0150] In some cases, the probability distribution of the transform coefficients can be well approximated by a Laplace distribution. The fast decay of scale with frequency (defined by indices m and n) is also well known. In some cases, choosing an exponential function to model the decay significantly simplifies the formulas for the derivatives and their calculations. In some cases, the values ​​of indices m and n are updated, for example, during the training process for the encoder proxy. For example, indices m and n can be adjusted based on a loss function during training.

[0151] Next, we will explain model parameter estimation. For example, given an array t m,n Among the modified transform coefficients in the block, a maximum likelihood (ML) method can be used to estimate the parameter vector x. As explained above, the array defined in Equation 6

[0152]

number

[0153] In array t m,n By replacing , prediction accuracy can be improved and numerical instability can be avoided. For simplicity of notation, Equations 9 and 10 can be defined as follows:

[0154]

number

[0155]

[0151] From Equations 6, 8, 9, and 10, it can be shown that the negative of the log-likelihood (e.g., the function to be minimized to obtain the optimal parameter vector) is as shown in Equation 11 below.

[0156]

number

[0157]

[0152] Maximum likelihood parameters

[0158]

number

[0159] The optimal solution corresponding to is defined by setting the gradient of Equation 11 to 0, which can be shown to correspond to the first set of equations:

[0160]

number

[0161]

[0153] Although a closed-form solution to this first set of equations is not known, the solution can be efficiently computed using Newton's method iterations defined by Equation 12 below.

[0162]

number

[0163] where 0<ξ≦1 is a multiplication factor added to ensure convergence.

[0164] In Equation 12, the gradient is:

[0165]

number

[0166]

[0156] And the corresponding Hessian matrix is:

[0167]

number

[0168]

[0157] In some cases, the output gradient of Equation 12 may be determined and used for backpropagation as part of a loss function by a loss measurement system, such as loss measurement engine 668 of FIG. 6, for example, to train a preprocessor, such as ML preprocessing engine 656, or a differentiable codec proxy, such as differentiable codec proxy 652.

[0169]

[0158] Some observations about the practical use of Newton's method are as follows.

[0170] (1) As mentioned above, the use of exponential functions simplifies the derivative expressions and their calculations, since some coefficients appear repeatedly in the above expressions. For example, the coefficients {w m,n’ mw m,n’ nw m,n m 2 w m,n’ n 2 w m,n’ mnw m,n Arrays with} can be computed once and reused in each iteration.

[0171] (2) Divergence can be avoided by using an adaptive step correction method, such as first trying ξ=1 at each iteration and halving the value of L(x) if it is not decreasing.

[0172] (3) Because the Hessian matrix is ​​symmetric, the step size for Newton's method in Equation 12 is more efficiently calculated using Cholesky matrix factorization instead of matrix inversion.

[0173]

[0162] Experimental results show that when a reasonably good initial solution is used, quadratic convergence begins after only 2 or 3 iterations. For example, in experiments using 8x8 DCT coefficients, the following initial solution can be used:

[0174]

number

[0175]

[0163] Next, we will discuss bit rate estimation. As shown in Figure 7, entropy coding 704 in video compression is applied to the integers obtained by quantization, and a probability value is assigned to each quantization interval. In some cases, to approximate this stage in a differentiable way, relaxation techniques developed for training end-to-end neural image codecs can be used, where the fixed set of intervals used for quantization and probability calculations are replaced by intervals surrounding the coefficient values.

[0176] In this approach, the probability values ​​corresponding to the values ​​used for entropy coding can be defined by Equation 13 below:

[0177]

number

[0178] In Equation 13, the cumulative distribution of the Laplace distribution is defined as in Equation 14 below.

[0179]

number

[0180] Using these definitions, an estimate of the number of bits used to code a block may be defined by Equation 15 below:

[0181]

number

[0182] where α is a normalization constant In some cases, the estimated number of bits may be generated by a differentiable codec proxy, such as, for example, the differentiable codec proxy 652 of Figure 6. The estimated number of bits may be used for back-propagation, for example, by a loss measurement system, such as, for example, the loss measurement engine 668.

[0183]

[0169] Next, partial derivative calculation will be described: To train a neural network, the estimated number of bits of gradient (eg, an array of partial derivatives) can be used according to Equation 16 below.

[0184]

number

[0185] In some cases, the array of partial derivatives can be calculated using automatic differentiation tools available in software tools (e.g., PyTorch), but to enable backpropagation, Equation 12 can be written as

[0186]

number

[0187] to create a convergent set of solutions, allowing the software to create the appropriate links for automatic differentiation. Such techniques can increase computational complexity by adding multiple stages of gradient calculations. A more efficient implementation can be achieved if automatic differentiation is removed from the maximum likelihood optimization and uses the optimal solutions to directly calculate their partial derivatives.

[0188] To calculate these derivatives,

[0189]

number

[0190] It can be defined as:

[0191]

[0172] Or, partial derivative calculation, optimality condition

[0192]

number

[0193] may correspond to the following set of MN vector equations:

[0194]

number

[0195]

[0173] In the formula,

[0196]

number

[0197] Using these results, equation 16 can be expanded as follows:

[0198]

number

[0199] This is the formula that defines the gradient calculation.

[0200]

number

[0201] During the ceremony,

[0202]

number

[0203]

[0175] Then, from equations 5 and 6, equation 17 is derived as follows:

[0204]

number

[0205]

[0176] In some aspects, equation (17) may be simplified by collecting the sum of terms that vary with index m, n. For example, the following function may be defined:

[0206]

number

[0207]

[0177] Then, the vector z becomes as follows:

[0208]

number

[0209]

[0178] Therefore, equation (17) can correspond to:

[0210]

number

[0211]

[0179] An illustrative example of experimental validation will now be described. For example, the systems and techniques described herein can be tested by modifying a reference implementation of the HEVC / H.265 video compression standard, called HM version 20.0, to output DCT coefficients used by the encoder in a low-delay configuration, luma transform dimensions limited to 8x8 blocks, QP values ​​22, 27, 32, and 37, and the number of bits used in each case.

[0212] Compression was applied to 201 frames of 10 HD test videos, preserving only the luma coefficients within inter-frames (e.g., intra-frames were excluded). For each frame, the bit rate is calculated as follows: (c) =0.5,ε (t) = 0, and A = 1, K = 2 and α = 1 in Equation 5 were first used in Equation 15. After comparing the estimated bit rate with the actual bit rate, the parameter α was calibrated to α = 5 / 3, which is the value used in all results presented below.

[0213]

[0181] In some cases, the choice of parameter α is not important because during neural network training, the bit rate estimate is multiplied by a constant as in Equation 1, and several constant values ​​need to be tested to get the best results. Similarly, an estimator calibrated from HEVC can be used with AVC by testing new scaling factors.

[0214] The results obtained using the systems and techniques described herein can be evaluated by measuring the ratio between the estimated frame bit rate and the corresponding value from the HM software. Figures 10-13 show histograms of these ratios calculated for 2000 frames of the test video, with each figure showing results for QP=22, 27, 32, and 37. As expected, the results are more peaked at low QP, which corresponds to the high-rate case where per-coefficient factors dominate. The histograms become more spread out as the QP value increases, and well-constrained errors can be observed even for the maximum QP=27.

[0215]

[0183] Figure 10 is a histogram of the ratio between the frame bit rate estimated using the systems and techniques described in this specification and the bit rate used by the HM software (HEVC / H.265 reference implementation) and QP=22.

[0216]

[0184] Figure 11 is a histogram of the ratio between the frame bit rate estimated using the systems and techniques described in this specification and the bit rate used by the HM software (HEVC / H.265 reference implementation) and QP=27.

[0217]

[0185] Figure 12 is a histogram of the ratio between the frame bit rate estimated using the systems and techniques described in this specification and the bit rate used by the HM software (HEVC / H.265 reference implementation) and QP=32.

[0218]

[0186] Figure 13 is a histogram of the ratio between the frame bit rate estimated using the systems and techniques described in this specification and the bit rate used by the HM software (HEVC / H.265 reference implementation) and QP=37.

[0219]

[0187] Figure 14 is a histogram of the ratio between the frame bit rate estimated using the technique for bit rate estimation to improve video coding using machine learning and the bit rate used by the HM software (HEVC / H.265 reference implementation), combining results for QP=22, 27, 32, and 37.

[0220]

[0188] Figure 14 shows the combined results for all QP values, which can be compared to values ​​estimated using the non-differentiable AGP entropy coding method shown in Figure 15, which uses a much simpler way of modeling the transform coefficients than that used by HEVC. Finally, Figure 16 shows the results obtained using the coefficient-by-coefficient differentiable estimation of Equation 3, which shows a wide spread of results and much lower accuracy.

[0221]

[0189] Figure 15 is a histogram of the ratio between the frame bit rate estimated using the AGP entropy coding method (non-differentiable) and the bit rate used by the HM software (HEVC / H.265 reference implementation), combining results for QP=22, 27, 32, and 37.

[0222]

[0190] Figure 16 is a histogram of the ratio between the frame bit rate estimated using the coefficient-wise differentiable estimation of Equation 3 and the bit rate used by the HM software (HEVC / H.265 reference implementation), combining results for QP = 22, 27, 32, and 37.

[0223]

[0191] For reference, Table 1 shows the average bit rates for different QP values, showing that they cover roughly an order of magnitude.

[0224] [Table 2]

[0225]

[0192] Aspects of the present disclosure address a problem that arises when the performance of a standard video codec is improved in some way using a neural network (referred to as standard video codec neural boosting), and the gradient of the performance measure cannot be backpropagated through the codec, preventing full optimization.

[0226]

[0193] Aspects of the present disclosure address the problem of reliably estimating the bitrate used by a video encoder while simultaneously estimating the corresponding derivatives and enabling gradient backpropagation for use during end-to-end training.

[0227]

[0194] This estimation is more accurate and reliable because it combines information from many of the discrete cosine transform (DCT) coefficients used for compression, similar to entropy coding in standard encoders. This is achieved using a new form of statistical model, assuming that the DCT coefficients have a Laplacian distribution, with three parameters determined using a maximum likelihood criterion.

[0228]

[0195] The model generates the scale parameter of a Laplacian distribution, a process similar to that performed by entropy coding hyperprior neural networks in end-to-end neural video codecs, and the similarity simplifies the integration of the systems and techniques described herein into systems for training neural networks.

[0229]

[0196] Experimental results comparing frame bit rates obtained using the systems and techniques described in this specification with accurate values ​​from the HEVC / H.265 HM codec show that the method provides accuracy very similar to non-unique and non-differentiable entropy coding methods, and is much more accurate than methods that use coefficient-by-coefficient estimates.

[0230]

[0197] By using a mathematical statistical model, it is possible to derive formulas defining all the necessary derivatives, which can be calculated more efficiently directly using C++ or CUDA GPU implementations instead of automatic differentiation.

[0231]

[0198] Figure 17 is a flow diagram illustrating a technique for performing bit rate estimation 1700 according to aspects of this disclosure. At operation 1702, the technique 1700 may include encoding one or more frames of video data using a video encoder, where the video encoder includes at least a quantization process such as those shown in Figures 6A-6B and 7. At operation 1704, the technique 1700 may include determining an actual bit rate for the encoded one or more frames, as shown in Figures 6A-6B and 7.

[0232] At operation 1706, the technique 1700 may include predicting an estimated bit rate using an encoder proxy, where the encoder proxy includes a statistical model for estimating the bit rate of one or more encoded frames, as shown in FIG. 8. In some cases, the encoder proxy estimates the output of one or more processes of the video encoder. In some cases, the statistical model is based on a Laplacian distribution of coefficients input to a quantization process of the video encoder. In some cases, the technique 1700 may also include estimating the output of the quantization process based on a maximum-likelihood estimation of the Laplacian distribution.

[0233] At operation 1708, technique 1700 may include determining a gradient of the estimated bit rate using an encoder proxy, as shown in FIG. 8. In some cases, the video encoder may include an entropy coding process after the quantization process, and further include estimating an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder. In some cases, the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process. In some cases, the gradient is determined based on at least a derivative of a statistical model.

[0234]

[0201] In operation 1710, the technique may include training an encoder proxy to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0235] 18 is a flow diagram illustrating a technique for performing bit rate estimation 1800 according to an aspect of the present disclosure. At operation 1802, the technique 1800 may include receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process, such as those shown in FIGS. 6A-6B and 7.

[0236] At operation 1804, technique 1800 may include predicting an estimated bit rate for one or more frames after being encoded by the video encoder using an encoder proxy, the encoder proxy including a statistical model for estimating the estimated bit rate, the statistical model trained based on a gradient of the estimated bit rate, as shown in FIGS. 6B-8 . In some cases, the encoder proxy estimates an output of one or more processes of the video encoder. In some cases, the statistical model is based on a Laplacian distribution of coefficients input to a quantization process of the video encoder. In some cases, technique 1800 may also include estimating an output of the quantization process based on a maximum likelihood estimation of the Laplacian distribution. In some cases, the video encoder may include an entropy coding process after the quantization process, further including estimating an output of the entropy coding process based on an interval around the coefficients input to the quantization process of the video encoder. In some cases, the estimated bit rate is based on an estimated output of the quantization process and the estimated output of the quantization process. In some cases, the gradient is determined based on at least a derivative of the statistical model.

[0237] At operation 1806, the technique 1800 may adjust one or more quality parameters based on the predicted estimated bit rate, as shown in Figure 6A. At operation 1808, the technique 1800 may encode one or more frames of video data using a video encoder, which includes at least a quantization process, as shown in Figures 6A and 7.

[0238] 19 shows an example computing device architecture 1900 of an example computing device that may implement various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other device. For example, the computing device architecture 1900 may be used as part of the system 200 of FIG. 2 and / or the system 300 of FIG. 3. The components of the computing device architecture 1900 are shown in electrical communication with each other using a connection 1905, such as a bus. The exemplary computing device architecture 1900 includes a processing unit (CPU or processor) 1910 and computing device connections 1905 that couple various computing device components to the processor 1910, including computing device memory 1915, such as read-only memory (ROM) 1920 and random access memory (RAM) 1925.

[0239] The computing device architecture 1900 may include a cache of high-speed memory directly connected to, adjacent to, or integrated as part of the processor 1910. The computing device architecture 1900 may copy data from the memory 1915 and / or the storage device 1930 to the cache 1912 for quick access by the processor 1910. In this manner, the cache may provide performance improvements that avoid delays to the processor 1910 while waiting for data. These and other modules may control or be configured to control the processor 1910 to perform various actions. Other computing device memory 1915 may also be available for use. The memory 1915 may include multiple different types of memory with different performance characteristics. The processor 1910 may include any general-purpose processor and hardware, or software services, such as service 1 1932, service 2 1934, and service 3 1936 stored in the storage device 1930, configured to control the processor 1910, as well as special-purpose processors in which software instructions are incorporated into the processor design. The processor 1910 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0240] To enable user interaction with computing device architecture 1900, input device(s) 1945 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Output device(s) 1935 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with computing device architecture 1900. Communications interface 1940 can generally govern and manage user input and computing device output. There is no constraint to operating on any particular hardware configuration, and therefore, as improved hardware or firmware configurations are developed, the basic functionality herein can be easily replaced with them.

[0241] The storage device 1930 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, random access memories (RAMs) 1925, a read-only memory (ROM) 1920, and hybrids thereof. The storage device 1930 may include services 1932, 1934, 1936 for controlling the processor 1910. Other hardware or software modules are contemplated. The storage device 1930 may be connected to a computing device connection 1905. In one aspect, a hardware module that performs a specific function may include software components stored on a computer-readable medium in association with the necessary hardware components, such as the processor 1910, the connection 1905, and the output device 1935, to perform that function.

[0242] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop computer, vehicle, drone, or other device) that includes or is coupled to one or more active depth-sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0243] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, or a processing system). As used herein, a device may be any electronic device having one or more parts capable of implementing at least some portions of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or a specific embodiment. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. The following description and examples use the term "system" to describe various aspects of the present disclosure, but the term "system" is not limited to a specific configuration, type, or number of objects.

[0244]

[0211] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks, including devices, device components, method steps or routines embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0245]

[0212] Particular embodiments may be described above as a process or method that is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0246]

[0213] The processes and methods according to the above-described embodiments can be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions, or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. A portion of the computer resources used can be accessible over a network. The computer-executable instructions can be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc.

[0247] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may also include non-transitory media that can store data and do not involve carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as flash memory, memory or memory devices, magnetic or optical disks, flash memory, USB devices implemented with non-volatile memory, networked storage devices, compact disks (CDs) or digital versatile disks (DVDs), among others. Code and / or machine-executable instructions may be stored on a computer-readable medium, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0248] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0249] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) to perform the necessary tasks can be stored in a computer-readable or machine-readable medium. A processor can perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small footprint personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein can also be embodied in peripheral devices or add-in cards. Such functionality can also be implemented among various chips on a circuit board or among various processes running within a single device, as further examples.

[0250]

[0217] The instructions, media for communicating such instructions, computing resources for executing those instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0251]

[0218] In the foregoing description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, while exemplary embodiments of the present application have been described in detail herein, it should be understood that, except as limited by the prior art, the concepts of the present application may be variously embodied and employed in other manners, and the appended claims are intended to be construed to include such variations. The various features and aspects of the present application described above may be used individually or in combination. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the present specification and drawings should be considered illustrative and not limiting. For illustrative purposes, methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0252]

[0219] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used in this specification can be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this description.

[0253]

[0220] Where a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0254]

[0221] The phrase "coupled to" refers to any component that is physically connected to another component, either directly or indirectly, and / or that is in communication with another component, either directly or indirectly (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0255]

[0222] Claim language or other language referring to "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language of "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include items not recited within the set of A and B.

[0256]

[0223] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0257] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may also form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. These technologies may also, or alternatively, be implemented at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0258] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor. Alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the above-described structures, any combination of the above-described structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0259]

[0226] Exemplary aspects of the present disclosure include:

[0227] Aspect 1: A method for processing video data. The method includes: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate for the encoded one or more frames; predicting an estimated bit rate using an encoder proxy, the encoder proxy including a statistical model for estimating a bit rate for the encoded one or more frames; determining a gradient of the estimated bit rate using the encoder proxy; and training the encoder proxy to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0260]

[0228] Aspect 2. The method of claim 1, wherein the encoder proxy estimates the output of one or more processes of the video encoder.

[0261]

[0229] Aspect 3. The method of claim 1 or 2, wherein the statistical model is based on a Laplacian distribution of coefficients input to a quantization process of the video encoder.

[0262]

[0230] Aspect 4. The method of claim 3, further comprising estimating the output of the quantization process based on a maximum likelihood estimation of the Laplace distribution.

[0263]

[0231] Aspect 5. A method according to any one of claims 1 to 4, wherein the video encoder further includes an entropy coding process after the quantization process, and further includes estimating the output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

[0264]

[0232] Aspect 6. The method of claim 5, wherein the estimated bit rate is based on an estimated output of a quantization process and an estimated output of a quantization process.

[0265]

[0233] Aspect 7. The method of claim 1, wherein the gradient is determined based on at least a derivative of the statistical model.

[0266]

[0234] Aspect 8. An apparatus for processing video data, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: encode one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determine an actual bitrate for the encoded one or more frames; predict an estimated bitrate using an encoder proxy, the encoder proxy including a statistical model for estimating the bitrate for the encoded one or more frames; determine a gradient of the estimated bitrate using the encoder proxy; and train the encoder proxy to predict the estimated bitrate based on the actual bitrate, the estimated bitrate, and the gradient.

[0267]

[0235] Aspect 9. The apparatus of claim 8, wherein the encoder proxy estimates the output of one or more processes of the video encoder.

[0268]

[0236] Aspect 10. The apparatus of claim 8 or 9, wherein the statistical model is based on a Laplacian distribution of coefficients input to a quantization process of a video encoder.

[0269] Aspect 11. The apparatus of claim 10, wherein the at least one processor is further configured to estimate the output of the quantization process based on a maximum likelihood estimation of the Laplace distribution.

[0270]

[0238] Aspect 12. An apparatus as described in any one of claims 8 to 11, wherein the video encoder further includes an entropy coding process after the quantization process, and the processor is further configured to estimate the output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

[0271]

[0239] Aspect 13. The apparatus of claim 12, wherein the estimated bit rate is based on an estimated output of a quantization process and an estimated output of a quantization process.

[0272]

[0240] Aspect 14. The apparatus of claim 8, wherein the gradient is determined based on at least a derivative of a statistical model.

[0273]

[0241] Aspect 15. A method for processing video data, comprising: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting an estimated bit rate for the one or more frames after encoding by the video encoder using an encoder proxy, the encoder proxy including a statistical model for estimating an estimated bit rate, the statistical model being trained based on a gradient of the estimated bit rate; adjusting one or more quality parameters based on the predicted estimated bit rate; and encoding the one or more frames of video data using the video encoder.

[0274]

[0242] Aspect 16. The method of claim 15, wherein the encoder proxy estimates the output of one or more processes of the video encoder.

[0275]

[0243] Aspect 17. A method according to claim 15 or 16, wherein the statistical model is based on a Laplacian distribution of coefficients input to a quantization process of the video encoder.

[0276]

[0244] Aspect 18. The method of claim 17, further comprising estimating the output of the quantization process based on a maximum likelihood estimation of the Laplace distribution.

[0277]

[0245] Aspect 19. A method according to any one of claims 15 to 18, wherein the video encoder further includes an entropy coding process after the quantization process, and the method further includes estimating the output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

[0278]

[0246] Aspect 20. The method of claim 19, wherein the estimated bit rate is based on an estimated output of a quantization process and an estimated output of a quantization process.

[0279]

[0247] Aspect 21. The method of claim 15, wherein the gradient is determined based on at least a derivative of a statistical model.

[0280]

[0248] Aspect 22. An apparatus for processing video data, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: receive one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; an encoder proxy, the encoder proxy including a statistical model for estimating an estimated bitrate, the statistical model being trained based on a gradient of the estimated bitrate; predict an estimated bitrate for the one or more frames after being encoded by the video encoder using the encoder proxy; adjust one or more quality parameters based on the predicted estimated bitrate; and encode the one or more frames of video data using the video encoder.

[0281]

[0249] Aspect 23. The apparatus of claim 22, wherein the encoder proxy estimates the output of one or more processes of the video encoder.

[0282]

[0250] Aspect 24. An apparatus as described in claim 22 or 23, wherein the statistical model is based on a Laplace distribution of coefficients input to a quantization process of a video encoder.

[0283]

[0251] Aspect 25. The apparatus of claim 24, wherein the at least one processor is further configured to estimate the output of the quantization process based on a maximum likelihood estimation of the Laplace distribution.

[0284]

[0252] Aspect 26. An apparatus as described in any one of claims 22 to 25, wherein the video encoder further includes an entropy coding process after the quantization process, and the processor is further configured to estimate the output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

[0285]

[0253] Aspect 27. The apparatus of claim 26, wherein the estimated bit rate is based on an estimated output of a quantization process and an estimated output of a quantization process.

[0286]

[0254] Aspect 28. The apparatus of claim 22, wherein the gradient is determined based on at least a derivative of a statistical model.

[0287]

[0255] Aspect 29. A method for processing video data, comprising: receiving one or more frames of video data for encoding by a video encoder; predicting an estimated bitrate of the one or more frames after encoding by the video encoder using an encoder proxy, the encoder proxy including a statistical model for estimating the estimated bitrate; determining a gradient of the estimated bitrate using the encoder proxy; encoding the one or more frames of video data using a video encoder, the video encoder including at least a quantization process; obtaining an actual bitrate of the encoded one or more frames; and updating the encoder proxy based on a comparison between the estimated bitrate and the actual bitrate.

[0288]

[0256] Aspect 30. The method of claim 29, wherein the encoder proxy estimates the output of one or more processes of the video encoder.

[0289]

[0257] Aspect 31. A method according to claim 29 or 30, wherein the statistical model is based on a Laplacian distribution of coefficients input to a quantization process of a video encoder.

[0290]

[0258] Aspect 32. The method of claim 31, further comprising estimating the output of the quantization process based on a maximum likelihood estimation of the Laplace distribution.

[0291]

[0259] Aspect 33. A method according to any one of claims 29 to 32, wherein the video encoder further includes an entropy coding process after the quantization process, and the method further includes estimating the output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

[0292]

[0260] Aspect 34. The method of claim 33, wherein the estimated bit rate is based on an estimated output of a quantization process and an estimated output of a quantization process.

[0293]

[0261] Aspect 35. The method of claim 29, wherein the gradient is determined based on at least a derivative of a statistical model.

[0294]

[0262] Aspect 36. An apparatus for processing video data, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor comprising: receiving one or more frames of video data for encoding by a video encoder; predicting an estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, the encoder proxy including a statistical model for estimating the estimated bitrate; determining a gradient of the estimated bitrate using the encoder proxy; encoding the one or more frames of video data using the video encoder, the video encoder including at least a quantization process; obtaining an actual bitrate of the encoded one or more frames; and updating the encoder proxy based on a comparison between the estimated bitrate and the actual bitrate.

[0295]

[0263] Aspect 37. The apparatus of claim 36, wherein the encoder proxy estimates the output of one or more processes of the video encoder.

[0296]

[0264] Aspect 38. An apparatus as described in claim 36 or 37, wherein the statistical model is based on a Laplace distribution of coefficients input to a quantization process of a video encoder.

[0297]

[0265] Aspect 39. The apparatus of claim 38, wherein the at least one processor is further configured to estimate the output of the quantization process based on a maximum likelihood estimation of the Laplace distribution.

[0298]

[0266] Aspect 40. An apparatus described in any one of claims 36 to 39, wherein the video encoder further includes an entropy coding process after the quantization process, and wherein at least one processor is further configured to estimate the output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

[0299]

[0267] Aspect 41. The apparatus of claim 39, wherein the estimated bit rate is based on an estimated output of a quantization process and an estimated output of a quantization process.

[0300]

[0268] Aspect 42. The apparatus of claim 36, wherein the gradient is determined based on at least a derivative of a statistical model.

[0301]

[0269] Aspect 43. The apparatus of any one of aspects 8 to 14, 22 to 28, and 37 to 42, wherein the apparatus includes an encoder.

[0302]

[0270] Aspect 44. The device of any one of aspects 8-14, 22-28, and 37-43, further comprising a display configured to display one or more output pictures.

[0303]

[0271] Aspect 45. The apparatus of any one of aspects 8-14, 22-28, and 37-44, further comprising a camera configured to capture one or more pictures.

[0304]

[0272] Aspect 46. A device described in any one of aspects 8 to 14, 22 to 28, and 37 to 45, wherein the device is a mobile device.

[0305] Aspect 47. An apparatus for processing video data, comprising means for performing one or more of the operations according to any of aspects 8-14, 22-28, and 37-46.

[0306]

[0274] Aspect 48. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to any one or more of aspects 1-7, 15-21, and / or 29-35.

[0307]

[0275] Aspect 49. An apparatus comprising one or more means for performing the operations according to any one or more of aspects 1-7, 15-21, and / or 29-35. The inventions described in the claims of the present application as originally filed are set forth below. [C1] A method for processing video data, comprising: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate for the encoded one or more frames; and predicting an estimated bit rate using an encoder proxy, the encoder proxy including a statistical model for estimating a bit rate for the encoded one or more frames; using the encoder proxy to determine a gradient of the estimated bit rate; and training the encoder proxy to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient. [C2] The method of C1, wherein the encoder proxy estimates an output of one or more processes of the video encoder. [C3] The method of C1, wherein the statistical model is based on a Laplacian distribution of coefficients input to the quantization process of the video encoder. [C4] The method of C3, further comprising estimating the output of the quantization process based on a maximum likelihood estimate of the Laplacian distribution. [C5] The method of C1, wherein the video encoder further includes an entropy coding process after the quantization process, and further includes estimating an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder. [C6] The method of C5, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process. [C7] The method of C1, wherein the gradient is determined based on at least a derivative of the statistical model. [C8] An apparatus for processing video data, comprising: at least one memory; at least one processor coupled to the at least one memory, the at least one processor comprising: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate for the encoded one or more frames; and predicting an estimated bit rate using an encoder proxy, the encoder proxy including a statistical model for estimating a bit rate for the encoded one or more frames; using the encoder proxy to determine a gradient of the estimated bit rate; and training the encoder proxy to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient. [C9] The apparatus of C8, wherein the encoder proxy estimates an output of one or more processes of the video encoder. [C10] The apparatus of C8, wherein the statistical model is based on a Laplace distribution of coefficients input to the quantization process of the video encoder. [C11] The apparatus of C10, wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimation of the Laplace distribution. [C12] The apparatus of C8, wherein the video encoder further includes an entropy coding process after the quantization process, and wherein the at least one processor is further configured to estimate an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder. [C13] The apparatus of C12, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process. [C14] The apparatus of C8, wherein the gradient is determined based on at least a derivative of the statistical model. [C15] A method for processing video data, comprising: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting the estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, the encoder proxy using a statistical model to estimate an estimated bitrate, the statistical model being trained based on a gradient of the estimated bitrate; adjusting one or more quality parameters of the video encoder based on the predicted estimated bitrate; encoding the one or more frames of the video data using the video encoder. [C16] The method of C15, wherein the encoder proxy estimates an output of one or more processes of the video encoder. [C17] The method of C15, wherein the statistical model is based on a Laplacian distribution of coefficients input to the quantization process of the video encoder. [C18] The method of C17, further comprising estimating the output of the quantization process based on a maximum likelihood estimate of the Laplace distribution. [C19] The method of C15, wherein the video encoder further includes an entropy coding process after the quantization process, and the method further includes estimating an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder. [C20] The method of C19, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process. [C21] The method of C15, wherein the gradient is determined based on at least a derivative of the statistical model. [C22] An apparatus for processing video data, comprising: at least one memory; at least one processor coupled to the at least one memory, the at least one processor comprising: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting the estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, the encoder proxy using a statistical model to estimate an estimated bitrate, the statistical model being trained based on a gradient of the estimated bitrate; adjusting one or more quality parameters of the video encoder based on the predicted estimated bitrate; encoding the one or more frames of the video data using the video encoder. [C23] The apparatus of C22, wherein the encoder proxy estimates an output of one or more processes of the video encoder. [C24] The apparatus of C22, wherein the statistical model is based on a Laplace distribution of coefficients input to the quantization process of the video encoder. [C25] The apparatus of C24, wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimation of the Laplace distribution. [C26] The apparatus of C22, wherein the video encoder further includes an entropy coding process after the quantization process, and wherein the at least one processor is configured to estimate an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder. [C27] The apparatus of C26, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process. [C28] The apparatus of C22, wherein the gradient is determined based on at least a derivative of the statistical model. [C29] The apparatus of C22, wherein the apparatus is the video encoder. [C30] The apparatus of C22, wherein the apparatus includes the video encoder.

Claims

1. 1. A method for processing video data, comprising: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate for the encoded frame or frames; predicting an estimated bitrate using an encoder proxy, the encoder proxy including a statistical model for estimating a bitrate of the encoded one or more frames; determining a gradient of the estimated bitrate using the encoder proxy; training the encoder proxy to predict the estimated bitrate based on the actual bitrate, the estimated bitrate, and the gradient.

2. The method of claim 1 , wherein the encoder proxy estimates an output of one or more processes of the video encoder.

3. The method of claim 1 , wherein the statistical model is based on a Laplacian distribution of coefficients input to the quantization process of the video encoder.

4. The method of claim 3 , further comprising estimating the output of the quantization process based on a maximum likelihood estimate of the Laplacian distribution.

5. 10. The method of claim 1 , wherein the video encoder further includes an entropy coding process after the quantization process, and further includes estimating an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

6. The method of claim 5 , wherein the estimated bit rate is based on an estimated output of the quantization process.

7. The method of claim 1 , wherein the gradient is determined based on at least a derivative of the statistical model.

8. 1. An apparatus for processing video data, comprising: at least one memory; at least one processor coupled to the at least one memory, the at least one processor: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate for the encoded frame or frames; predicting an estimated bitrate using an encoder proxy, the encoder proxy including a statistical model for estimating a bitrate of the encoded one or more frames; determining a gradient of the estimated bitrate using the encoder proxy; training the encoder proxy to predict the estimated bitrate based on the actual bitrate, the estimated bitrate, and the gradient.

9. The apparatus of claim 8 , wherein the encoder proxy estimates an output of one or more processes of the video encoder.

10. The apparatus of claim 8 , wherein the statistical model is based on a Laplacian distribution of coefficients input to the quantization process of the video encoder.

11. The apparatus of claim 10 , wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimation of the Laplacian distribution.

12. 9. The apparatus of claim 8, wherein the video encoder further includes an entropy coding process after the quantization process, and the at least one processor is further configured to estimate an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

13. The apparatus of claim 12 , wherein the estimated bit rate is based on an estimated output of the quantization process.

14. The apparatus of claim 8 , wherein the gradient is determined based on at least a derivative of the statistical model.

15. 1. A method for processing video data, comprising: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting the estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, the encoder proxy using a statistical model to estimate an estimated bitrate, the statistical model being trained based on a gradient of the estimated bitrate; adjusting one or more quality parameters of the video encoder based on the predicted estimated bitrate; encoding the one or more frames of the video data using the video encoder.

16. The method of claim 15 , wherein the encoder proxy estimates an output of one or more processes of the video encoder.

17. The method of claim 15 , wherein the statistical model is based on a Laplacian distribution of coefficients input to the quantization process of the video encoder.

18. The method of claim 17 , further comprising estimating the output of the quantization process based on a maximum likelihood estimate of the Laplacian distribution.

19. 16. The method of claim 15, wherein the video encoder further includes an entropy coding process after the quantization process, the method further including estimating an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

20. The method of claim 19, wherein the estimated bit rate is based on an estimated output of the quantization process.

21. The method of claim 15 , wherein the gradient is determined based on at least a derivative of the statistical model.

22. 1. An apparatus for processing video data, comprising: at least one memory; at least one processor coupled to the at least one memory, the at least one processor: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting the estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, the encoder proxy using a statistical model to estimate an estimated bitrate, the statistical model being trained based on a gradient of the estimated bitrate; adjusting one or more quality parameters of the video encoder based on the predicted estimated bitrate; encoding the one or more frames of the video data using the video encoder.

23. The apparatus of claim 22 , wherein the encoder proxy estimates an output of one or more processes of the video encoder.

24. 23. The apparatus of claim 22, wherein the statistical model is based on a Laplacian distribution of coefficients input to the quantization process of the video encoder.

25. 25. The apparatus of claim 24, wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimate of the Laplacian distribution.

26. 23. The apparatus of claim 22, wherein the video encoder further includes an entropy coding process after the quantization process, and wherein the at least one processor is configured to estimate an output of the entropy coding process based on intervals around coefficients input to the quantization process of the video encoder.

27. 27. The apparatus of claim 26, wherein the estimated bit rate is based on an estimated output of the quantization process.

28. The apparatus of claim 22 , wherein the gradient is determined based on at least a derivative of the statistical model.

29. The device of claim 22 , wherein the device is the video encoder.

30. The apparatus of claim 22 , wherein the apparatus comprises the video encoder.

Images