Bit rate estimation for video coding with machine learning enhancement

Abstract

Techniques for processing video data are described herein. For example, a technique may include encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate for the encoded one or more frames; predicting the estimated bitrate using an encoder agent, the encoder agent comprising a statistical model for estimating the bitrate of the encoded one or more frames; determining a gradient of the estimated bitrate using an encoder agent; and training an encoder agent to predict the estimated bitrate based on the actual bitrate, the estimated bitrate, and the gradient.

Description

Technical Field

[0001] In general, the present disclosure relates to video coding (eg, encoding and / or decoding video data). For example, aspects of the present disclosure relate to systems and techniques for performing bitrate estimation to enhance video coding using machine learning. Background Art

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes a large amount of data used to meet the needs of consumers and video providers. For example, consumers of video data expect high-quality video including high fidelity, high resolution, high frame rate, etc. As a result, the large amount of video data required to meet these needs places a burden on the communication networks and devices that process and store video data.

[0003] Video decoding techniques can be used to compress video data. One goal of video decoding is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As evolving video services become available, coding techniques with better decoding efficiency are needed. Summary of the invention

[0004] A brief summary of one or more aspects disclosed herein is given below. Therefore, the following summary should not be regarded as an extensive review of all expected aspects, nor should the following summary be regarded as identifying key or important elements related to all expected aspects or delineating the scope associated with any particular aspect. Therefore, the following summary gives certain concepts related to one or more aspects involving the mechanisms disclosed herein in a brief form to precede the specific embodiments given below.

[0005] Systems and techniques for transcoding (e.g., encoding and / or decoding) images and / or video content are described. In one illustrative example, a method of processing video data is provided. The method includes: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate of the one or more frames encoded; predicting an estimated bit rate using an encoder agent, the encoder agent including a statistical model for estimating the bit rate of the one or more frames encoded; determining a gradient of the estimated bit rate using the encoder agent; and training the encoder agent to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0006] In another example, a device for processing video data is provided. The device includes: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: encode one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determine an actual bit rate of the one or more frames encoded; predict an estimated bit rate using an encoder agent, the encoder agent including a statistical model for estimating the bit rate of the one or more frames encoded; determine a gradient of the estimated bit rate using the encoder agent; and train the encoder agent to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0007] In another example, a non-transitory computer-readable storage medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to perform the following operations: receive one or more frames of video data for encoding by a video encoder, the video encoder comprising at least a quantization process; predict an estimated bit rate of the one or more frames after being encoded by the video encoder using an encoder agent, wherein the encoder agent comprises a statistical model for estimating the estimated bit rate, and wherein the statistical model is trained based on a gradient of the estimated bit rate; adjust one or more quality parameters based on the predicted estimated bit rate; and encode the one or more frames of video data using the video encoder, the video encoder comprising at least a quantization process.

[0008] In another example, an apparatus is provided. The apparatus includes: a unit for receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; a unit for predicting an estimated bit rate of the one or more frames after being encoded by the video encoder using an encoder proxy, wherein the encoder proxy includes a statistical model for estimating the estimated bit rate, and wherein the statistical model is trained based on a gradient of the estimated bit rate; a unit for adjusting one or more quality parameters based on the predicted estimated bit rate; and a unit for encoding the one or more frames of video data using the video encoder, the video encoder including at least a quantization process.

[0009] In some aspects, the apparatus includes a mobile device (e.g., a mobile phone or so-called "smart phone", a tablet computer or other type of mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television (e.g., a networked television), a vehicle (or a computing device of a vehicle), or other devices. In some aspects, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more images and / or one or more videos including video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to send one or more video frames and / or grammatical data to at least one device on a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and embodiments will become more fully apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative embodiments of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1 Example implementations of a system on a chip (SOC) according to some examples are shown.

[0014] Figure 2 is a block diagram illustrating an encoding device and a decoding device according to some examples.

[0015] Figure 3 is a diagram illustrating an example of a system including a device operable to perform image and / or video transcoding (encoding and decoding) using a machine learning transcoding system according to some examples;

[0016] Figure 4A-4D is a diagram illustrating an example of a neural network according to some examples;

[0017] Figure 5 is a diagram illustrating an example of a deep convolutional network according to some examples;

[0018] Fig. 6A and Figure 6B is a block diagram illustrating an example implementation of a video codec neural lifting system according to some examples;

[0019] Figure 7 is a diagram illustrating encoder elements of an example video encoder according to some examples;

[0020] Figure 8 is a block diagram illustrating elements of an example differential encoder agent according to some examples.

[0021] Fig. 9 are graphs of example adjustment functions and derivatives of the adjustment functions plotted according to some examples;

[0022] Figure 10-Figure 16 is a histogram of example ratios between frame bit rates according to some examples;

[0023] Fig.17 is a flow chart illustrating a technique for performing bit rate estimation according to aspects of the present disclosure;

[0024] Fig.18 is a flow chart illustrating a technique for performing bit rate estimation according to aspects of the present disclosure;

[0025] Fig.19 is an example computing device that can implement the various techniques described herein. DETAILED DESCRIPTION

[0026] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments and embodiments can be applied independently, and some of them and embodiments can be applied in combination, as will be apparent to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent that various embodiments can be implemented without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0027] The following description provides only example embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the following description of the example embodiments will provide a description of implementations for implementing the example embodiments to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0028] Digital video data can include large amounts of data, particularly as the demand for high-quality video data continues to grow. For example, consumers of video data generally expect increasingly higher quality video with high fidelity, high resolution, high frame rate, etc. However, the large amounts of video data required to meet such demands can place a heavy burden on communication networks and on devices that process and store video data.

[0029] Various techniques can be used to decode video data. Video decoding can be performed according to a specific video decoding standard or can be performed using one or more machine learning systems or algorithms. Example video decoding standards include universal video coding (VVC), high-efficiency video coding (HEVC), advanced video coding (AVC), moving picture experts group (MPEG) coding (e.g., MPEG-5 basic video coding (EVC) or other MPEG-based coding), AOMedia video 1 (AV1), etc. Video decoding generally uses a prediction method (e.g., inter-prediction or intra-prediction) that utilizes redundancy present in a video image or sequence. A common goal of video decoding technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As the demand for video services grows and new video services become available, decoding technologies with better decoding efficiency, performance, and rate control are needed.

[0030] Video decoding equipment implements video compression technology to efficiently encode and decode video data. Video compression technology can include applying different prediction modes to reduce or eliminate redundancy inherent in video sequences, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques. The video encoder can divide each picture of the original video sequence into rectangular areas (referred to as video blocks or decoding units (described in more detail below)). These video blocks can be encoded using specific prediction modes.

[0031] The video blocks may be divided into one or more groups of smaller blocks in one or more ways. Blocks may include coding tree blocks, prediction blocks, transform blocks, and / or other suitable blocks. Unless otherwise specified, references to "blocks" may generally refer to such video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as will be understood by one of ordinary skill in the art). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., coding tree unit (CTU), decoding unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may indicate a decoding logic unit encoded in a bitstream, and a block may indicate a portion of a video frame buffer for which a process is directed.

[0032] For inter-prediction mode, the video encoder can search for blocks similar to the block being encoded in a frame (or picture) located at another time position, which is called a reference frame or reference picture. The video encoder can limit the search to a certain spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component can be used to locate the best match. For intra-prediction mode, the video encoder can use spatial prediction techniques to form a prediction block based on data from previously encoded neighboring blocks in the same picture.

[0033] The video encoder can determine the prediction error. For example, the prediction can be determined as the difference between the pixel values ​​in the block being encoded and the predicted block. The prediction error can also be referred to as a residual. The video encoder can also use transform coding (for example, using the form of discrete cosine transform (DCT), discrete sine transform (DST) or other suitable transforms) to apply the transform to the prediction error to generate transform coefficients. After the transform, the video encoder can quantize the transform coefficients. The quantized transform coefficients and motion vectors can be represented using syntax elements and form a decoded representation of the video sequence together with the control information. In some cases, the video encoder can entropy decode the syntax elements to further reduce the number of bits required for its representation.

[0034] The video decoder can use the syntax elements and control information discussed above to construct predictive data (e.g., a predictive block) for decoding the current frame. For example, the video decoder can add the prediction block and the compressed prediction error. The video decoder can determine the compressed prediction error by weighting the transform basis function using quantized coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0035] As described in more detail below, systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively, "systems and techniques") for performing bitrate estimation to enhance video decoding using machine learning are described herein. In some devices, a video codec may be implemented using custom hardware, such as an application specific integrated circuit (ASIC). Custom hardware helps achieve high efficiency in terms of processing speed and power usage, but greatly reduces flexibility because additional features require the design and deployment of new hardware, which can be slow and expensive.

[0036] Techniques have been proposed to improve performance by adaptively modifying the video before encoding and after decoding, while taking advantage of the ubiquity of codec hardware. Recent proposals demonstrate the advantages of using machine learning and neural networks for this purpose, as they can use advanced algorithms and extensive training to better identify how video compression can be improved. This approach can be called neural lifting of standard video codecs.

[0037] A fundamental problem with many of these techniques is that neural-based solutions can be designed and optimized more effectively when they can use derivatives of bitrate and distortion performance measures. However, these derivatives may not be available from complex standard video codecs, and therefore they cannot be effectively incorporated into end-to-end system designs, resulting in suboptimal performance.

[0038] The present disclosure uses a new type of codec proxy to at least solve this problem. For example, a codec proxy according to the systems and techniques described herein can effectively and accurately estimate codec performance and partial derivatives (e.g., all partial derivatives) required for optimization. The codec proxy can be integrated with backpropagation techniques (e.g., gradient backpropagation) used by neural network training tools. The experimental results discussed below use comparisons with the bit rate of the HEVC / H.265 video standard to demonstrate the accuracy of the estimate.

[0039] Various aspects of the present disclosure will be described with reference to the accompanying drawings.

[0040] Figure 1 An example implementation of a system on a chip (SOC) 100 is shown, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0041] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104; a DSP 106; a connectivity block 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.; and a multimedia processor 112, which may, for example, detect and recognize gestures. In one embodiment, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0042] SOC 100 can be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into CPU 102 may include code for searching a lookup table (LUT) for a stored multiplication result corresponding to the product of the input value and the filter weight. The instructions loaded into CPU 102 may also include code for disabling a multiplier during a multiplication operation of the product when a lookup table hit of the product is detected. In addition, the instructions loaded into CPU 102 may include code for storing the calculated product of the input value and the filter weight when a lookup table miss of the product is detected.

[0043] The SOC 100 and / or its components may be configured to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video coding) using standards-based video coding and / or using machine learning techniques. Figure 2 and Figure 3 Examples of standards-based and machine learning-based video decoding systems are described.

[0044] Figure 2 is a block diagram illustrating an example of a system 200 including an encoding device 204 and a decoding device 212, which can encode and decode video data, respectively, according to examples described herein. In some examples, the encoding device 204 and / or the decoding device 212 can include Figure 1 SOC 100. Encoding device 204 can be part of a source device, and decoding device 212 can be part of a receiving device (sometimes referred to as a client device). In some examples, the source device may also include a decoding device similar to decoding device 212. In some examples, the receiving device may also include an encoding device similar to encoding device 204. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, an Internet Protocol (IP) camera, a server device in a server system including one or more server devices (e.g., a video streaming server system or other suitable server system), a head-mounted display (HMD), a head-up display (HUD), smart glasses (e.g., virtual reality (VR) glasses, augmented reality (AR) glasses or other smart glasses), or any other suitable electronic device.

[0045] Components of system 200 may include and / or may be implemented using electronic circuits or other electronic hardware, which may include SOC 100 and / or one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0046] Although system 200 is shown to include certain components, one of ordinary skill in the art will appreciate that system 200 may include more than Figure 2 For example, in some cases, system 200 may also include one or more memory devices other than storage 208 and storage 218 (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, NPUs, and / or other processing devices) in communication and / or electrically connected to the one or more memory devices, one or more wireless interfaces for performing wireless communications (e.g., including one or more transceivers and baseband processors for each wireless interface), one or more wired interfaces for performing communications via one or more hardwired connections (e.g., serial interfaces such as universal serial bus (USB) inputs, lighting connectors, and / or other wired interfaces), and / or Figure 2 Other components not shown.

[0047] The decoding techniques described herein are applicable to video decoding in a variety of multimedia applications, including streaming video transmission (e.g., over the Internet), television broadcasting or transmission, encoding of digital video for storage on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 200 can support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0048] In some examples, the encoding device 204 (or encoder) can be used to encode video data using a video coding standard or protocol to generate a coded video bitstream. Examples of video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Vision, ITU-T H.262 or ISO / IEC MPEG-2 Vision, ITU-T H.263, ISO / IEC MPEG-4 Vision, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its scalable video coding (SVC) and multi-view video coding (MVC) extensions, high efficiency video coding (HEVC) or ITU-T H.265, universal video coding (VVC) or ITU-TH.266, and / or other video coding standards. One or more video coding standards have extensions associated with other aspects of video coding. For example, various extensions of HEVC handle multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC), and multi-view extensions (MV-HEVC), and scalable extensions (SHVC).

[0049] Many of the embodiments described herein may be performed using video codecs such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, their extensions, or other suitable coding standards that are already available or not yet available or developed, such as the machine learning-based video coding described below. Therefore, although the techniques and systems described herein may be described with reference to a specific video coding standard, a person of ordinary skill in the art will understand that the description should not be interpreted as being applicable only to that specific standard.

[0050] Reference Figure 2 , video source 202 may provide video data to encoding device 204. Video source 202 may be part of a source device, or may be part of a device other than a source device. Video source 202 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider that provides video data, a video feed interface that receives video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0051] The video data from the video source 202 may include one or more input pictures. A picture may also be referred to as a "frame". A picture or frame is a still image, which in some cases is part of a video. In some examples, the data from the video source 202 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays (denoted as SL, SCb, and SCr). SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chroma samples, and SCr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to as "chroma" samples in this article. In other cases, a picture may be monochrome and may include only an array of luma samples.

[0052] The encoder engine 206 (or encoder) of the encoding device 204 encodes the video data to generate a coded video bitstream. In some examples, the coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. According to HEVC, a decoded video sequence (CVS) includes a series of access units (AUs) starting with an AU having a random access point picture in the base layer and having certain attributes (e.g., a RASL flag (e.g., NoRaslOutputFlag) equal to 1), until but not including the next AU having a random access point picture in the base layer and having certain attributes. The AU includes one or more decoded pictures and control information corresponding to the decoded pictures sharing the same output time. The decoded slices of the picture are encapsulated in the bitstream level as data units called network abstraction layer (NAL) units. For example, an HEVC video bitstream may include one or more CVSs including NAL units. Each NAL unit in the NAL unit has a NAL unit header. The syntax elements in the NAL unit header use designated bits and are therefore visible to all types of systems and transport layers (eg, transport streams, Real-time Transport (RTP) protocol, file formats, etc.).

[0053] There are two types of NAL units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes decoded picture data that forms a decoded video bitstream. For example, the bit sequence that forms the decoded video bitstream is present in the VCL NAL unit. The VCL NAL unit may include a slice or slice segment (described below) of the decoded picture data, and the non-VCL NAL unit includes control information related to one or more decoded pictures. In some cases, the NAL unit may be referred to as a packet. The HEVC AU includes a VCL NAL unit (which contains decoded picture data) and a non-VCL NAL unit corresponding to the decoded picture data (if any). In addition to other information, the non-VCL NAL unit may also contain parameter sets, which have high-level information related to the encoded video bitstream. For example, the parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of the bitstream may reference a single active PPS, SPS, and / or VPS to allow the decoding device 212 to access information that can be used to decode the slice or other portion of the bitstream.

[0054] A NAL unit may contain a sequence of bits (e.g., a coded video bitstream, a CVS of a bitstream, etc.) that form a coded representation of video data (e.g., a coded representation of a picture in a video). The encoder engine 206 generates a coded representation of a picture by dividing each picture into a plurality of slices. A slice is independent of other slices, so information in a slice can be decoded without having to rely on data from other slices in the same picture. A slice includes one or more slice segments (including independent slice segments and one or more dependent slice segments that depend on previous slice segments (if any)).

[0055] Then in HEVC, the slice is divided into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples and the syntax of the samples are called coding tree units (CTUs). CTUs can also be called "tree blocks" or "largest coding units" (LCUs). CTUs are the basic processing units for HEVC encoding. CTUs can be divided into multiple coding units (CUs) of different sizes. A CU contains arrays of luma and chroma samples, which are called coding blocks (CBs).

[0056] Luma and chroma CBs can be further divided into prediction blocks (PBs). A PB is a block of samples of a luminance component or a chroma component that uses the same motion parameters for inter-prediction or intra-block copy (IBC) prediction (when available or enabled for use). The luminance PB and one or more chroma PBs and associated syntax together form a prediction unit (PU). For inter-prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) are signaled in the bitstream for each PU and are used for inter-prediction of the luminance PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. The CB may also be divided into one or more transform blocks (TBs). A TB represents a square block of samples of a color component, on which a residual transform (e.g., the same two-dimensional transform in some cases) is applied to decode the prediction residual signal. A transform unit (TU) represents a TB of luminance and chroma samples, and corresponding syntax elements. Transform decoding is described in more detail below.

[0057] The size of the CU corresponds to the size of the decoding mode, and the shape can be square. For example, the size of the CU can be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other appropriate size, up to the size of the corresponding CTU. The phrase "N×N" is used herein to refer to the pixel size of the video block in terms of vertical and horizontal dimensions (e.g., 8 pixels×8 pixels). The pixels in the block can be arranged in rows and columns. In some embodiments, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. The syntax data associated with the CU can describe, for example, the division of the CU into one or more PUs. The division mode may be different between whether the CU is encoded in an intra-prediction mode or an inter-prediction mode. The PU can be divided into a non-square shape. The syntax data associated with the CU can also describe, for example, the division of the CU into one or more TUs according to the CTU. The shape of the TU can be square or non-square.

[0058] According to HEVC, a transform unit (TU) may be used to perform a transform. TUs may be different for different CUs. The size of a TU may be determined based on the size of a PU within a given CU. The size of a TU may be the same as or smaller than a PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT may correspond to TUs. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may be quantized by the encoder engine 206.

[0059] Once the picture of the video data is divided into CUs, the encoder engine 206 predicts each PU using a prediction mode. The prediction unit or prediction block is subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled inside the bitstream using syntax data. The prediction mode may include intra prediction (or intra-picture prediction) or inter prediction (or inter-picture prediction). Intra prediction exploits the correlation between spatially adjacent samples within a picture. For example, in the case of intra prediction, each PU is predicted from adjacent image data in the same picture using, for example, DC prediction to find the average value of the PU, plane prediction to fit the plane surface to the PU, directional prediction to extrapolate from adjacent data, or any other suitable type of prediction. Inter prediction uses the temporal correlation between pictures to derive motion compensated predictions for blocks of image samples. For example, in the case of inter prediction, motion compensated prediction is used to predict each PU from image data in one or more reference pictures (before or after the current picture in output order). For example, a decision can be made at the CU level whether to decode a picture area using inter-picture prediction or intra-picture prediction.

[0060] As described above, in some cases, the encoder engine 206 and the decoder engine 216 (described in more detail below) can be configured to operate according to VVC. According to VVC, the video decoder (e.g., the encoder engine 206 and / or the decoder engine 216) divides the picture into multiple coding tree units (CTUs) (wherein the CTB of the luminance sample and one or more CTBs of the chrominance sample, as well as the syntax of the sample, are referred to as CTUs). The video decoder can divide the CTU according to a tree structure (e.g., a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure eliminates the concept of multiple partition types, such as the separation between CU, PU, ​​and TU in HEVC. The QTBT structure includes two levels, which include a first level divided according to quadtree partitions, and a second level divided according to binary tree partitions. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the coding units (CUs).

[0061] In the MTT partition structure, blocks can be partitioned using quadtree partitioning, binary tree partitioning, and one or more types of ternary tree partitioning. Ternary tree partitioning is a partitioning that divides a block into three sub-blocks. In some examples, ternary tree partitioning divides a block into three sub-blocks without dividing the original block through the center. The partition types in MTT (e.g., quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0062] In some examples, the video coder may use a single QTBT or MTT structure to represent each of the luma component and the chroma components, while in other examples, the video coder may use two or more QTBT or MTT structures, e.g., one QTBT or MTT structure for the luma component and another QTBT or MTT structure for the two chroma components (or two QTBT and / or MTT structures for corresponding chroma components).

[0063] The video decoder can be configured to use quadtree partitioning for each HEVC, QTBT partitioning, MTT partitioning, or other partitioning structures. For illustrative purposes, the description herein may refer to QTBT partitioning. However, it should be understood that the technology of the present disclosure can also be applied to a video decoder configured to use quadtree partitioning or other types of partitioning.

[0064] As described above, intra-picture prediction exploits the correlation between spatially adjacent samples within a picture. There are multiple intra-prediction modes (also referred to as "intra-modes"). In some examples, intra-prediction of a luminance block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-prediction mode and an angular mode adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction are indexed as shown in Table 1 below. In other examples, more intra-modes may be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may be different from those used in HEVC.

[0065] Internal prediction mode Association Name 0 INTRA_PLANAR 1 INTRA_DC 2..34 INTRA_ANGULAR2..INTRA_ANGULAR34

[0066] Table 1 – Specification of internal prediction modes and associated names

[0067] Inter-picture prediction uses the temporal correlation between pictures to derive motion compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies a horizontal displacement and Δy specifies a vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) may have integer sample precision (also referred to as integer precision), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may have fractional sample precision (also referred to as fractional pixel precision or non-integer precision) to more accurately capture the motion of the underlying object without being limited by the integer pixel grid of the reference frame. The accuracy of the motion vector may be represented by the quantization level of the motion vector. For example, the quantization level may be integer precision (e.g., 1 pixel) or fractional pixel precision (e.g., 1 / 4 pixel, 1 / 2 pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference picture to derive a prediction signal. For example, samples available at integer positions may be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. A previously decoded reference picture is indicated by a reference index (refIdx) of a reference picture list. Motion vectors and reference indices may be referred to as motion parameters. Two types of inter-picture prediction may be performed, including unidirectional prediction and bidirectional prediction.

[0068] In the case of inter-prediction using bidirectional prediction, two sets of motion parameters (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bidirectional prediction, each prediction block uses two motion compensated prediction signals and generates B prediction units. The two motion compensated predictions are combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. Reference pictures that can be used for bidirectional prediction are stored in two separate lists (denoted as List 0 and List 1). Motion parameters can be derived at the encoder using a motion estimation process.

[0069] In the case of inter prediction using unidirectional prediction, one motion parameter set (Δx0, y0, refIdx0) is used to generate motion compensated prediction from the reference picture. For example, in the case of unidirectional prediction, each prediction block uses at most one motion compensated prediction signal and generates P prediction units.

[0070] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, the horizontal component of the motion vector (Δx), the vertical component of the motion vector (Δy), the resolution of the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), the reference picture to which the motion vector points, a reference index, a reference picture list of the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0071] After performing prediction using intra prediction and / or inter prediction, the encoding device 204 may perform transformation and quantization. For example, after prediction, the encoder engine 206 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a decoded current pixel block (PU) and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., using inter prediction or intra prediction), the encoder engine 206 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantizes the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented by a two-dimensional block format (e.g., a two-dimensional matrix or a pixel value array). In such an example, the residual block is a two-dimensional representation of the pixel value.

[0072] Any residual data that may remain after performing the prediction is transformed using a block transform, which may be based on a discrete cosine transform (DCT), a discrete sine transform (DST), an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., kernels of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some examples, a TU may be used for a transform and quantization process implemented by the encoder engine 206. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, a block transform may be used to transform residual values ​​into transform coefficients, and the residual values ​​may be quantized and scanned using the TU to generate serialized transform coefficients for entropy coding.

[0073] In some embodiments, after performing intra-prediction decoding or inter-prediction decoding using the PU of the CU, the encoder engine 206 may calculate the residual data of the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). As described above, the residual data may correspond to the pixel difference between the pixels of the uncoded picture and the prediction value corresponding to the PU. The encoder engine 206 may form one or more TUs including the residual data for the CU (which includes the PU), and may transform the TU to generate transform coefficients for the CU. The TU may include coefficients in the transform domain after applying the block transform.

[0074] The encoder engine 206 can perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization can reduce the bit depth associated with some or all coefficients. In one example, a coefficient with an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0075] Once quantization is performed, the decoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), partition information, and any other suitable data (e.g., other syntax data). Different elements of the decoded video bitstream can be entropy encoded by the encoder engine 206. In some examples, the encoder engine 206 can scan the quantized transform coefficients using a predefined scan order to generate a serialized vector that can be entropy encoded. In some examples, the encoder engine 206 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 206 can entropy encode the vector. For example, the encoder engine 206 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval partitioning entropy coding, or another suitable entropy coding technique.

[0076] The output 210 of the encoding device 204 can send the NAL units constituting the encoded video bitstream data to the decoding device 212 of the receiving device through the communication link 220. The input 214 of the decoding device 212 can receive the NAL unit. The communication link 220 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The wireless network may include any wireless interface or a combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFiTM, radio frequency (RF), UWB, WiFi direct connection, cellular, long-term evolution (LTE), WiMaxTM, etc.). The wired network may include any wired interface (e.g., optical fiber, Ethernet, Ethernet over power line, Ethernet on coaxial cable, digital signal line (DSL), etc.). Various devices such as base stations, routers, access points, bridges, gateways, switches, etc. may be used to implement wired and / or wireless networks. The encoded video bitstream data may be modulated according to a communication standard (e.g., a wireless communication protocol) and sent to a receiving device.

[0077] In some examples, the encoding device 204 may store the encoded video bitstream data in the storage 208. The output 210 may retrieve the encoded video bitstream data from the encoder engine 206 or from the storage 208. The storage 208 may include any of a variety of distributed or locally accessed data storage media. For example, the storage 208 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. The storage 208 may also include a decoded picture buffer (DPB) for storing reference pictures used in inter-prediction. In another example, the storage 208 may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In such cases, the receiving device including the decoding device 212 can access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing the encoded video data and sending the encoded video data to the receiving device. Example file servers include network servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device may access the encoded video data through any standard data connection, including an Internet connection. Access may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from storage 208 may be a streaming transmission, a download transmission, or a combination thereof.

[0078] The input 214 of the decoding device 212 receives the encoded video bitstream data, and the video bitstream data may be provided to the decoder engine 216, or to the storage 218, for later use by the decoder engine 216. For example, the storage 218 may include a DPB for storing reference pictures used in inter-prediction. A receiving device including the decoding device 212 may receive the encoded video data to be decoded via the storage 208. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and sent to the receiving device. The communication medium for sending the encoded video data may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that may help facilitate communication from a source device to a receiving device.

[0079] The decoder engine 216 can decode the coded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more decoded video sequences that constitute the coded video data. The decoder engine 216 can rescale and perform an inverse transform on the coded video bitstream data. The residual data is passed to the prediction stage of the decoder engine 216. The decoder engine 216 predicts a block of pixels (e.g., a PU). In some examples, the prediction is added to the output of the inverse transform (residual data).

[0080] The video decoding device 212 may output the decoded video to a video destination device 222, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 222 may be part of a receiving device that includes the decoding device 212. In some aspects, the video destination device 222 may be part of a separate device other than the receiving device.

[0081] In some embodiments, the video encoding device 204 and / or the video decoding device 212 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 204 and / or the video decoding device 212 may also include other hardware or software necessary to implement the decoding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 204 and the video decoding device 212 may be integrated as part of a combined encoder / decoder (codec) in the respective devices.

[0082] Figure 2The example system shown is an illustrative example that can be used in this article. The technology for processing video data using the technology described in this article can be performed by any digital video encoding and / or decoding device. Although in general, the technology of the present disclosure is performed by a video encoding device or a video decoding device, these technologies can also be performed by a combined video encoder-decoder commonly referred to as a "CODEC". In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are only examples of such decoding devices, wherein the source device generates decoded video data for transmission to the receiving device. In some examples, the source device and the receiving device can be operated in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting or video telephony.

[0083] As described above, in some examples, SOC 100 and / or its components may be configured to use machine learning techniques to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video decoding). For example, encoding device 204 (or encoder) may be configured to use a machine learning system with a deep learning architecture (e.g., by utilizing Figure 1 The NPU 108 of the SOC 100 of the encoding device 204 encodes the video data. In some cases, using a deep learning architecture to perform video compression and / or decompression can improve the efficiency of video compression and / or decompression on the device. For example, the encoding device 204 can use machine learning-based video decoding technology to more efficiently compress the video, and the compressed video can be sent to the decoding device 212, and the decoding device 212 can use machine learning-based technology to decompress the compressed video.

[0084] A neural network is an example of a machine learning system, and a neural network may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network may include feature maps or activation maps, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. A node may include one or more weights indicating the importance of a node in one or more layers in a layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers used to determine simple and low-level features of the input, and later layers establishing hierarchies of more complex and abstract features.

[0085] Deep learning architectures can learn hierarchies of features. For example, if visual data is presented, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if auditory data is presented, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Higher layers can also learn to recognize common visual objects or spoken phrases.

[0086] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motor vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to recognize cars, trucks, and airplanes.

[0087] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in a higher layer. Hierarchical representations can be established in successive layers of a feedforward network, as described above. Neural networks can also have recursive or feedback (also known as top-down) connections. In a recursive connection, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. Recursive architectures can help identify patterns that span more than one block of input data that is passed to the neural network in sequence. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections may be helpful when the recognition of high-level concepts may help distinguish specific low-level features of the input. The connections between the layers of a neural network can be fully connected or locally connected. The following is a discussion of Figure 4A-Figure 5 Various examples of neural network architectures are described.

[0088] Figure 3A system 300 is depicted that includes a device 302 configured to perform video encoding using a machine learning decoding system 310. The device 302 is coupled to a camera 307 and a storage medium 314 (e.g., a data storage device). In some embodiments, the camera 307 is configured to provide image data 308 (e.g., a video data stream) to the processor 304 for encoding by the machine learning decoding system 310. In some embodiments, the device 302 can be coupled to and / or include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 302 can be coupled to a microphone and / or other input device (e.g., a keyboard, a mouse, a touch input device such as a touch screen and / or a touchpad, and / or other input devices). In some examples, the camera 307, the storage medium 314, the microphone, and / or other input devices can be part of the device 302.

[0089] The device 302 is also coupled to the second device 390 via a transmission medium 318 (e.g., one or more wireless networks, one or more wired networks, or a combination thereof). For example, the transmission medium 318 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The transmission medium 318 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 318 may include a router, a switch, a base station, or any other device that may help facilitate communication from a source device to a receiving device. The wireless network may include any wireless interface or a combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFiTM, radio frequency (RF), UWB, WiFi direct connection, cellular, long-term evolution (LTE), WiMaxTM, etc.). The wired network may include any wired interface (e.g., optical fiber, Ethernet, Ethernet over power line, Ethernet over coaxial cable, digital signal line (DSL), etc.). Various devices such as base stations, routers, access points, bridges, gateways, switches, etc. may be used to implement wired and / or wireless networks. The encoded video bitstream data may be modulated according to a communication standard (eg, a wireless communication protocol) and transmitted to a receiving device.

[0090] The device 302 includes one or more processors (referred to herein as "processors") 304 coupled to a memory 306, a first interface ("I / F 1") 312, and a second interface ("I / F 2") 316. The processor 304 is configured to receive image data 308 from a camera 307, from the memory 306, and / or from a storage medium 314. The processor 304 is coupled to the storage medium 314 via the first interface 312 (e.g., via a memory bus) and to a transmission medium 318 via a second interface 316 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0091] Device 390 is similar to device 302 and includes one or more processors (referred to herein as "processors") 394 coupled to memory 392, a first interface ("I / F 1") 396, and a second interface ("I / F 2") 398. Processor 392 is configured to receive data from transmission medium 318, from memory 306, and / or from storage medium 314 via second interface 396. Processor 394 is coupled to storage medium 399 via first interface 398 (e.g., via a memory bus) and to transmission medium 318 via second interface 396 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0092] Processor 304 includes a machine learning encoding system 310. Machine learning decoding system 310 includes an encoder portion 362. Encoder portion 362 is configured to receive input data 370 and process input data 370 to generate encoded data 374 based at least in part on input data 370. In some cases, machine learning encoding system 310 may include both encoder portion 362 and a decoder portion (e.g., decoder portion 366), shown here as being included in processor 394 of device 390. In some implementations, machine learning decoding system 310 may include one or more autoencoders.

[0093] In some embodiments, the encoder portion 362 of the machine learning decoding system 310 is configured to perform lossy compression of the input data 370 to generate the encoded data 374 , such that the encoded data 374 has fewer bits than the input data 370 .

[0094] As shown, the encoder portion 362 of the machine learning decoding system 310 may include a machine learning based preprocessor 363 and an encoder 364. As described above, some embodiments of the processor 304 include multiple processors, and the elements of the encoder portion 362 may be included on different processors of the multiple processors (e.g., executed by different processors of the multiple processors). For example, the encoder 364 may be included on a custom processor (e.g., an application specific integrated circuit (ASIC)).

[0095] In some cases, while video consumption is growing very rapidly, technical requirements are becoming increasingly demanding as resolution, frame rate, and dynamic range continue to increase. In order to provide the expected quality of service as well as the required data processing speed, high throughput, and low power consumption, it is best to use custom hardware to support current consumer video applications, such as ASICs that implement video compression standards such as H.264 / AVC and H.265 / HEVC.

[0096] In some cases, updating ASICs can be difficult and relatively slow, which can limit the opportunity to improve existing ASICs as new technologies and capabilities become available. Machine learning and neural network implementations can be used to help enhance the performance of existing ASICs using a technique referred to herein as video codec neural boosting. A machine learning based preprocessor 363 can be used to perform video codec neural boosting.

[0097] In some embodiments, the preprocessor 363 based on machine learning can be included in a machine learning accelerator or other processors optimized for performing machine learning / artificial intelligence processing. The preprocessor 363 based on machine learning can include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures. The preprocessor 363 based on machine learning can receive encoder parameters 366 for configuring the encoder 364 as input. The encoder parameters 366 can include parameters for configuring the operation of the encoder 364, such as quality parameters (e.g., quality settings, parameters for enabling / disabling optimization, or other parameters for controlling how the encoder 364 operates). In some cases, the preprocessor 363 based on machine learning can also receive codec internal data 368 from the encoder 364 as input. Codec internal data 368 can include information or settings for performing one or more steps of the encoding process, such as scaling transform coefficients, quantizer step sizes, etc. Based on the input (e.g., input data 370, encoder parameters 366, and / or codec internal data 368), the machine learning-based preprocessor 363 can generate codec improvement information 372 (e.g., control side information and / or pixel side information) for improving the performance of the encoder 364. The machine learning-based preprocessor 363 can pass the codec improvement information 372 together with the encoded data 374 to the decoder part 366 for decoding. The machine learning-based preprocessor 363 can pass the encoder parameters 366 together with the input data 370 to the encoder 364 for encoding. In some cases, the encoder 364 can encode the input data 370 based on the encoder parameters 366 using existing video compression standards such as H.264 / AVC and H.265 / HEVC, and generate the encoded data 374. The encoded data 374 can be sent to the device 390 on the transmission medium 318 via the second interface 316.

[0098] The processor 394 of the device 390 includes a machine learning decoding system 350. The machine learning decoding system 350 includes a decoder portion 366. The device 390 can receive the encoded data 374 and the codec improvement information 372 via the second interface 396, and pass the encoded data 374 and the codec improvement information 372 to the decoder portion 366. In this example, the decoder portion 366 includes a decoder 352 and a machine learning based post-processor 354. The decoder portion 366 is configured to receive the encoded data 376 and process the encoded data 376 to generate a representation 378 based on the input data 370, which can be displayed to a user, for example, via a display (not shown). In some cases, the decoder 352 can decode the encoded data 376 using existing video compression standards such as H.264 / AVC and H.265 / HEVC, and generate decoded data 356. The decoded data 356 can be passed to the machine learning based post-processor 354. In some cases, machine learning decoding system 350 may include both an encoder portion (e.g., encoder portion 362) and a decoder portion 366.

[0099] The machine learning based post-processor 354 can receive the decoded data 356 and the codec boost information 372. The machine learning based post-processor 354 can use one or more trained machine learning algorithms to enhance the decoding 356 based on the input codec boost information 372 to generate a representation 378. In some examples, the machine learning based post-processor 354 of the decoder portion includes a neural network, which can include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures. The processor 394 can be configured to send the representation 378 to a storage medium 399 or output the representation 378 for display on a display (not shown).

[0100] The components of system 300 may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

[0101] Although system 300 is shown as including certain components, one of ordinary skill in the art will appreciate that system 300 may include more than Figure 3 More or fewer components than those shown. For example, system 300 may also include or may be part of a computing device that includes an input device and an output device (not shown). In some embodiments, system 300 may also include or may be part of a computing device that includes one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) that communicate and / or are electrically connected to the one or more memory devices, one or more wireless interfaces for performing wireless communications (e.g., including one or more transceivers and baseband processors for each wireless interface), one or more wired interfaces for performing communications via one or more hardwired connections (e.g., serial interfaces such as universal serial bus (USB) inputs, lighting connectors, and / or other wired interfaces), and / or Figure 3 Other components not shown.

[0102] In some embodiments, the system 300 can be implemented locally by and / or included in a computing device. For example, the computing device can include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses or other AR devices), a wearable device, a server (e.g., in a software as a service (SaaS) system or other server-based system), a television, and / or any other computing device having the resource capabilities to perform the techniques described herein.

[0103] As mentioned above, some video decoding systems utilize neural networks or other machine learning systems to compress video and / or image data. Neural networks can be designed to have a variety of connection modes. In a feedforward network, information is passed from a lower layer to a higher layer, wherein each neuron in a given layer communicates with a neuron in a higher layer. A hierarchical representation can be established in a continuous layer of a feedforward network, as described above. Neural networks can also have recursive or feedback (also referred to as top-down) connections. In a recursive connection, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. A recursive architecture can help identify patterns that span more than one input data block in an input data block that is sequentially passed to a neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept may help distinguish specific low-level features of the input, a network with many feedback connections may be helpful.

[0104] The connections between the layers of a neural network can be fully connected or partially connected. Figure 4A An example of a fully connected neural network 402 is shown. In the fully connected neural network 402, neurons in a first layer may transmit their output to every neuron in a second layer, such that every neuron in the second layer will receive input from every neuron in the first layer. Figure 4B An example of a locally connected neural network 404 is shown. In the locally connected neural network 404, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 404 may be configured such that each neuron in the layer will have the same or similar connection pattern, but with connection strengths that may have different values ​​(e.g., 410, 412, 414, and 416). The connection patterns of the local connections may produce spatially different receptive fields in higher layers because higher layer neurons in a given area may receive inputs that are tuned through training to properties of a restricted portion of the total input to the network.

[0105] An example of a locally connected neural network is a convolutional neural network. Figure 4C An example of a convolutional neural network 406 is shown. The convolutional neural network 406 can be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 408). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful. According to aspects of the present disclosure, the convolutional neural network 406 can be used to perform one or more aspects of video compression and / or decompression.

[0106] One type of convolutional neural network is a deep convolutional network (DCN). Figure 4DA detailed example of a DCN 400 is shown, which is designed to recognize visual features from an image 426, which is input from an image capture device 430 (e.g., a vehicle-mounted camera). The DCN 400 of the current example can be trained to recognize traffic signs and numbers provided on traffic signs. Of course, the DCN 400 can be trained for other tasks, such as recognizing lane markings or recognizing traffic lights.

[0107] The DCN 400 may be trained using supervised learning. During training, an image, such as an image 426 of a speed limit sign, may be presented to the DCN 400, and then a forward pass may be calculated to produce an output 422. The DCN 400 may include a feature extraction portion and a classification portion. Upon receiving the image 426, a convolutional layer 432 may apply a convolutional kernel (not shown) to the image 426 to generate a first feature map set 418. As an example, the convolutional kernel of the convolutional layer 432 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first feature map set 418, four different convolutional kernels are applied to the image 426 at the convolutional layer 432. A convolutional kernel may also be referred to as a filter or a convolutional filter.

[0108] The first feature map set 418 may be subsampled by a maximum pooling layer (not shown) to generate a second feature map set 420. The maximum pooling layer reduces the size of the first feature map set 418. That is, the size of the second feature map set 420 (e.g., 14x14) is smaller than the size of the first feature map set 418 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second feature map set 420 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent feature map sets (not shown).

[0109] exist Figure 4D In the example of , the second feature map set 420 is convolved to generate a first feature vector 424. In addition, the first feature vector 424 is further convolved to generate a second feature vector 428. Each feature of the second feature vector 428 may include a number corresponding to a possible feature of the image 426, such as "symbol", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 428 into probabilities. Therefore, the output 422 of the DCN 400 is the probability that the image 426 includes one or more features.

[0110] In this example, the probability of "sign" and "60" in output 422 is higher than the probability of others in output 422, such as "30", "40", "50", "70", "80", "90", and "100". Before training, the output 422 produced by DCN 400 may be incorrect. Therefore, the error between the output 422 and the target output can be calculated. The target output is the ground truth of image 426 (e.g., "sign" and "60"). The weights of DCN 400 can then be adjusted so that the output 422 of DCN 400 is more closely aligned with the target output.

[0111] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease when the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the neurons activated in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0112] In practice, the error gradient of the weights may be computed over a small number of examples so that the computed gradient is close to the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate for the entire system has stopped decreasing or until the error rate has reached a target level. After learning, new images may be presented to the DCN, and a forward pass through the network may produce an output 422 that may be considered an inference or prediction of the DCN.

[0113] A deep belief network (DBN) is a probabilistic model comprising multiple layers of hidden nodes. A DBN can be used to extract a hierarchical representation of a training data set. A DBN can be obtained by stacking layers of a restricted Boltzmann machine (RBM). RBM is an artificial neural network that can learn a probability distribution of a set of inputs. Since RBM can learn probability distributions without information about the category to which each input should be classified, RBM is often used for unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of the input and target class from the previous layer) and can be used as a classifier.

[0114] A deep convolutional network (DCN) is a network of convolutional networks that is configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0115] The DCN may be a feed-forward network. Furthermore, as described above, connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared among the neurons in the first layer. The feed-forward and shared connections of the DCN may be used for fast processing. For example, the computational burden of a DCN may be much less than that of a similarly sized neural network that includes recursive or feedback connections.

[0116] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels (e.g., red, green, and blue channels of a color image), then the convolutional network trained on this input can be thought of as three-dimensional, with two spatial dimensions along the axes of the image and a third dimension that captures color information. The output of the convolutional connection can be thought of as forming a feature map in a subsequent layer, with each element of the feature map (e.g., 420) receiving input from a series of neurons in the previous layer (e.g., feature map 418) and from each of the multiple channels. The values ​​in the feature map can be further processed using nonlinearities (e.g., correction, max(0,x)). Values ​​from neighboring neurons can be further merged, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0117] Figure 5 is a block diagram showing an example of a deep convolutional network 550. The deep convolutional network 550 may include multiple different types of layers based on connections and weight sharing. Figure 5 As shown, the deep convolutional network 550 includes convolution blocks 554A and 554B. Each of the convolution blocks 554A and 554B may be configured with a convolution layer (CONV) 556, a normalization layer (LNorm) 558, and a maximum pooling layer (MAX POOL) 560.

[0118] The convolution layer 556 may include one or more convolution filters that may be applied to the input data 552 to generate a feature map. Although only two convolution blocks 554A, 554B are shown, the present disclosure is not limited thereto, and any number of convolution blocks (e.g., blocks 554A, 554B) may be included in the deep convolution network 550 according to design preferences. The normalization layer 558 may normalize the output of the convolution filter. For example, the normalization layer 558 may provide whitening or lateral suppression. The maximum pooling layer 560 may provide downsampling aggregation in space for local invariance and dimensionality reduction.

[0119] For example, the parallel filter bank of the deep convolutional network can be loaded on the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank can be loaded on the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 550 can access other processing blocks that may exist on the SOC 100, such as the sensor processor 114 and the navigation module 120 dedicated to sensors and navigation, respectively.

[0120] The deep convolutional network 550 may also include one or more fully connected layers, such as layer 562A (labeled "FC1") and layer 562B (labeled "FC2"). The deep convolutional network 550 may also include a logistic regression (LR) layer 564. Between each layer 556, 558, 560, 562A, 562B, 564 of the deep convolutional network 550 are weights (not shown) to be updated. The output of each of the layers (e.g., 556, 558, 560, 562A, 562B, 564) may be used as an input to a subsequent layer of the layers (e.g., 556, 558, 560, 562A, 562B, 564) in the deep convolutional network 550 to learn a hierarchical feature representation from the input data 552 (e.g., image, audio, video, sensor data, and / or other input data) provided at the first convolutional block 554A. The output of the deep convolutional network 550 is a classification score 566 for the input data 552. The classification score 566 may be a set of probabilities, where each probability is a probability that the input data includes a feature from a feature set.

[0121] While conventional video compression techniques are designed empirically using human knowledge and intuition in some cases, neural networks and other machine learning tools rely on large amounts of training data and efficient learning algorithms. In general, there are two main categories of learning algorithms: (1) techniques based on discrete trial-and-error testing with on-the-fly generation of optimization strategies, and (2) techniques that compute the gradient of the objective (or loss) function and use automatic differentiation in the form of gradient backpropagation to optimize system parameters. Both approaches have been successfully used to solve real-world problems, but the availability of derivatives of performance metrics often makes the second approach more efficient and therefore more commonly used. For example, the second approach is the approach adopted in well-known development tools such as TensorFlow and PyTorch.

[0122] One problem with video decoding systems is that several operations implemented by standard video codecs are not differentiable. In addition, video coding systems sometimes include several stages, such as adaptive deblocking filters, adaptive loop filters, etc., which may be linear and differentiable, but are quite complex and difficult to integrate into training procedures.

[0123] In some cases, when designing a neural boosting system (e.g., during the learning (or training) phase), the compressed data and decoded video of the codec are not used directly, because only a measure of the codec performance and its derivatives are needed for optimization. Therefore, if a sufficiently good estimate of the measure and its derivatives can be obtained, the full codec is not needed. Systems designed for this purpose are referred to herein as differentiable codec proxies.

[0124] As mentioned above, entropy decoding is one of the last stages (and in some cases the last stage) of encoding (compression), which defines the value and number of bits to be added to the compressed data bitstream. Modern standard-based video coding methods (e.g., VVC, HEVC, AV1, etc.) use adaptive arithmetic decoding to achieve high-quality compression performance. The bitstream generated by adaptive arithmetic decoding can only be encoded and decoded sequentially. For example, a data element can only be recovered by first decoding all previous elements, because the decoder needs to reach the same state that the encoder had when decoding the element.

[0125] Fig. 6A600 is a block diagram illustrating an example implementation of a video codec neural lifting system according to aspects of the present disclosure. In the example video codec neural lifting system, when a codec with neural lifting is deployed within the video codec neural lifting system, a standard codec (e.g., VVC, HEVC, AVC, MPEG, AV1, or other similar codecs) can be used. Video data 604 can be input to an ML preprocessing engine 606. The ML preprocessing engine 606 can include one or more ML models. One or more ML models can perform a variety of preprocessing tasks. Examples of preprocessing tasks can include spatial upsampling, temporal upsampling, quality optimization, compression artifact removal, selective denoising, dynamic frame resizing, and the like. In general, these preprocessing tasks are intended to improve the quality of video and / or video compression without modifying the video encoder 608, which can be a video encoder that complies with existing standards (e.g., VVCV, HEVC, AVC, MPEG, AV1, or other similar encoders). For example, one preprocessing task can be to apply an ML model to perform selective denoising to eliminate noise types that the encoder may be less able to handle, thereby improving the compression of the video. As another example, the ML model can dynamically adjust encoder parameters 614 based on the input video data 604 to improve the compression or quality of the video. The compressed video 610 can be sent 612 to a playback device (which can be any device, including an encoding device) for decoding by a video decoder 616. In some cases, the video decoder 616 can be a standard decoder (e.g., VVCV, HEVC, AVC, MPEG, AV1, or other similar decoder). The video data 604 output by the video decoder 616 can be input to an ML post-processing engine 618. The ML post-processing engine 618 can perform various post-processing tasks. For example, the ML post-processing engine 618 can apply one or more ML models to dynamically add noise similar to the noise removed by the ML pre-processing engine 606 to the output video.

[0126] Figure 6B6 is a block diagram 650 illustrating the training of a video codec neural lifting system according to aspects of the present disclosure. In some cases, during the network learning (training) phase, the codec is replaced by a differentiable codec agent 652 that enables gradient backpropagation 654. Video data 654 can be input to an ML pre-processing engine 656. In some cases, the ML pre-processing engine 656 can be the ML pre-processing engine 606 being trained. The output from the ML pre-processing engine 656 is input to the differentiable codec agent 652. The differentiable codec agent 652 can estimate the bit rate 670 of the compressed video that a video encoder (e.g., video encoder 608) can output. The differentiable codec agent 652 can pass the estimated bit rate 670 to the loss measurement engine 668. The differentiable codec agent 652 can also pass the video data 654 to the ML post-processing engine 668. In some cases, the ML post-processing engine 668 can be the ML post-processing engine 618 being trained. The output from the ML post-processing engine 668 may be passed to a loss measurement engine 668. The loss measurement engine 668 may compare the output from the ML post-processing engine 668 and the estimated bitrate 670 to a ground truth reference to calculate a loss function and gradients 672. The gradients 672 may be back-propagated to the ML pre-processing engine 656, the differentiable codec proxy 652, and the ML post-processing engine 668 for training.

[0127] In some cases, the efficacy of using a microcodec proxy 652 depends on the accuracy of the proxy estimate, which is defined by two factors, controlled by a selected quality parameter referred to herein as QP. The bitrate R (which is related to the QP) corresponds to the number of bits used by the encoder to compress a video frame or block within a frame, divided by the number of pixels normalized to bits per pixel. The bitrate decreases as the QP increases. The distortion D (which is also related to the QP) is a measure of the difference between the original video pixels and the decoded video pixels, also normalized per pixel. For example, the distortion can correspond to a mean square error or a more complex measure that approximates human subjective preferences. The distortion increases as the QP increases.

[0128] Bitrate and distortion can be combined into a single loss function, such as the loss function defined in Equation 1 below:

[0129] L(QP)=D(QP)+λ(QP)R(QP),

[0130] Equation (1)

[0131] Where λ(QP) is a Lagrange multiplier defined together with the quantization step size according to the definition of QP in the video standard.

[0132] One observation is that distortion measures strongly depend on the quantization step size predefined by the encoder parameter QP and are therefore more predictable and easier to estimate. On the other hand, bit rates are more difficult to estimate because they are affected by complex statistical dependencies between many pixels.

[0133] Some video encoders (e.g. some HEVC / H.265 standard video encoders) implement a so-called hybrid coding configuration, which combines predictive coding and transform coding. The components that perform prediction and transformation are linear and therefore differentiable and more easily integrated directly into the training process.

[0134] In some cases, some encoder elements are non-linear and non-differentiable, such as those defined by quantization and entropy decoding processes. As used herein, non-differentiable can refer to processes that cannot be usefully differentiable for gradient backpropagation and / or loss measurement. For example, a function that rounds decimal numbers can be a quantized basic form, where input decimal numbers (e.g., 3.14 and 2.56) are rounded to 3. The derivative of the output of the function over the input range (e.g., a set of decimal numbers from 2.51 to 3.50 (assuming two decimal places)) will simply be zero from 2.51 to 3.49, and then infinity for 3.50. Such differential outputs may not be useful for gradient backpropagation because there is no true gradient.

[0135] Figure 7 706 ) is a block diagram illustrating encoder elements of a video encoder 700 (e.g., a HEVC or H.265 encoder) in accordance with aspects of the present disclosure. In some cases, certain encoder elements may be non-linear and non-differentiable, such as the quantization 702 and entropy coding 704 processes. Assuming a block has size M×N pixels, a linear transform (typically a discrete cosine transform) is applied to the residual (e.g., an array of differences between predicted pixel values ​​and actual pixel values). The resulting array of transform coefficients d_(m,n) is then scaled (divided by a positive quantizer step size s 706), m=0,1,…,M-1, n=0,1,…,N-1, and quantization is performed according to the following equation 2:

[0136] q m,n =sign(c m,n )[|c m,n |+ξ],

[0137] Equation (2)

[0138] where ξ is an offset that defines the dead zone quantization type.

[0139] Quantized transform coefficient array q m,nis entropy coded, defining the number of bits used to encode the block. Since quantization is non-differentiable, any proxy estimate needs to use the information from the scaled transform coefficient array c m,n The value of , not q m,n .

[0140] In some cases, the image decoding differentiable agent can estimate the number of bits as a function, such as Equation 3 below:

[0141]

[0142] Here, μ is a normalization constant.

[0143] However, current video coding standards use complex binary adaptive arithmetic coding, and they process the array in multiple passes, using decoding context to exploit the m,n Compression is improved by taking into account the statistical dependencies between the elements of . Therefore, any estimator like the above equation that does not take into account the dependencies between coefficients will have very limited accuracy.

[0144] As described above, according to various aspects of the present disclosure, the present invention describes a method for m,n Systems and techniques for at least addressing this problem by defining a joint statistical model for all values ​​of , computing maximum likelihood estimates of the model parameters, and then using the model to estimate the number of bits. Such processes are differentiable.

[0145] Figure 8 800 . In the differentiable encoder proxy 800 , the non-differentiable quantization 702 and entropy coding 704 elements of an encoder (e.g., video encoder 700 ) can be replaced by a differentiable rate estimation process of the encoder proxy 800 . The differentiable rate estimation process includes a noise and adjustment process 802 , a parameter estimation process 804 , and a block mode 806 . It is assumed that the proxy implementation approximates the main encoder decisions, such as selecting the correct prediction 808 , block size (M, N) and transform type 810 .

[0146] Regularization and coefficient adjustment will now be described. For example, in some cases, quantization may change the scaled transform coefficients in a process similar to adding random quantization noise η, which has a roughly uniform distribution in the interval [-0.5, 0.5]. For this purpose, two random number arrays with uniform distribution can be defined as In some cases, of the two arrays of random values, one may be chosen to be non-zero and the other may be chosen to be zero.

[0147] In some cases, this technique of adding random numbers is used to account for quantization of latent variables when training an end-to-end neural codec. However, in this example, it is included because it has been empirically observed to improve estimation accuracy and because it can also be used as a form of regularization, as it avoids numerical instabilities during estimation when all transform coefficients have very small or zero magnitudes.

[0148] Another factor is that, although not required by the standard, video codecs often use a dead-zone quantizer (e.g., Equation 2), which increases the probability of quantizing to zero values. To approximate this feature, the "adjustment" function is defined as Equation 4 below:

[0149]

[0150] Where x is a parameter of the function, and A and integer K are positive constants (parameters). In some cases, the adjustment function can be updated, for example, during the training process of the encoder agent. For example, the values ​​of A and K or x can be adjusted based on the loss function.

[0151] An example derivative of Equation 4 is illustrated below in Equation 5:

[0152]

[0153]

[0154] Fig. 9 is a graph of an example adjustment function and a derivative of the adjustment function 900 according to aspects of the present disclosure. In this example, Fig. 9 A first line 902 is shown corresponding to Equation 4, given Based on these definitions, random variables can be defined in the statistical model for bit rate estimation, such as shown in Equation 6 below:

[0155]

[0156] Fig. 9 The second line 904 is shown corresponding to the derivative of Equation 4 (eg, Equation 5).

[0157] The definition of statistical models and parameter estimates will now be discussed. For example, in some cases, a statistical model may be defined using a vector x containing the three parameters of the model and may be based on one or more assumptions. The first assumption for which the statistical model may be defined is that the array t m,n All elements of are independent and have a Laplace distribution (which has zero mean and scaling s m,n (x)). For example, the array t m,n The probability distribution function of the elements of can be described by the following equation 7:

[0158]

[0159] To simplify the representation, we can scale s instead of the standard deviation to parameterize the Laplace distribution.

[0160] The second assumption for which the statistical model can be defined is that the Laplace distribution scaling array is defined by Equation 8 as follows:

[0161]

[0162] In some cases, the probability distribution of the transform coefficients can be well approximated by a Laplace distribution. The rapid decay of scaling with frequency (defined by the indices m and n) is also well known. In some cases, choosing an exponential function to model the decay greatly simplifies the derivative formula and its calculation. In some cases, the values ​​of the indices m and n are updated, for example, during the training process of the encoder agent. For example, the indices m and n can be adjusted based on the loss function during training.

[0163] Model parameter estimation will now be described. For example, given an array of modified transform coefficients in a block t m,n , the parameter vector x can be estimated using the maximum likelihood (ML) method. As explained above, by using the array t defined in Equation 6 ′ m,n To replace the array t m,n , which can improve the prediction accuracy and avoid numerical instability. To simplify the representation, Equation 9 and Equation 10 can be defined as follows:

[0164]

[0165] From Equation 6, Equation 8, Equation 9, and Equation 10, the negative of the log-likelihood (e.g., the function to be minimized to obtain the optimal parameter vector) can be shown, such as shown in Equation 11 below:

[0166]

[0167] The optimal solution corresponding to the maximum likelihood parameter is defined by setting the gradient of Equation 11 to zero, which can be shown to correspond to the first set of equations:

[0168] The first set of equations has no known closed-form solution, but the solution can be efficiently computed using Newton's method iterations, defined by Equation 12 below:

[0169]

[0170] Among them, 0<ξ≤1 is the multiplication factor added to ensure convergence.

[0171] In Equation 12, the gradient is:

[0172]

[0173] And the corresponding Hessian matrix is:

[0174]

[0175] In some cases, the output gradient of Equation 12 may be determined and used, for example, by a loss measurement system (e.g., loss measurement engine 668 of FIG. 6 ) for backpropagation as part of a loss function for training a preprocessor (e.g., ML preprocessing engine 656 ) or a differentiable codec agent (e.g., differentiable codec agent 652 ).

[0176] Some observations on the practical application of Newton's method are as follows:

[0177] (1) As mentioned above, the use of exponential functions simplifies the derivative formula and its calculation because several factors appear repeatedly in the above equation. For example, with the factor {w m,n ,mw m,n ,nw m,n ,m 2 w m,n ,n 2 w m,n ,mn w m,n} can be computed once and reused in each iteration.

[0178] (2) Divergence can be avoided by using an adaptive step size correction method, for example, for each iteration, first try ξ = 1, and if L(x) does not decrease, halve its value.

[0179] (3) Since the Hessian matrix is ​​symmetric, the Newton method step size in Equation 12 is more efficiently calculated using Cholesky matrix decomposition rather than inverse matrix decomposition.

[0180] Experimental results show that when a reasonably good initial solution is used, quadratic convergence begins after only 2 or 3 iterations. For example, in experiments using 8×8 DCT coefficients, the following initial solution can be used:

[0181]

[0182] Bit rate estimation will now be described. Figure 7As shown, entropy decoding 704 in video compression is applied to the integers obtained by quantization, where a probability value is assigned to each quantization interval. In some cases, in order to approximate this stage using a differentiable method, a relaxation technique developed for training an end-to-end neural image codec can be adopted, in which a fixed set of intervals used for quantization and probability calculation is replaced by intervals around the coefficient values.

[0183] Using this method, the probability value corresponding to the value to be used for entropy decoding can be defined by the following equation 13:

[0184]

[0185] In Equation 13, the cumulative distribution of the Laplace distribution can be defined as defined in Equation 14 below:

[0186]

[0187] Using these definitions, an estimate of the number of bits used to decode a block can be defined by the following equation 15:

[0188]

[0189] Here, α is a normalization constant.

[0190] In some cases, the estimated number of bits may be generated, for example, by a differentiable codec agent (e.g., differentiable codec agent 652 of FIG. 6). For example, a loss measurement system (e.g., loss measurement engine 668) may use the estimated number of bits for back propagation.

[0191] The partial derivative calculation will now be described. To train a neural network, the gradient of the estimated number of bits (eg, the partial derivative array) may be used according to Equation 16 below:

[0192]

[0193] In some cases, the partial derivative arrays can be computed using automatic differentiation tools available in software tools such as PyTorch, but it should be noted that in order to enable backpropagation, Equation 12 may need to be replaced by l = 0, 1, 2, ..., thereby creating an aggregate sequence of solutions and allowing the software to create the correct links for automatic differentiation. Such a technique may increase computational complexity because it adds multiple stages of gradient calculation. A more efficient implementation may be performed if automatic differentiation is removed from the maximum likelihood optimization and direct calculation of those partial derivatives using the optimal solution is used.

[0194] To compute those derivatives, we can define and

[0195] Or partial derivative calculation, optimality condition It can correspond to the following set of MN vector equations:

[0196]

[0197] in,

[0198]

[0199] Using these results, Equation 16 can be expanded to:

[0200]

[0201] to the equation defining the gradient computation:

[0202]

[0203] in,

[0204] Then, according to Equation 5 and Equation 6, Equation 17 can be derived as follows:

[0205]

[0206] In some aspects, equation (17) can be simplified by collecting the sum of terms that vary with index m, n. For example, the following function can be defined:

[0207]

[0208] Ψ m,n (b)=(t m,n +b) ψ m,n (b), Equation (20)

[0209] and a vector z such that:

[0210]

[0211] Therefore, equation (17) can be corresponded to:

[0212]

[0213] An illustrative example of experimental validation will now be described. For example, the systems and techniques described herein can be tested by modifying a reference implementation of the HEVC / H.265 video compression standard, referred to as HM version 20.0, to output DCT coefficients used by an encoder in a low-latency configuration, with the luma transform dimension limited to 8×8 blocks, QP values ​​of 22, 27, 32, and 37, and the number of bits used in each case.

[0214] Compression is applied to 201 frames of 10 HD test videos, and only the luminance coefficients in the inter-frames (i.e., excluding the intra-frames) are saved. For each frame, ∈ in Eq. (5) is used. (c) =0.5,∈ (t) = 0, A = 1, K = 2 in Equation 5 to estimate the bit rate, and α = 1 was initially used in Equation 15. After comparing the estimated bit rate with the actual bit rate, the parameter α was calibrated to α = 5 / 3, which is the value used in all results presented next.

[0215] In some cases, the choice of parameter α is not important because during neural network training, the bitrate estimate is multiplied by a constant as shown in Equation 1, and several constant values ​​need to be tested to obtain the best results. Similarly, the estimator calibration form HEVC can be used with AVC by testing new scaling factors.

[0216] The results obtained using the systems and techniques described herein can be evaluated by measuring the ratio of the estimated frame bit rate to the corresponding value from the HM software. Figure 10-13 Histograms of these ratios calculated for 2,000 frames of the test video are shown, with each plot showing results for QP = 22, 27, 32, and 37. As expected, the results peak at low QP, which corresponds to the high rate case where the per-coefficient factor dominates. As the QP value increases, the histogram becomes more dispersed, and well-constrained errors can be observed even for the maximum QP = 27.

[0217] Fig.10 is a histogram of the ratio between frame bitrates estimated using the systems and techniques described herein and the bitrate used by the HM software (HEVC / H.265 reference implementation) and QP=22.

[0218] Fig.11 is a histogram of the ratio between frame bitrates estimated using the systems and techniques described herein and the bitrate used by the HM software (HEVC / H.265 reference implementation) and QP=27.

[0219] Fig.12is a histogram of the ratio between frame bitrates estimated using the systems and techniques described herein and the bitrate used by the HM software (HEVC / H.265 reference implementation) and QP=32.

[0220] Fig.13 is a histogram of the ratio between frame bitrates estimated using the systems and techniques described herein and the bitrate used by the HM software (HEVC / H.265 reference implementation) and QP=37.

[0221] Fig.14 is a histogram of the ratio between the frame bitrate estimated using the technique for enhancing bitrate estimation using machine learning for video decoding and the bitrate used by the HM software (HEVC / H.265 reference implementation), for combined results of QP=22, 27, 32 and 37.

[0222] Fig.14 shows the combined results for all QP values, which can be compared with using Fig.15 The values ​​estimated by the non-differentiable AGP entropy decoding method shown are compared, which uses a much simpler way to model the transform coefficients than the method used by HEVC. Finally, Fig.16 Results obtained using per-coefficient differentiable estimates of Equation 3 are shown, where the results are widely distributed, indicating much lower precision.

[0223] Fig.15 is a histogram of the ratio between the frame bitrate estimated using the AGP entropy coding method (non-differentiable) and the bitrate used by the HM software (HEVC / H.265 reference implementation), for combined results of QP=22, 27, 32 and 37.

[0224] Fig.16 is a histogram of the ratio between the frame bitrate estimated using the per-coefficient differentiable estimate of Equation 3 and the bitrate used by the HM software (HEVC / H.265 reference implementation), for the combined results of QP=22, 27, 32, and 37.

[0225] For reference, Table 1 shows the average bitrates for different QP values, indicating that they roughly cover an order of magnitude.

[0226] Quality Settings QP=22 QP=27 QP=32 QP=37 Average bit rate 0.045 bits / pixel 0.11 bits / pixel 0.23 bits / pixel 0.48 bits / pixel

[0227] Table 1

[0228] Aspects of the present disclosure address the problem that occurs when a neural network is used to enhance the performance of a standard video codec in several ways (which is referred to as standard video codec neural boosting) and the gradients of the performance measure cannot be back-propagated through the codec, thereby preventing full optimization.

[0229] Aspects of the present disclosure address the problem of reliably estimating the bitrate used by a video encoder while simultaneously estimating the corresponding derivatives and allowing the use of gradient backpropagation during end-to-end training.

[0230] The estimate is more accurate and reliable because, similar to entropy decoding in standard encoders, it combines information from many discrete cosine transform (DCT) coefficients used for compression. This is achieved using a new form of statistical model that assumes the DCT coefficients have a Laplace distribution with three parameters determined using the maximum likelihood criterion.

[0231] The model generates scaling parameters for the Laplace distribution, a process similar to that done by the entropy decoding hyper-prior neural network of the end-to-end neural video codec, and this similarity simplifies the integration of the systems and techniques described herein into systems for neural network training.

[0232] Experimental results comparing frame bit rates obtained using the systems and techniques described herein with exact values ​​from the HEVC / H.265HM codec show that this approach provides accuracy very similar to non-specific and non-differentiable entropy coding methods and much more accurate than methods using per-coefficient estimates.

[0233] Due to the use of mathematical statistical models, it is possible to derive formulas defining all necessary derivatives, which can be computed more efficiently directly using C++ or CUDA GPU implementations instead of automatic differentiation.

[0234] Fig.17 1700 is a flow chart illustrating a technique for performing bit rate estimation 1700 according to aspects of the present disclosure. At operation 1702, technique 1700 may include encoding one or more frames of video data using a video encoder that includes at least a quantization process, such as Figure 6A-6B and Figure 7 At operation 1704, technique 1700 may include: determining Figure 6A-6B and Figure 7 The actual bit rate of the encoded frame or frames is shown.

[0235] At operation 1706, technique 1700 may include predicting an estimated bit rate using an encoder agent including a statistical model for estimating a bit rate of one or more frames being encoded, such as Figure 8In some cases, the encoder agent estimates the output of one or more processes of the video encoder. In some cases, the statistical model is based on a Laplace distribution of coefficients of a quantization process input to the video encoder. In some cases, the technique 1700 may also include estimating the output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0236] At operation 1708, technique 1700 may include using an encoder agent to determine a gradient of an estimated bit rate, such as Figure 8 As shown. In some cases, the video encoder may include an entropy decoding process after the quantization process, and further include estimating the output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder. In some cases, the estimated bit rate is based on the estimated output of the quantization process and the estimated output of the quantization process. In some cases, the gradient is determined based on at least a derivative of the statistical model.

[0237] At operation 1710 , the technique may include training an encoder agent to predict an estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0238] Fig.18 1800 is a flow chart illustrating a technique for performing bit rate estimation 1800 according to aspects of the present disclosure. At operation 1802, technique 1800 may include receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process, such as Figure 6A-6B and Figure 7 shown.

[0239] At operation 1804, technique 1800 may include predicting an estimated bitrate of one or more frames after being encoded by a video encoder using an encoder proxy, wherein the encoder proxy includes a statistical model for estimating the estimated bitrate, and wherein the statistical model is trained based on a gradient of the estimated bitrate, such as Figure 6B-Figure 8 As shown. In some cases, the encoder agent estimates the output of one or more processes of the video encoder. In some cases, the statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder. In some cases, technology 1800 may also include: estimating the output of the quantization process based on the maximum likelihood estimate of the Laplace distribution. In some cases, the video encoder may include an entropy decoding process after the quantization process, and also includes estimating the output of the entropy decoding process based on the interval around the coefficients of the quantization process input to the video encoder. In some cases, the estimated bit rate is based on the estimated output of the quantization process and the estimated output of the quantization process. In some cases, the gradient is determined based on at least the derivative of the statistical model.

[0240] At operation 1806, technique 1800 may adjust one or more quality parameters based on the predicted estimated bit rate, such as Fig. 6A At operation 1808, technique 1800 may encode one or more frames of video data using a video encoder that includes at least a quantization process, such as Fig. 6A and Figure 7 shown.

[0241] Fig.19 An example computing device architecture 1900 is shown for an example computing device that can implement the various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. For example, the computing device architecture 1900 may be used as Figure 2 System 200 and / or Figure 3 1900. The components of the computing device architecture 1900 are shown in electrical communication with each other using connections 1905 (e.g., a bus). The example computing device architecture 1900 includes a processing unit (CPU or processor) 1910 and computing device connections 1905 that couple various computing device components (including computing device memory 1915, such as read-only memory (ROM) 1920 and random access memory (RAM) 1925) to the processor 1910.

[0242] The computing device architecture 1900 may include a cache of high-speed memory directly connected to the processor 1910, adjacent to the processor 1910, or integrated as part of the processor 1910. The computing device architecture 1900 may copy data from the memory 1915 and / or the storage device 1930 to the cache 1912 for fast access by the processor 1910. In this way, the cache can provide a performance improvement that avoids delays in the processor 1910 when waiting for data. These and other modules can control or be configured to control the processor 1910 to perform various actions. Other computing device memories 1915 may also be used. The memory 1915 may include a variety of different types of memory with different performance characteristics. The processor 1910 may include any general-purpose processor and hardware or software services (e.g., service 1 1932, service 2 1934, and service 3 1936) stored in the storage device 1930 and configured to control the processor 1910, as well as a dedicated processor, wherein the software instructions are incorporated into the processor design. Processor 1910 may be a self-contained system including multiple cores or processors, a bus, a memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0243] To enable a user to interact with the computing device architecture 1900, the input device 1945 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 1935 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-modal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 1900. The communication interface 1940 can generally govern and manage user input and computing device output. There is no limitation on operation on any particular hardware arrangement, and thus the basic features here can be easily replaced with improved hardware or firmware arrangements (as they are developed).

[0244] The storage device 1930 is a non-volatile memory and can be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a random access memory (RAM) 1925, a read-only memory (ROM) 1920, and a mixture thereof. The storage device 1930 may include services 1932, 1934, 1936 for controlling the processor 1910. Other hardware or software modules are contemplated. The storage device 1930 may be connected to the computing device connection 1905. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium that is connected to the necessary hardware components (e.g., the processor 1910, the connection 1905, the output device 1935, etc.) to perform the function.

[0245] Aspects of the present disclosure are applicable to any suitable electronic device (e.g., a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or being coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0246] The term "device" is not limited to one or a specific number of physical objects (e.g., a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the description and examples below use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific embodiments. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the description and examples below use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0247] Specific details are provided in the description above to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be implemented without using these specific details. For clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, separate functional blocks including functional blocks, functional blocks including devices, device components, steps or routines in methods embodied in software, or a combination of hardware and software. In addition to the components shown in the figure and / or described herein, additional components may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form, so as not to obscure embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring embodiments.

[0248] Various embodiments can be described above as a process or method, which is depicted as a flow chart, flow diagram, data flow diagram, structure diagram or block diagram. Although a flow chart can describe an operation as a sequential process, many operations in the operation can be performed in parallel or concurrently. In addition, the order of these operations can be rearranged. The process terminates when its operation is completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0249] The processes and methods according to the examples described above can be implemented using computer executable instructions stored or otherwise available from a computer readable medium. For example, such instructions may include instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Some portions of the computer resources used may be accessed over a network. The computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc.

[0250] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and do not include carrier waves and / or temporary electronic signals that are propagated wirelessly or by wired connections. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media such as flash memory, memory or memory devices, disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, compact disks (CDs) or digital versatile disks (DVDs), any suitable combination thereof, etc. Codes and / or machine executable instructions may be stored on computer-readable media, which may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameter data, etc. may be passed, forwarded, or sent via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0251] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0252] The equipment implementing the process and method according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may adopt any one of a variety of form factors. When implemented with software, firmware, middleware, or microcode, the program code or code segment (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc., of laptop computers, smart phones, mobile phones, tablet devices, or other small form factors. The functionality described herein may also be embodied in peripheral devices or add-on cards. As a further example, such functionality may also be implemented on circuit boards between different chips or different processes executed in a single device.

[0253] Instructions, media for communicating such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in this disclosure.

[0254] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it will be understood that the inventive concept can be embodied and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except as limited by the prior art. The various features and aspects of the application described above can be used alone or in combination. In addition, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications outside of those described herein. Therefore, this description and the accompanying drawings will be considered illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be appreciated that in an alternative embodiment, the method can be performed in an order different from the described order.

[0255] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols without departing from the scope of the present specification.

[0256] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0257] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0258] Claim language or other language that refers to "at least one" in a set and / or "one or more" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that refers to "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that refers to "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one" in a set and / or "one or more" in a set does not limit the set to the items listed in the set. For example, claim language that refers to "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0259] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate the interchangeability of hardware and software, the above generally describes the various illustrative components, blocks, modules, circuits, and steps around their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in an alternative manner for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0260] The technology described herein can also be implemented in electronic hardware, computer software, firmware or any combination thereof. Such technology can be implemented in any of various devices (such as general-purpose computers, wireless communication device mobile phones or integrated circuit devices with multiple uses for applications included in wireless communication device mobile phones and other devices). Any features described as modules or components can be implemented together in an integrated logic device, or individually implemented as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including a program code, and the program code includes instructions for executing one or more of the methods described above when executed. Computer-readable data storage media can form part of a computer program product, and the computer program product can include packaging materials. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. Additionally or alternatively, these techniques may be implemented at least in part via a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0261] The program code can be executed by a processor, which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, or any combination of the aforementioned structures, or any other structure or device suitable for the implementation of the techniques described herein.

[0262] Illustrative aspects of the disclosure include:

[0263] Aspect 1: A method for processing video data. The method includes: encoding one or more frames of video data using a video encoder, the video encoder including at least a quantization process; determining an actual bit rate of the one or more frames encoded; using an encoder agent to predict an estimated bit rate, the encoder agent including a statistical model for estimating the bit rate of the one or more frames encoded; using the encoder agent to determine a gradient of the estimated bit rate; and training the encoder agent to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0264] Aspect 2. The method of claim 1, wherein the encoder agent estimates outputs of one or more processes of the video encoder.

[0265] Aspect 3. The method of any of claims 1-2, wherein the statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

[0266] Aspect 4. The method of claim 3, further comprising estimating an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0267] Aspect 5. The method according to any of claims 1 to 4, wherein the video encoder further includes an entropy decoding process after the quantization process, and further includes estimating the output of the entropy decoding process based on the interval around the coefficients of the quantization process input to the video encoder.

[0268] Aspect 6. The method of claim 5, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

[0269] Aspect 7. The method according to claim 1, wherein the gradient is determined based on at least a derivative of the statistical model.

[0270] Aspect 8. A device for processing video data, the device comprising: at least one memory; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: encode one or more frames of video data using a video encoder, the video encoder comprising at least a quantization process; determine an actual bit rate of the one or more encoded frames; use an encoder agent to predict an estimated bit rate, the encoder agent comprising a statistical model for estimating the bit rate of the one or more encoded frames; use the encoder agent to determine a gradient of the estimated bit rate; and train the encoder agent to predict the estimated bit rate based on the actual bit rate, the estimated bit rate, and the gradient.

[0271] Aspect 9. The apparatus of claim 8, wherein the encoder agent estimates outputs of one or more processes of the video encoder.

[0272] Aspect 10. The apparatus of any of claims 8-9, wherein the statistical model is based on a Laplace distribution of coefficients of the quantization process input to the video encoder.

[0273] Aspect 11. The apparatus of claim 10, wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0274] Aspect 12. An apparatus according to any of claims 8 to 11, wherein the video encoder further comprises an entropy decoding process after the quantization process, and wherein the processor is further configured to estimate the output of the entropy decoding process based on an interval around coefficients of the quantization process input to the video encoder.

[0275] Aspect 13. The apparatus of claim 12, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

[0276] Aspect 14. The apparatus according to claim 8, wherein the gradient is determined based on at least a derivative of the statistical model.

[0277] Aspect 15. A method for processing video data, the method comprising: receiving one or more frames of video data for encoding by a video encoder, the video encoder comprising at least a quantization process; predicting an estimated bit rate of the one or more frames after being encoded by the video encoder using an encoder agent, wherein the encoder agent comprises a statistical model for estimating the estimated bit rate, and wherein the statistical model is trained based on a gradient of the estimated bit rate; adjusting one or more quality parameters based on the predicted estimated bit rate; and encoding the one or more frames of video data using the video encoder.

[0278] Clause 16. The method of claim 15, wherein the encoder agent estimates outputs of one or more processes of the video encoder.

[0279] Aspect 17. The method of any of claims 15-16, wherein the statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

[0280] Aspect 18. The method of claim 17, further comprising estimating an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0281] Aspect 19. The method of any of claims 15-18, wherein the video encoder further comprises an entropy decoding process after the quantization process, and wherein the method further comprises estimating an output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder.

[0282] Clause 20. The method of claim 19, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

[0283] Aspect 21. The method of claim 15, wherein the gradient is determined based on at least a derivative of the statistical model.

[0284] Aspect 22. A device for processing video data, the device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: receive one or more frames of video data for encoding by a video encoder, the video encoder comprising at least a quantization process; predict an estimated bit rate of the one or more frames after being encoded by the video encoder using an encoder agent, wherein the encoder agent comprises a statistical model for estimating the estimated bit rate, and wherein the statistical model is trained based on a gradient of the estimated bit rate; adjust one or more quality parameters based on the predicted estimated bit rate; and encode the one or more frames of video data using the video encoder.

[0285] Clause 23. The apparatus of claim 22, wherein the encoder agent estimates outputs of one or more processes of the video encoder.

[0286] Aspect 24. The apparatus of any of claims 22-23, wherein the statistical model is based on a Laplace distribution of coefficients of the quantization process input to the video encoder.

[0287] Aspect 25. The apparatus of claim 24, wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0288] Aspect 26. The apparatus of claims 22-25, wherein the video encoder further comprises an entropy decoding process subsequent to the quantization process, and wherein the processor is further configured to estimate an output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder.

[0289] Aspect 27. The apparatus of claim 26, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

[0290] Aspect 28. The apparatus of claim 22, wherein the gradient is determined based on at least a derivative of the statistical model.

[0291] Aspect 29. A method for processing video data, the method comprising: receiving one or more frames of video data for encoding by a video encoder; predicting an estimated bit rate of the one or more frames after being encoded by the video encoder using an encoder agent, wherein the encoder agent includes a statistical model for estimating the estimated bit rate; determining a gradient of the estimated bit rate using the encoder agent; encoding the one or more frames of video data using the video encoder, the video encoder including at least a quantization process; obtaining an actual bit rate of the encoded one or more frames; and updating the encoder agent based on a comparison between the estimated bit rate and the actual bit rate.

[0292] Clause 30. The method of claim 29, wherein the encoder agent estimates outputs of one or more processes of the video encoder.

[0293] Clause 31. The method of any of claims 29-30, wherein the statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

[0294] Aspect 32. The method of claim 31, further comprising estimating an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0295] Aspect 33. A method according to any of claims 29-32, wherein the video encoder also includes an entropy decoding process after the quantization process, and wherein the method further includes: estimating the output of the entropy decoding process based on the interval around the coefficients of the quantization process input to the video encoder.

[0296] Aspect 34. The method of claim 33, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

[0297] Aspect 35. The method of claim 29, wherein the gradient is determined based on at least a derivative of the statistical model.

[0298] Aspect 36. A device for processing video data, the device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: receive one or more frames of video data for encoding by a video encoder; predict an estimated bit rate of the one or more frames after being encoded by the video encoder using an encoder agent, wherein the encoder agent includes a statistical model for estimating the estimated bit rate; use the encoder agent to determine a gradient of the estimated bit rate; encode the one or more frames of video data using the video encoder, the video encoder including at least a quantization process; obtain an actual bit rate of the encoded one or more frames; and update the encoder agent based on a comparison between the estimated bit rate and the actual bit rate.

[0299] Clause 37. The apparatus of claim 36, wherein the encoder agent estimates outputs of one or more processes of the video encoder.

[0300] Clause 38. The apparatus of any of Claims 36-37, wherein the statistical model is based on a Laplace distribution of coefficients of the quantization process input to the video encoder.

[0301] Aspect 39. The apparatus of claim 38, wherein the at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

[0302] Aspect 40. The apparatus of claims 36-39, wherein the video encoder further comprises an entropy decoding process subsequent to the quantization process, and wherein the at least one processor is further configured to estimate an output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder.

[0303] Aspect 41. The apparatus of claim 39, wherein the estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

[0304] Aspect 42. The apparatus of claim 36, wherein the gradient is determined based on at least a derivative of the statistical model.

[0305] Aspect 43. An apparatus according to any one of aspects 8 to 14, aspects 22 to 28, and aspects 37 to 42, wherein the apparatus comprises an encoder.

[0306] Aspect 44: The apparatus according to any one of aspects 8 to 14, aspects 22 to 28, and aspects 37 to 43, further comprising: a display configured to display one or more output pictures.

[0307] Aspect 45. The apparatus according to any one of aspects 8 to 14, aspects 22 to 28, and aspects 37 to 44, further comprising a camera configured to capture one or more pictures.

[0308] Aspect 46. An apparatus according to any one of aspects 8 to 14, aspects 22 to 28, and aspects 37 to 45, wherein the apparatus is a mobile device.

[0309] Aspect 47. An apparatus for processing video data, comprising means for performing one or more of the operations according to any one of aspects 8 to 14, aspects 22 to 28, and aspects 37 to 46.

[0310] Aspect 48. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform operations according to any one or more of Aspects 1-7, Aspects 15-21, and / or Aspects 29-35.

[0311] Aspect 49. An apparatus comprising one or more units for performing operations according to any one or more of aspects 1-7, aspects 15-21, and / or aspects 29-35.

Claims

1. A method for processing video data, the method comprising: encoding one or more frames of video data using a video encoder, the video encoder comprising at least a quantization process; determining an actual bit rate of the encoded one or more frames; predicting an estimated bit rate using an encoder agent, the encoder agent comprising a statistical model for estimating a bit rate of one or more frames being encoded; determining, using the encoder agent, a gradient of the estimated bit rate; as well as The encoder agent is trained based on the actual bitrate, the estimated bitrate, and the gradient to predict the estimated bitrate.

2. The method according to claim 1, wherein: The encoder agent estimates outputs of one or more processes of the video encoder.

3. The method according to claim 1, wherein: The statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

4. The method of claim 3, further comprising estimating an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

5. The method according to claim 1, wherein: The video encoder also includes an entropy coding process following the quantization process, and further includes estimating an output of the entropy coding process based on intervals around coefficients of the quantization process input to the video encoder.

6. The method according to claim 5, wherein: The estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

7. The method according to claim 1, wherein: The gradient is determined based on at least a derivative of the statistical model.

8. A device for processing video data, the device comprising: at least one memory; as well as at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: encoding one or more frames of video data using a video encoder, the video encoder comprising at least a quantization process; determining an actual bit rate of the encoded one or more frames; predicting an estimated bit rate using an encoder agent, the encoder agent comprising a statistical model for estimating a bit rate of one or more frames being encoded; determining a gradient of the estimated bit rate using the encoder agent; and The encoder agent is trained based on the actual bitrate, the estimated bitrate, and the gradient to predict the estimated bitrate.

9. The device according to claim 8, wherein: The encoder agent estimates outputs of one or more processes of the video encoder.

10. The device according to claim 8, wherein: The statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

11. The device according to claim 10, wherein: The at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

12. The device according to claim 8, wherein: The video encoder also includes an entropy decoding process following the quantization process, and wherein the at least one processor is further configured to estimate an output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder.

13. The device according to claim 12, wherein: The estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

14. The device according to claim 8, wherein: The gradient is determined based on at least a derivative of the statistical model.

15. A method for processing video data, the method comprising: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting an estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, wherein the encoder proxy estimates the estimated bitrate using a statistical model, and wherein the statistical model is trained based on a gradient of the estimated bitrate; adjusting one or more quality parameters of the video encoder based on the predicted estimated bit rate; and The one or more frames of video data are encoded using the video encoder.

16. The method according to claim 15, wherein: The encoder agent estimates outputs of one or more processes of the video encoder.

17. The method according to claim 15, wherein: The statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

18. The method of claim 17, further comprising estimating an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

19. The method according to claim 15, wherein: The video encoder also includes an entropy decoding process subsequent to the quantization process, and wherein the method further includes estimating an output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder.

20. The method according to claim 19, wherein: The estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

21. The method according to claim 15, wherein: The gradient is determined based on at least a derivative of the statistical model.

22. An apparatus for processing video data, the apparatus comprising: at least one memory; as well as at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: receiving one or more frames of video data for encoding by a video encoder, the video encoder including at least a quantization process; predicting an estimated bitrate of the one or more frames after being encoded by the video encoder using an encoder proxy, wherein the encoder proxy estimates the estimated bitrate using a statistical model, and wherein the statistical model is trained based on a gradient of the estimated bitrate; adjusting one or more quality parameters of the video encoder based on the predicted estimated bit rate; and The one or more frames of video data are encoded using the video encoder.

23. The device according to claim 22, wherein: The encoder agent estimates outputs of one or more processes of the video encoder.

24. The device according to claim 22, wherein: The statistical model is based on the Laplace distribution of coefficients of the quantization process input to the video encoder.

25. The device according to claim 24, wherein: The at least one processor is further configured to estimate an output of the quantization process based on a maximum likelihood estimate of the Laplace distribution.

26. The device according to claim 22, wherein: The video encoder also includes an entropy decoding process following the quantization process, and wherein the at least one processor is configured to estimate an output of the entropy decoding process based on intervals around coefficients of the quantization process input to the video encoder.

27. The device according to claim 26, wherein: The estimated bit rate is based on an estimated output of the quantization process and an estimated output of the quantization process.

28. The device according to claim 22, wherein: The gradient is determined based on at least a derivative of the statistical model.

29. The device according to claim 22, wherein: The device is the video encoder.

30. The device according to claim 22, wherein: The apparatus comprises the video encoder.

Images