Learned b-frame coding using p-frame coding system

Abstract

This document describes techniques for processing video data, such as performing learned bidirectional decoding using a one-way decoding system and interpolated reference frames. For example, the process may include: obtaining a first reference frame and a second reference frame. The process may include: generating a third reference frame at least partially by performing interpolation between the first and second reference frames. The process may include: performing one-way inter-frame prediction on the input frame based on the third reference frame, for example by estimating motion between the input frame and the third reference frame, and generating a deformed frame at least partially by deforming one or more pixels of the third reference frame based on the estimated motion. The process may include: generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including the bidirectional prediction frame.

Description

Technical Field

[0001] In general, this disclosure relates to video decoding (e.g., encoding and / or decoding video data). For example, aspects of this disclosure relate to learning bidirectional predictive frame (B-frame) decoding using a one-way decoding system (e.g., a one-way predictive frame (P-frame) encoder-decoder or CODEC). Prior Technology

[0002] Many devices and systems allow for the processing and output of video data for consumption. Digital video data comprises vast amounts of data to meet ever-increasing demands for video quality, performance, and functionality. For example, consumers of video data typically desire high-fidelity, high-resolution, high-frame-rate, and other high-quality video. Meeting these demands often requires large amounts of video data, placing a significant burden on communication networks and equipment that process and store it. Video decoding technology can be used to compress video data. An example goal of video decoding is to compress video data into a form using a lower bit rate while avoiding or minimizing degradation in video quality. As evolving video services become available and the demand for large amounts of video data continues to increase, there is a need for decoding technologies with better performance and efficiency. Summary of the Invention

[0003] The systems and techniques described herein include bidirectional (e.g., B-frame) decoding using a learned unidirectional (e.g., P-frame) decoding system that can utilize reference frames generated by interpolation between reference frames. According to at least one example, a method for processing video data is provided. The method may include: obtaining a first reference frame and a second reference frame; generating a third reference frame at least in part by performing interpolation between the first and second reference frames; and performing unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0004] In another example, a system for processing video data is provided. The system may include a bidirectional decoding engine and a unidirectional decoding engine with an interpolation engine. In some cases, the system may include at least one memory and one or more processors (e.g., implemented in circuitry) coupled to the memory. The interpolation engine is configured to: obtain a first reference frame and a second reference frame; and generate a third reference frame, at least in part, by performing interpolation between the first and second reference frames. The unidirectional decoding engine is configured to perform unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0005] In another example, a non-transitory computer-readable medium is provided for processing video data. This non-transitory computer-readable medium may include instructions stored thereon that, when executed by one or more processors, cause the processors to: obtain a first reference frame and a second reference frame; generate a third reference frame, at least in part, by performing interpolation between the first and second reference frames; and perform unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0006] According to another example, an apparatus for processing video data is provided. The apparatus may include: components for obtaining a first reference frame and a second reference frame; components for generating a third reference frame, at least partially by performing interpolation between the first and second reference frames; and components for performing unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0007] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: estimating motion between the input frame and the third reference frame; and generating a deformed frame, at least in part, by deforming one or more pixels of the third reference frame based on the estimated motion.

[0008] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: determining a residual at least in part by determining the difference between the input frame and the deformed frame; and using the residual to generate a prediction residual.

[0009] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame; and storing the reconstructed frame in memory.

[0010] In some respects, the motion is estimated using a first autoencoder, and the prediction residual is generated using a second autoencoder.

[0011] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: obtaining a fourth reference frame; estimating motion between an additional input frame and the fourth reference frame; generating an additional deformed frame by deforming one or more pixels of the fourth reference frame, at least in part, based on the estimated motion; and generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

[0012] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: determining a first set of motion information representing pixel motion from a first reference frame to a third reference frame; determining a second set of motion information representing pixel motion from a second reference frame to the third reference frame; generating first deformation information by performing a deformation function on the first reference frame at least in part using the first set of motion information; generating second deformation information by performing the deformation function on the second reference frame at least in part using the second set of motion information; and generating the third reference frame based on the first deformation information and the second deformation information.

[0013] In some aspects, the first set of motion information and the second set of motion information are determined based on the pixel motion between the first reference frame and the second reference frame.

[0014] In some respects, the first set of motion information includes a first optical flow map, and the second set of motion information includes a second optical flow map.

[0015] In some respects, the deformation function includes a bilinear interpolation function.

[0016] In some respects, the estimated motion between the input frame and the third reference frame includes optical flow data.

[0017] In some respects, the estimated motion between the input frame and the third reference frame includes dynamic convolutional data.

[0018] In some respects, the estimated motion between the input frame and the third reference frame includes block-based motion data.

[0019] According to at least one example, a method for processing video data is provided. The method may include: obtaining an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; estimating motion between the input frame and the interpolation reference frame based on the motion information; generating a deformed frame by deforming one or more pixels of the interpolation reference frame, at least in part, based on the estimated motion; and generating a reconstructed frame representing the input frame, the reconstructed frame including a bidirectional prediction frame, based on the deformed frame and a prediction residual.

[0020] In another example, a system for processing video data is provided. The system may include at least one memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The at least one processor is configured to: obtain an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; estimate motion between the input frame and the interpolation reference frame based on the motion information; generate a deformed frame by deforming one or more pixels of the interpolation reference frame, at least in part, based on the estimated motion; and generate a reconstructed frame representing the input frame, the reconstructed frame including a bidirectional prediction frame, based on the deformed frame and a prediction residual.

[0021] In another example, a non-transitory computer-readable medium for processing video data is provided. This non-transitory computer-readable medium may include instructions stored thereon that, when executed by one or more processors, cause the processors to: obtain an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; estimate motion between the input frame and the interpolation reference frame based on the motion information; generate a deformed frame by deforming one or more pixels of the interpolation reference frame, at least in part, based on the estimated motion; and generate a reconstructed frame representing the input frame, the reconstructed frame including a bidirectional prediction frame, based on the deformed frame and a prediction residual.

[0022] According to another example, an apparatus for processing video data is provided. The apparatus may include: components for obtaining an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; components for estimating motion between the input frame and the interpolation reference frame based on the motion information; components for generating a deformed frame by deforming one or more pixels of the interpolation reference frame, at least in part, based on the estimated motion; and components for generating a reconstructed frame representing the input frame based on the deformed frame and a prediction residual, the reconstructed frame including a bidirectional prediction frame.

[0023] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: obtaining a first reference frame and a second reference frame; and generating the interpolated reference frame at least in part by performing interpolation between the first reference frame and the second reference frame.

[0024] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: obtaining residuals from a bitstream; and generating predictive residuals based on the obtained residuals.

[0025] In some respects, the motion information is obtained from the bit stream.

[0026] In some aspects, the methods, apparatus, and non-transitory computer-readable media described above may include: obtaining a fourth reference frame; estimating motion between an additional input frame and the fourth reference frame; generating an additional deformed frame, at least in part, by deforming one or more pixels of the fourth reference frame based on the estimated motion; and generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

[0027] In some aspects, the device may include, or be part of, a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a television, a video game console, or other device. In some aspects, the device includes a mobile device (e.g., a mobile phone or so-called "smartphone"). In some aspects, the device also includes at least one camera for capturing one or more image or video frames. For example, the device may include one or more cameras (e.g., an RGB camera) for capturing one or more images and / or one or more videos including video frames. In some aspects, the device includes a display for displaying one or more images, videos, notifications, or other displayable material. In some aspects, the device includes a transmitter configured to transmit reconstructed video frames to at least one device via a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing devices or components.

[0028] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter of the invention should be understood by referring to appropriate portions of the entire specification of this patent, any or all of the accompanying drawings, and each claim.

[0029] The foregoing, as well as other features and embodiments, will become more apparent from the following description, claims, and drawings. Simple Explanation of the Diagram

[0030] The illustrative embodiments of this application are described in detail below with reference to the accompanying drawings:

[0031] Figure 1 illustrates, based on some examples of this disclosure, an example image processing system that can implement the various techniques described herein;

[0032] Figure 2A illustrates an example of a fully connected neural network according to some examples of this disclosure;

[0033] Figure 2B illustrates an example of a locally connected neural network according to some examples of this disclosure;

[0034] Figure 2C illustrates examples of convolutional neural networks according to some examples of this disclosure;

[0035] Figure 2D illustrates detailed examples of deep convolutional networks (DCNs) designed to identify features from images, according to some examples of this disclosure;

[0036] Figure 3 is a block diagram illustrating another exemplary DCN according to some examples of this disclosure;

[0037] Figure 4 is a diagram illustrating examples of unidirectional prediction frame (P-frame) decoding systems configured to perform unidirectional motion compression and compensation to generate predictions, based on some examples of the present disclosure.

[0038] Figure 5A is a diagram illustrating examples of bidirectional predictive frame (B-frame) decoding systems configured to perform bidirectional motion compression and compensation to generate predictions, according to some examples of the present disclosure;

[0039] Figure 5B is a diagram illustrating examples of B-frame decoding systems configured to use frame interpolation to generate predictions, based on some examples of the present disclosure.

[0040] Figure 6 is a diagram illustrating examples of B-frame decoding systems configured to perform learned B-frame decoding using a P-frame decoding system and a frame interpolation engine, according to some examples of the present disclosure;

[0041] Figure 7A is a diagram illustrating examples of P-frame decoding systems that can be used to perform one-way and / or two-way decoding, according to some examples of the present disclosure;

[0042] Figure 7B is a diagram illustrating examples of B-frame decoding systems that can be used to perform bidirectional decoding using a P-frame decoding system, according to some examples of the present disclosure;

[0043] Figure 7C is a diagram illustrating examples of frame interpolation engines that can be used by a B-frame decoding system, based on some examples of the present disclosure.

[0044] Figure 8 is a diagram illustrating examples of motion estimation techniques according to some examples of the present disclosure;

[0045] Figure 9 is a diagram illustrating examples of I-frame, B-frame, and I-frame (IBI) decoding techniques according to some examples of the present disclosure;

[0046] Figure 10 is a diagram illustrating examples of I-frame, B-frame, and P-frame (IBP) decoding techniques according to some examples of the present disclosure;

[0047] Figure 11 is a diagram illustrating example ordering schemes for bidirectional frame prediction, based on some examples of the present disclosure;

[0048] Figure 12 is a diagram illustrating an example hierarchical scheme for bidirectional frame prediction, based on some examples of the present disclosure.

[0049] Figures 13A and 13B are graphs illustrating the performance results of the B-frame decoding system described herein, based on some examples of the present disclosure.

[0050] Figures 14A-14F are graphs illustrating further performance results of the B-frame decoding system described herein, based on some examples of the present disclosure.

[0051] Figure 15 is an example flowchart illustrating a process for processing video data, based on some examples of this disclosure;

[0052] Figure 16 is a flowchart illustrating another example of processing for processing video data, based on some examples of this disclosure; and

[0053] Figure 17 shows an example computing system that can be used to implement the various aspects described in this paper. Implementation

[0054] Certain aspects and embodiments of this disclosure are provided below. It will be apparent to those skilled in the art that some of these aspects and embodiments can be applied independently, and some can be applied in combination. In the following description, specific details are set forth for ease of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting.

[0055] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments is intended to provide enabling descriptions for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0056] Digital video data can encompass large amounts of data, especially with the continued growth in demand for high-quality video data. For example, video data consumers typically expect increasingly higher video quality, including high fidelity, resolution, and frame rate. However, meeting these demands often requires large amounts of video data, placing a significant burden on the communication networks and equipment used to process and store this data.

[0057] Various technologies can be used to decode video data. In some cases, video decoding can be performed according to specific video decoding standards and / or schemes. Example video decoding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding, and Universal Video Coding (VVC). One goal of video decoding technology is to compress video data into a form using a lower bit rate while avoiding or minimizing video quality degradation. With the increasing demand for video services and the emergence of new video services, there is a need for decoding technologies with higher efficiency and performance.

[0058] Video decoding can use prediction methods such as intra-frame prediction or inter-frame prediction, which utilize redundancy present in video frames or other image or frame sequences. Intra-frame prediction is performed using data within a single video frame and is based on the spatial characteristics of the frame. A frame encoded using intra-frame prediction is called an I-frame. Inter-frame prediction of a frame is performed based on the temporal characteristics of that frame relative to other frames. For example, inter-frame prediction of a video frame can be performed by identifying regions of other video frames that vary relative to the video frame and regions that are redundant relative to the video frame (e.g., a substantially unchanged background region). Redundancy can be removed, resulting in residuals of the video frames. The residuals can be further encoded (e.g., using entropy decoding), and the results can be included in a bitstream stored, transmitted, or otherwise output.

[0059] Examples of inter-frame prediction include uni-prediction and bi-prediction. Uni-prediction involves using a single reference frame when performing inter-frame prediction. Frames decoded using uni-prediction are called P-frames. Bi-prediction involves using two reference frames when performing inter-frame prediction. Frames encoded using bi-prediction are called B-frames.

[0060] In some cases, machine learning systems can be used to perform video encoding (compression) and decoding (decompression). Generally, machine learning (ML) is a subset of artificial intelligence (AI). ML systems include algorithms and statistical models that computer systems can use to perform various tasks through dependency patterns and reasoning. An example of an ML system is a neural network (also called an artificial neural network), which can consist of a set of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in a variety of applications and / or devices, such as image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, and more.

[0061] In a neural network, individual nodes mimic biological neurons by taking input data and performing operations on that data. The results of the operations performed on the input data are selectively passed to other neurons. Weights are associated with each vector and node in the network, and these values ​​constrain how input and output data are associated. For example, the input data of each node can be multiplied by its corresponding weight value, and the products can be summed. The sum of the products can be adjusted by optional biases, and a priming function can be applied to the result to produce the node's output signal or "output priming" (sometimes called a priming map or feature map). These weights can initially be determined by an iterative stream of data used to train the network. For example, weights can be established during training phases where the network learns how to identify specific categories based on the characteristics of its typical input data. In one example, the network can be trained to learn a specific task by adjusting the parameter values ​​associated with neurons (e.g., priming parameters and / or weights, biases, etc.), adding and / or removing neurons or even layers of neurons, adding or deleting edges between neurons, etc.

[0062] Different types of neural networks exist, such as autoencoders, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and multilayer perceptron (MLP) neural networks. A CNN can consist of an ensemble of artificial neurons, each with a receptive field (e.g., a localized region of the input space), which collectively tile the input space. CNNs have many applications. For example, CNNs are widely used in pattern recognition and classification. RNNs work by storing the output of a layer and feeding that output back to the input to help predict the outcome of the prediction layer. In an MLP neural network, data is fed into the input layer, and one or more hidden layers provide an abstraction level to the data. The output layer can then make predictions based on this abstraction. MLPs may be particularly suitable for classification prediction problems where the input is assigned a category or label.

[0063] In a hierarchical neural network architecture (called a deep neural network when there are multiple hidden layers), the output of the first layer's artificial neurons becomes the input of the second layer's artificial neurons, the output of the second layer's artificial neurons becomes the input of the third layer's artificial neurons, and so on. Convolutional neural networks can be trained to recognize hierarchical structures of features. The computation in a convolutional neural network architecture can be distributed across a set of processing nodes, which can be configured in one or more computation chains. These multi-layer architectures can be trained layer by layer and fine-tuned using backpropagation.

[0064] In some examples, machine learning-based P-frame and B-frame decoding systems can be used to perform unidirectional and bidirectional predictions, respectively. In some cases, such systems may include neural network architectures (e.g., one or more deep neural networks, such as one or more autoencoders). An example of a machine learning-based P-frame decoding system can perform motion compression and motion compensation on the current frame and a reference frame to determine the motion prediction between the two frames. This motion prediction can be used to modify the pixels of the reference frame (e.g., by moving the pixels of the reference frame according to the motion vectors included in the motion prediction) to produce a prediction for the current frame. The residual portion of the P-frame decoding system can generate a prediction residual representing the difference between the predicted frame and the current frame. The predicted residual can be combined with the prediction to generate a reconstructed current frame.

[0065] Various types of machine learning-based B-frame decoding systems can be used to perform bidirectional prediction. One example of a machine learning-based B-frame decoding system is similar to the P-frame system described above, but uses two reference frames instead of a single reference frame to generate the reconstructed current frame. Another example of a machine learning-based B-frame decoding system includes a frame interpolation engine that generates predictions based on interpolation between two reference frames. The residual portion then determines the prediction residuals combined with the predictions from the interpolation engine.

[0066] This document describes systems, methods (also called processes), apparatuses, and computer-readable media (collectively, "systems and technologies") for performing bidirectional (e.g., B-frame) decoding using a unidirectional (e.g., P-frame) decoding system to perform learning. As used herein, the term decoding can refer to encoding (e.g., compression), decoding (e.g., decompression), or both. For example, a bidirectional decoding system (also called a B-frame decoding system) is provided, comprising a frame interpolation engine for generating interpolated reference frames based on at least two reference frames. In some examples, the frame interpolation engine may use motion information (e.g., optical flow or other motion-based techniques) associated with the reference frames used to generate the interpolated reference frames to generate the interpolated reference frames.

[0067] A bidirectional decoding system can use a unidirectional decoding system (also known as a P-frame decoding system) to generate bidirectional prediction frames (B-frames). For example, a unidirectional decoding system can use an interpolated reference frame to generate a B-frame. A B-frame can include a frame generated by referencing data from previous and future frames (e.g., frames before and after the B-frame in the display order and / or decoding order). In addition to being used for B-frame decoding, a unidirectional decoding system can also be used to generate unidirectional prediction frames (P-frames) when performing a unidirectional prediction mode on a particular frame. For example, a unidirectional decoding system can use an actual reference frame to generate a P-frame. A P-frame can include a frame generated by referencing data from a single reference frame, which can include a previous frame (e.g., a frame before the B-frame in the decoding order) or a future frame (e.g., a frame after the B-frame in the decoding order). As used herein, the terms frame, picture, and image are used interchangeably.

[0068] In some examples, a one-way decoding system may include a video encoder and a video decoder that can use machine learning to process video data and generate predictions for input frames. In some aspects, the video encoder may include an encoder neural network of a motion prediction system (e.g., a motion autoencoder), and the video decoder may include a decoder neural network of a motion prediction system. For example, the video encoder (e.g., the encoder neural network of a motion prediction system) may use machine learning to generate latent data (e.g., a bitstream) representing the predicted motion between an input frame and a reference frame (e.g., an interpolated reference frame or a real reference frame, such as a reconstructed P or B frame or I frame not based on interpolation when performing bidirectional prediction). The video decoder (e.g., the decoder neural network of a motion prediction system) may use machine learning to process the latent data and predict or reconstruct the motion between the input frame and the reference frame. The one-way decoding system may use the predicted motion to generate predictions for the input frame by deforming the corresponding portions of the reference frame. For example, the predicted motion may include motion vectors (e.g., an optical flow map including motion vectors for each pixel or pixel block of the input frame). The morphing engine can move the pixels of a reference frame based on motion vectors to generate a prediction of the input frame (called a morphing frame).

[0069] In some examples, the video encoder and video decoder of a one-way decoding system can also use machine learning to generate residuals of the input frames. In some aspects, a one-way decoding system may include a residual prediction system (e.g., a residual autoencoder) comprising encoder neural networks and decoder neural networks that can generate residuals for the input frames. For example, the video encoder (e.g., the encoder neural network of a residual prediction system) may receive residuals as input, which include the differences between the input frame and a reference frame (e.g., an interpolated reference frame when performing bidirectional prediction, or the actual reference frame when performing one-way prediction). Using the residuals as input, the video encoder can use machine learning to generate latent data (e.g., a bitstream) representing the residuals. The video decoder (e.g., the decoder neural network of a residual prediction system) can use machine learning to process the latent data and predict or reconstruct the residuals. The one-way decoding system can generate reconstructed frames representing the input frames by adding the predicted residuals to the predictions generated for the input frames (e.g., generated by a warp engine).

[0070] In some examples, the system can transmit latent data representing motion and latent data representing residuals (e.g., in an encoded video bitstream) to another device for decoding. For example, the other device may include a video decoder (e.g., the motion prediction system and residual prediction system described above), which can use machine learning to reconstruct the input frame based on the latent data representing motion and latent data representing residuals, similar to what is described above (e.g., predicting motion between the input and reference frames by processing the motion latent data, generating a prediction of the input frame by deforming the reference frame according to the predicted motion, predicting residuals by processing the residual latent data, and generating a reconstructed frame by adding the predicted residuals to the prediction generated for the input frame).

[0071] Other aspects of the system and techniques will be described with reference to the accompanying drawings. Figure 1 illustrates an example implementation of an image processing system 100, which in some cases can be used to implement the systems and techniques described herein. The image processing system 100 may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Information such as parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., weighted neural networks), latency, frequency information, task information, and image data may be stored in memory blocks associated with a neural processing unit (NPU) 108, in memory blocks associated with the CPU 102, in memory blocks associated with a graphics processing unit (GPU) 104, in memory blocks associated with a digital signal processor (DSP) 106, in memory block 118, and / or may be distributed across multiple blocks. Instructions executed at CPU 102 can be loaded from program memory associated with CPU 102 and / or from memory block 118.

[0072] The image processing system 100 may also include other processing blocks for performing specific functions, such as a GPU 104; a DSP 106; a connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.); and / or a multimedia processor 112 that may, for example, detect image features. In some examples, an NPU 108 may be implemented in the CPU 102, DSP 106, and / or GPU 104. In some cases, the image processing system 100 may also include one or more sensors 114, one or more image signal processors (ISPs) 116, and / or a storage device 120.

[0073] In some examples, the image processing system 100 may implement an ARM instruction set architecture for one or more processors. In one aspect of this disclosure, the instructions loaded into the CPU 102 may include code for searching a stored multiplication result corresponding to the product of the input value and filter weights in a lookup table (LUT). The instructions loaded into the CPU 102 may also include code for disabling the multiplier during the multiplication operation of the product when a lookup table hit is detected. Furthermore, the instructions loaded into the CPU 102 may include code for storing the computed product of the input value and filter weights when a lookup table miss is detected.

[0074] Image processing system 100 may be part of a computing device or a plurality of computing devices. In some examples, image processing system 100 may be part of an electronic device (or a plurality of electronic devices), such as a camera system (e.g., a digital camera, IP camera, camcorder, security camera, etc.), a telephone system (e.g., a smartphone, cellular phone, conferencing system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smartwatch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a system-on-a-chip (SoC), a digital media player, a game console, a video streaming device, a server, a drone, a computer in a car, an Internet of Things (IoT) device, or any other suitable electronic device.

[0075] In some implementations, CPU 102, GPU 104, DSP 106, NPU 108, connectivity block 110, multimedia processor 112, one or more sensors 114, ISP 116, memory block 118, and / or storage device 120 may be part of the same computing device. For example, in some cases, CPU 102, GPU 104, DSP 106, NPU 108, connectivity block 110, multimedia processor 112, one or more sensors 114, ISP 116, memory block 118, and / or storage device 120 may be integrated into a smartphone, laptop computer, tablet computer, smart wearable device, video game system, server, and / or any other computing device. In other embodiments, CPU 102, GPU 104, DSP 106, NPU 108, connectivity block 110, multimedia processor 112, one or more sensors 114, ISP 116, memory block 118, and / or storage device 120 may be part of two or more separate computing devices.

[0076] Image processing system 100 and / or its components may be configured to perform video compression and / or decompression (also known as video encoding and / or decoding, collectively referred to as video encoding) using techniques according to various aspects of this disclosure discussed herein. By using deep learning architectures and the techniques described herein to perform video compression and / or decompression, various aspects of this disclosure can improve the efficiency of video compression and / or decompression on the device and / or reduce associated resource requirements and / or usage. For example, a device using the video decoding techniques described herein can compress video data more efficiently, reduce the amount of data transmitted to the destination device in compressed video data, and the destination device can efficiently receive and decompress the compressed video data. In some examples, the deep learning architectures and techniques described herein can reduce the amount of data exchanged between decoding devices or components (e.g., encoders and decoders) to decode video content. The reduced amount of data transmitted for video decoding can reduce latency, improve performance, and reduce the cost or burden of computing resources (e.g., bandwidth, memory, storage, power, computing, hardware, etc.).

[0077] As mentioned above, neural networks are examples of machine learning systems and can include an input layer, one or more hidden layers, and an output layer. Information is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is produced by output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include feature maps or priming maps, where these maps can include artificial neurons (or nodes). Feature maps can include filters, kernels, etc. Nodes can include one or more weights used to indicate the importance of nodes in one or more layers. In some cases, deep learning networks can have a series of many hidden layers, with earlier layers used to determine simple and low-level features of the input, while later layers build a hierarchy of more complex and abstract features.

[0078] Deep learning architectures can learn hierarchical structures of features. For example, if visual data is presented, the first layer can learn to identify features (e.g., edges) in the input stream. In another example, if auditory data is presented, the first layer can learn to identify spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize features, such as the shape of the visual data or the sound combinations of the auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects and / or spoken phrases.

[0079] Neural networks can be designed with various connection patterns. In feedforward networks, information is passed from lower to higher layers, and each neuron in a given layer communicates with neurons in higher layers. As mentioned above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output of a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across multiple blocks of input data that are passed to the neural network sequentially. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections are helpful when the recognition of high-level concepts can help distinguish specific low-level features of the input.

[0080] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, neurons in a first hidden layer can transmit their outputs to each neuron in a second hidden layer, such that each neuron in the second layer will receive input from each neuron in the first layer. Figure 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, neurons in a first hidden layer can connect to a limited number of neurons in a second hidden layer. More generally, the locally connected layers of a locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connectivity strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of locally connected networks can produce spatially different receptive fields in higher layers because neurons in higher layers in a given region can receive inputs with properties tuned to a restricted portion of the network's total input through training.

[0081] An example of a locally connected neural network is a convolutional neural network. Figure 2C illustrates an example of a convolutional neural network 206. Convolutional neural network 206 can be configured such that the connection strength associated with the input of each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful. According to various aspects of this disclosure, convolutional neural network 206 can be used to perform one or more aspects of video compression and / or decompression.

[0082] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2D shows an example of a DCN 200, which is designed to recognize features of an image 226 input from an image capturing device 230 (e.g., a camera or image sensor). In some examples, the DCN 200 of the current example can be trained to recognize visual features in the image 226 (e.g., one or more objects or signs in the image 226).

[0083] In some examples, supervised learning can be used to train DCN 200. During training, an image (e.g., image 226) can be presented to DCN 200, and the forward pass can then be computed to produce output 222. DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226, convolutional layer 232 can apply a convolutional kernel (not shown) to image 226 to generate a first set of feature maps 218. For example, the convolutional kernel of convolutional layer 232 can be a 5x5 kernel, which generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 in convolutional layer 232. A convolutional kernel can also be called a filter or a convolutional filter.

[0084] The first set of feature maps 218 can be resampled by a max-pooling layer (not shown) to generate the second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14x14) is smaller than the size of the first set of feature maps 218 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved by one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0085] In the example of Figure 2D, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a label corresponding to a possible feature of image 226, such as "symbol", "60", and "100". A softmax function (not shown) can convert the labels in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 is the probability that image 226 includes one or more features.

[0086] In this example, the probabilities of "sign" and "60" in output 222 are higher than those of other labels in output 222 (e.g., "30", "40", "50", "70", "80", "90", and "100"). Before training, the output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the fundamental fact of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that the output 222 of DCN 200 is closer to the target output.

[0087] To adjust the weights, the learning algorithm can compute the gradient vector of the weights. The gradient indicates how much the error will increase or decrease when the weights are adjusted. At the top layers, the gradient can directly correspond to the weight values ​​connecting the neurons activated in the penultimate layer to the neurons in the output layer. In lower layers, the gradient can depend on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights can be called "backpropagation" because it involves "passing backward" through the neural network.

[0088] In practice, the error gradient of the weights can be computed on a small number of examples, making the computed gradient close to the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches the target level. After learning, a new image can be presented to the DCN, and the forward pass of the network can produce an output that can be considered an inference or prediction of the DCN.222

[0089] Deep belief networks (DBNs) are probabilistic models containing multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking Restricted Boltzmann Machine (RBM) layers. An RBM is an artificial neural network that learns a probability distribution for a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be assigned to, they are often used for unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of the previous layer's inputs and the target class) and used as a classifier.

[0090] Deep convolutional networks (DCNs) are networks of convolutional networks configured with additional pooling and normalization layers. DCNs can achieve high performance on many tasks. DCNs can be trained using supervised learning, where the input and output targets are known for many examples and are used to modify the network weights using gradient descent.

[0091] DCNs can be feedforward networks. Furthermore, as mentioned above, neurons in the first layer share connections from neurons in the first layer of the DCN to a set of neurons in the next higher layer. The feedforward and shared connections of DCNs can be used for fast processing. For example, the computational burden of a DCN may be much less than that of a similarly sized neural network containing recurrent or feedback connections.

[0092] Each layer of a convolutional network can be considered as processing a spatially invariant template or fundamental projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be considered three-dimensional, with two spatial dimensions along the image axes and a third dimension capturing color information. The output of the convolutional connections can be viewed as forming feature maps in subsequent layers, where each element of the feature map (e.g., 220) receives input from a series of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed nonlinearly (e.g., corrected, max(0,x)). Values ​​from neighboring neurons can be further merged, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0093] Figure 3 is a block diagram illustrating an example of a deep convolutional network 350. The deep convolutional network 350 may include multiple layers of different types based on connectivity and weight sharing. As shown in Figure 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0094] Convolutional layer 356 may include one or more convolutional filters that can be applied to input data 352 to generate feature maps. Although only two convolutional blocks 354A and 354B are shown, this disclosure is not limited thereto, and any number of convolutional blocks (e.g., blocks 354A and 354B) may be included in the deep convolutional network 350 according to design preferences. Normalization layer 358 may normalize the output of the convolutional filters. For example, normalization layer 358 may provide whitening or lateral suppression. Max pooling layer 360 may provide spatial downsampling aggregation for local invariance and dimensionality reduction.

[0095] For example, the parallel filter bank of the deep convolutional network can be loaded onto the CPU 102 or GPU 104 of the image processing system 100 to achieve high performance and low power consumption. In some examples, the parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the image processing system 100. The deep convolutional network 350 can access other processing blocks that may exist on the image processing system 100.

[0096] The deep convolutional network 350 may include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may include a logistic regression (LR) layer 364. Weights (not shown) to be updated are located between each layer 356, 358, 360, 362, and 364 of the deep convolutional network 350. The output of each layer (e.g., 356, 358, 360, 362, and 364) can be used as input to subsequent layers (e.g., 356, 358, 360, 362, and 364) in the deep convolutional network 350 to learn hierarchical feature representations from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) provided at the first convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is the probability that the input data includes features from a set of features.

[0097] Another type of neural network is the autoencoder. An autoencoder can be trained (e.g., using training data and one or more loss functions) to receive input and generate a version of that input at its output (e.g., essentially copying its input to its output). Autoencoders can be trained to learn efficient data encoding in an unsupervised manner. For example, given an image of an object, an autoencoder can first encode the image into a lower-dimensional latent representation, and then decode the latent representation back into an image of the object. An autoencoder can learn (through training) to compress input data while minimizing reconstruction error.

[0098] As mentioned above, digital video data can include a large amount of data, which can place a significant burden on the communication networks and devices that process and store video data. For example, recording uncompressed video content typically results in a substantial increase in file size as the resolution of the recorded video content increases. In an illustrative example, uncompressed 16-bit per channel video recorded at 1080p / 24 (e.g., a resolution of 1920 pixels wide and 1080 pixels high, capturing 24 frames per second) might occupy 12.4 megabytes per frame, or 297.6 megabytes per second. Uncompressed 16-bit per channel video recorded at 4K resolution at 24 frames per second might occupy 49.8 megabytes per frame, or 1195.2 megabytes per second.

[0099] Network bandwidth is another constraint that can make large video files problematic. For example, video content is often transmitted over wireless networks (e.g., LTE, LTE-Advanced, New Radio (NR), WiFi™, Bluetooth™, or other wireless networks) and can constitute a large portion of consumer internet traffic. Therefore, it is desirable to reduce the amount of bandwidth used to transmit video content over these networks.

[0100] Because uncompressed video content can result in large files with considerable memory requirements for physical storage and significant bandwidth requirements for transmission, video decoding techniques can be used to compress and decompress such video content, as further described herein. To reduce the size of video content—and therefore the amount of storage involved in storing video content and the amount of bandwidth involved in transmitting video content—various video decoding techniques can be performed according to specific video decoding standards and / or schemes (e.g., HEVC, AVC, MPEG, VVC, etc.). Video decoding can use prediction methods such as inter-frame prediction or intra-frame prediction, which take advantage of redundancy present in video images or sequences. One goal of video decoding techniques is to compress video data into a form using a lower bit rate while avoiding or minimizing video quality degradation. With the increasing demand for video services and the emergence of new video services, there is a need for decoding techniques with higher decoding efficiency, performance, and bitrate control.

[0101] Encoding devices can encode video data according to video decoding standards to generate an encoded video bitstream. In some cases, the encoded video bitstream (or "video bitstream" or "bitstream") may include a series of one or more encoded video sequences. Encoding devices can generate an encoded representation of a picture by dividing each picture into multiple slices. Slices are independent of other slices, so information in a slice is decoded without relying on data from other slices within the same picture. A slice consists of one or more segments, including independent segments and one or more dependent segments (if present) that depend on previous segments. In HEVC, slices are divided into decoder tree blocks (CTBs) for luma and chroma samples. The CTB of the luma samples and one or more CTBs of the chroma samples, along with the sample syntax, are called decoder tree units (CTUs). CTUs can also be called "tree blocks" or "maximum decoder units" (LCUs). CTUs are the basic processing unit of HEVC encoding. A CTU can be subdivided into multiple decoder units (CUs) of different sizes. The CU contains an array of luminance and chrominance samples called a decoder block (CB).

[0102] The luma and chroma CBs can be further decomposed into prediction blocks (PBs). A PB is a sample block of the luma or chroma component that uses the same motion parameters for inter-frame prediction or intra-block copy (IBC) prediction (when available or enabled). The luma PB and one or more chroma PBs, together with the associated syntax, form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) are signaled in the bitstream of each PU, and these parameters are used for inter-frame prediction of the luma PB and one or more chroma PBs. Motion parameters can also be referred to as motion information. CBs can also be divided into one or more transform blocks (TBs). A TB represents a square block of samples of the chroma component on which a residual transform (e.g., in some cases the same two-dimensional transform) is applied to decode the prediction residual signal. A transform unit (TU) represents a TB of luma and chroma samples and the corresponding syntax elements. Transform decoding is described in more detail below.

[0103] According to the HEVC standard, a Transform Unit (TU) can be used to perform the transform. The size of the TU can be determined based on the size of the PU within a given Cubic Unit (CU). The TU can be the same size as or smaller than the PU. In some cases, a quadtree structure called a Residual Quadtree (RQT) can be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT can correspond to TUs. The pixel differences associated with the TU can be transformed to produce transform coefficients. These transform coefficients can then be quantized by an encoding device.

[0104] Once the video data is segmented into Units (CUs), the encoding device uses a prediction mode to predict each Processing Unit (PU). The prediction unit or block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled within the bitstream using syntax data. Prediction modes can include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction utilizes the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture, such as using DC prediction to find the average value of the PU, using planar prediction to fit a planar surface to the PU, using orientation prediction to infer from adjacent data, or any other suitable type of prediction. Inter-frame prediction uses the temporal correlation between pictures to derive motion-compensated predictions for blocks of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (in output order before or after the current picture) using motion-compensated prediction. For example, at the CU level, a decision can be made as to whether to use inter-picture prediction or intra-picture prediction to decode a picture region.

[0105] In some examples, a slice type is assigned to one or more slices of an image. Slice types can include I-slices, P-slices, and B-slices. An I-slice (intra-frame, independently decodeable) is an image slice encoded solely by intra-frame prediction and is therefore independently decodeable because an I-slice only requires intra-frame data to predict any prediction unit or prediction block of the slice. A P-slice (one-way prediction frame) is a slice of an image that can be decoded using both intra-frame prediction and one-way inter-frame prediction. Each prediction unit or prediction block within a P-slice is decoded using either intra-frame or inter-frame prediction. When inter-frame prediction is applied, the prediction unit or prediction block is predicted using a reference image, and therefore the reference sample comes from a reference region within a frame. A B-slice (two-way prediction frame) is an image slice that can be decoded using both intra-frame and inter-frame prediction (e.g., two-way or one-way prediction). Prediction units or prediction blocks of a B-slice can be predicted bidirectionally from two reference images. Each image can contribute a reference region, and the sample sets of the two reference regions can be weighted (e.g., with equal weights or different weights) to produce the prediction signal for the two-way prediction block. As explained above, each slice of an image is decoded independently. In some cases, an image can be decoded into only one slice.

[0106] After performing prediction using intra-frame and / or inter-frame prediction, the encoding device can perform transform and quantization. For example, after prediction, the encoding device can compute residual values ​​corresponding to the PU. The residual values ​​can include pixel differences between the current pixel block (PU) being decoded and the prediction block used to predict the current block (e.g., a predicted version of the current block). For example, after generating a prediction block (e.g., using inter-frame or intra-frame prediction), the encoding device can generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel differences that quantize the differences between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block can be represented in a two-dimensional block format (e.g., a two-dimensional matrix or an array of pixel values). In such examples, the residual block is a two-dimensional representation of the pixel values.

[0107] Any residual data that may remain after prediction is performed is transformed using a block transform, which can be based on the discrete cosine transform, discrete sine transform, integer transform, wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., sizes of 32 x 32, 16 x 16, 8 x 8, 4 x 4, or other suitable sizes) can be applied to the residual data in each CU. In some embodiments, TUs can be used for transform and quantization processes implemented by the coding device. A given CU having one or more PUs may also include one or more TUs. As described in further detail below, the residual values ​​can be transformed into transform coefficients using block transforms, and then TUs can be used for quantization and scanning to produce serialized transform coefficients for entropy decoding.

[0108] Encoding devices can perform quantization of transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent these coefficients. For example, quantization can reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0109] Once quantization is performed, the encoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partitioning information, and any other suitable data (e.g., other grammatical data). The encoding device can then entropy-encode the different elements of the encoded video bitstream. In some examples, the encoding device can scan the quantized transform coefficients using a predefined scan order to produce a serialized vector that can be entropy-encoded. In some examples, the encoding device can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoding device can entropy-encode that vector. For example, the encoding device can use context-adaptive variable-length decoding, context-adaptive binary arithmetic decoding, grammar-based context-adaptive binary arithmetic decoding, probability interval partitioning entropy decoding, or another suitable entropy coding technique.

[0110] Encoding devices can store encoded video bitstreams and / or transmit encoded video bitstream data to receiving devices via communication links, the receiving devices including decoding devices. Decoding devices can decode encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extraction of elements of one or more encoded video sequences constituting the encoded video data. The decoding devices can then perform rescaling and inverse transform on the encoded video bitstream data. The residual data is then passed to the prediction stage of the decoding devices. The decoding devices then use intra-frame prediction, inter-frame prediction, IBC, and / or other types of prediction to predict pixel blocks (e.g., PUs). In some examples, the prediction is added to the output of the inverse transform (residual data). The decoding devices can output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a content consumer.

[0111] Video decoding systems and technologies defined by various video decoding standards (e.g., the HEVC video decoding technology described above) can retain most of the information in the original video content and can be a priori defined based on signal processing and information theory concepts. However, while various compression algorithms may be universally applicable (e.g., suitable for any type of video content), these video decoding technologies may lack certain characteristics that are beneficial in network-based environments or other types of environments. For example, a video decoding system should be able to achieve low-latency operation. However, video decoding systems and related technologies operating according to existing video decoding standards can encode and transmit video in units of several frames (sometimes called picture groups or GoPs), and may need to wait for a batch of video frames (e.g., GoPs) to accumulate before transmission. This batch-based video decoding results in higher latency compared to the latency achievable using frame-by-frame decoding and transmission.

[0112] In some cases, a machine learning-based P-frame decoding system (e.g., one or more neural networks) can be used to perform uni-prediction. Figure 4 is a diagram illustrating an example of a machine learning-based P-frame decoding system 400. Components of the P-frame decoding system 400 include a motion compression engine 402, a motion compensation engine 404, and a residual compression engine 406. The components of the P-frame decoding system 400 may include one or more neural networks configured to perform uni-prediction using the motion compression engine 402, the motion compensation engine 404, and the residual compression engine 406.

[0113] For example, motion compression engine 402 can obtain input frame 401 and reference frame 403. Motion compression engine 402 can perform machine learning-based motion compression to determine motion prediction 403 between input frame 401 and reference frame. Motion compression engine 402 can provide the predicted motion information to motion compensation engine 404. Motion compensation engine 404 can also receive reference frame 403 as input. Motion compensation engine 404 can use motion prediction to modify the pixels of reference frame and generate prediction for input frame 403. For example, motion compensation engine 404 can move the pixels of reference frame according to the motion vector contained in the motion prediction. Residual compression engine 406 can generate prediction residuals representing the difference between prediction and input frame 401. The prediction residuals can then be combined with the prediction to generate a reconstructed input frame. The reconstructed input frame can be output for storage, display, etc.

[0114] In some cases, a machine learning-based B-frame decoding system can be used to perform bidirectional prediction. Figure 5A is a diagram illustrating an example of a B-frame decoding system 500. Components of the B-frame decoding system 500 include a motion compression engine 502, a motion compensation engine 504, and a residual compression engine 506. The B-frame decoding system 500 may include one or more neural networks configured to perform bidirectional inter-frame prediction (bidirectional prediction) using the motion compression engine 502, the motion compensation engine 504, and the residual compression engine 506. Bidirectional prediction is performed based on two reference frames (represented in Figure 5A as reference frame 0 (Ref 0) and reference frame 1 (Ref 1)) instead of a single reference frame.

[0115] For example, input frame 501 and reference frames Ref 0 and Ref 1 can be provided as input to motion compression engine 502. Motion compression engine 502 can perform machine learning-based motion compression to determine a first prediction of motion between input frame 501 and the first reference frame Ref 0, and a second prediction of motion between input frame 501 and the second reference frame Ref 1. Motion compensation engine 504 can receive reference frames Ref 0 and Ref 1 and the predicted motion from motion compression engine 502 as input. Motion compensation engine 504 can use the two motion predictions to modify the pixels of reference frames Ref 0 and Ref 1 and generate a prediction for input frame 501. For example, motion compensation engine 504 can move the pixels of the first reference frame Ref 0 according to the first motion prediction, and can move the pixels of the second reference frame Ref 1 according to the second motion prediction. Motion compensation engine 504 can merge these two predictions to generate a merged prediction for input frame 501.

[0116] The residual compression engine 506 can generate prediction residuals representing the difference between the merged prediction and the input frame 501. The prediction residuals can then be combined with the merged prediction to generate a reconstructed input frame, which can be output for storage or display.

[0117] Figure 5B is a diagram illustrating another example of a B-frame decoding system 510. Components of the B-frame decoding system 510 include a frame interpolation engine 512 and a residual compression engine 516. Components of the B-frame decoding system 500 may include one or more neural networks configured to perform bidirectional prediction using the frame interpolation engine 512 and the residual compression engine 516.

[0118] Similar to the B-frame decoding system 500 in Figure 5A, the B-frame decoding system 510 can perform bidirectional prediction based on two reference frames (denoted as reference frame 0 (Ref 0) and reference frame 1 (Ref 1)). However, the interpolation engine 512 can generate a prediction for the input frame 511 by performing interpolation between reference frames Ref 0 and Ref 1. The prediction can then be output to the residual compression engine 516. The residual compression engine 506 can generate a prediction residual representing the difference between the prediction and the input frame 511. The B-frame decoding system 510 can combine the prediction residual with the merged prediction to generate a reconstructed input frame. The B-frame decoding system 510 can output the reconstructed input frame for storage, display, etc.

[0119] The B-frame decoding systems described in Figures 5A and 5B may not be ideal in certain situations. For example, the B-frame decoding system 500 in Figure 5A performs bidirectional motion compression to generate two motion predictions based on two reference frames Ref 0 and Ref 1 and the input frame 511. The B-frame decoding system 500 modifies the reference frames Ref 0 and Ref 1 based on the two motion predictions and then merges the modified frames to generate a prediction for the input frame 511. Such a process involves a great deal of computation during training and inference (after training the machine learning system and using it to process the input video data). Furthermore, the individual B-frame decoding system and P-frame decoding system (Figure 4) (of Figures 5A or 5B) require separate neural networks (with different parameters, such as weights, biases, etc.). Training the machine learning systems 500 and 510 in Figures 5A and 5B also requires a large amount of training data (e.g., two reference frames for each prediction). In addition, the B-frame decoding system 510 in Figure 5B determines an accurate interpolation only under the linear motion assumption, and this prediction may not be aligned with the input frame. In some cases, the B-frame decoding system 510 can use a nonlinear frame interpolator; however, misalignment can still occur in these situations. The B-frame decoding system 510 relies solely on residuals to compensate for misalignment.

[0120] As described above, this paper describes systems and techniques for B-frame decoding using a P-frame decoding system and an interpolated reference frame to perform learning. The systems and techniques described herein can efficiently and intelligently predict bidirectional prediction frames (B-frames) using reference frames interpolated from the reference frames (e.g., using motion information such as optical flow information). These systems and techniques enable efficient, high-quality video decoding with lower resource requirements (e.g., lower computational and bandwidth requirements). For example, when a neural network is used in a P-frame decoding system, the P-frame decoding system requires a set of neural network parameters (e.g., weights, biases, etc.) to perform bidirectional prediction (B-frames) and unidirectional prediction (P-frames). Furthermore, since only a single reference frame (the interpolated reference frame) is used to perform bidirectional prediction, only a single set of motion information is required for bidirectional prediction of each B-frame. In some cases, training a machine learning system (e.g., a neural network) may require less training data than training the B-frame decoding system 500 of Figure 5B.

[0121] B-frame decoding system 600 also provides improved rate-distortion performance, for example, compared to B-frame decoding system 500 of Figure 5A and B-frame decoding system 510 of Figure 5B. For example, for a given input frame, when reference frames from the past (e.g., Ref 0) and the future (e.g., Ref 1) are both available, for example under the linear motion assumption, B-frame decoding system 600 can determine a coarse prediction of input frame 611 by linear interpolation of the two reference frames. This prediction consumes zero bits because the two reference frames are already available and no motion field or other information is needed to perform the interpolation. B-frame decoding system 500 of Figure 5A uses bidirectional motion (e.g., flow or deformation), which does not utilize such information and sends motion information (e.g., optical flow) about the two reference frames (e.g., as if the decoder / receiver is unaware of the object's motion). As mentioned above, regarding B-frame decoding system 510 of Figure 5B, the interpolation result of frame interpolation engine 512 is accurate only under the linear motion assumption. In such a system, the prediction may not be aligned with the input frame. Even when using a nonlinear frame interpolator, misalignment can still occur. In these cases, the decoding system relies solely on residuals to compensate for the misalignment. As a result, the decoding efficiency of this system is significantly reduced compared to the B-frame decoding system 600 in Figure 6, which can use inexpensive side information (e.g., interpolation between reference frames) to correct the misalignment, and then use the residuals to determine the reconstructed input frame.

[0122] Figure 6 is a diagram illustrating an example of a B-frame decoding system 600 configured to perform learning-based B-frame decoding. The B-frame decoding system 600 includes various components for processing video data. As shown, the components of the B-frame decoding system 600 include a frame interpolation engine 622 and a P-frame decoding system 624. Using the frame interpolation engine 622 and the P-frame decoding system 624, the B-frame decoding system 600 can perform bi-directional inter-frame prediction of the input frame 611 to generate a reconstructed version (or decoded version) of the input frame 611, referred to as the reconstructed input frame 615. The frame interpolation engine 622 can generate an interpolated reference frame 613 based on processing reference frame 0 (Ref 0) and reference frame 1 (Ref 1). The frame interpolation engine 622 can utilize any type of interpolation technique. In some examples, the frame interpolation engine 622 can use motion information associated with reference frames Ref 0 and Ref 1 to generate the interpolated reference frame 613. For example, as described with reference to FIG7C, the frame interpolation engine 622 can determine the interpolation reference frame 615 based on bidirectional motion information (e.g., forward optical flow and backward optical flow) determined for reference frames Ref 0 and Ref 1. In some cases, the interpolation reference frame 615 can be stored, for example, as a reference frame for one or more other input frames.

[0123] B-frame decoding system 600 can use P-frame decoding system 624 to generate the reconstructed input frame 615 (as a B-frame). For example, as shown in FIG. 6, P-frame decoding system 624 can use interpolated reference frame 613 as input to generate the reconstructed input frame 615. P-frame decoding system 624 can use any type of P-frame decoding technique. In some examples, P-frame decoding system 624 may include at least one machine learning-based video encoder and at least one machine learning-based video decoder to process interpolated reference frame 613 and input frame 611 to generate the reconstructed input frame 615. In some embodiments, as described below with respect to FIG. 7A and FIG. 7B, P-frame decoding system 624 may include a motion neural network system having an encoder neural network and a decoder neural network. The motion neural network system may include any type of neural network structure, such as an autoencoder, a convolutional neural network (CNN), and / or other neural network structures. The encoder neural network of the motion neural network system can generate latent data representing motion data for input frame 611. The decoder neural network of the motor neural network system can reconstruct motion from latent data to generate predicted motion for input frame 611. The P-frame decoding system 624 can use the predicted motion to deform and interpolate reference frame 613 to generate a prediction for input frame 611.

[0124] The P-frame decoding system 624 may also include a residual neural network system comprising an encoder neural network and a decoder neural network. The residual neural network system may include any type of neural network architecture, such as an autoencoder, a CNN, and / or other neural network architectures. The P-frame decoding system 624 may generate a residual for the input frame 611, for example, by determining the difference between the deformed interpolation reference frame 613 (using the predicted motion deformation) and the input frame 611. The encoder neural network of the residual neural network system may process the residual to generate residual latent data representing the residual. The decoder neural network of the residual neural network system may reconstruct the residual from the residual latent data to generate a predicted residual for the input frame 611. The P-frame decoding system 624 may generate a reconstructed input frame 615 by adding the predicted residual to the deformed interpolation reference frame 613.

[0125] In some examples, the P-frame decoding system 624 can be used to perform bidirectional prediction (to generate B-frames) and also to perform unidirectional prediction (to generate P-frames). For example, the P-frame decoding system 624 can be used to generate P-frames when determining whether to perform unidirectional prediction decoding on an input frame, and can be used to generate B-frames when determining whether to perform bidirectional prediction decoding on an input frame. Details regarding the selection of different decoding modes (e.g., intra-frame prediction, unidirectional prediction, or bidirectional prediction) for certain frames are illustrated with reference to Figures 9-12. When a unidirectional prediction mode is selected for an input frame, the P-frame decoding system 624 can process the actual reference frame (e.g., reference frame Ref 0 or reference frame Ref 1) to generate a reconstructed frame as a P-frame. Details regarding unidirectional prediction using the P-frame decoding system 624 are illustrated below with reference to Figure 7A. When a bidirectional prediction mode is selected for an input frame, the P-frame decoding system 624 can process an interpolated reference frame (e.g., interpolated reference frame 613) to generate a reconstructed frame as a B-frame. The details of bidirectional prediction using the P-frame decoding system 624 are described below with reference to Figure 7B.

[0126] Although the B-frame decoding system 600 is shown as including certain components, those skilled in the art will understand that the B-frame decoding system 600 may include more components than those shown in FIG. 6. In some cases, other components of the B-frame decoding system 600 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, the B-frame decoding system 600 may include one or more display devices, one or more other processing engines, one or more image sensors (e.g., cameras), one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), and / or one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communication, such as cellular network communication (e.g., using the Long Term Evolution (LTE) protocol, the 5G / New Radio (NR) protocol, or other cellular protocols), wireless local area network (WLAN) communication (e.g., using the 802.11 Wi-Fi protocol), Bluetooth or other short-range communication, any combination thereof, and / or other communication. Those skilled in the art will understand that the B-frame decoding system 600 may include one or more other software and / or hardware components not shown in FIG. 6.

[0127] In some embodiments, other components of the B-frame decoding system 600 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof and / or may be used to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the B-frame decoding system 600.

[0128] B-frame decoding system 600 may include a computing device or part of a computing device. For example, the computing device may include a mobile or landline phone (e.g., a smartphone, cellular phone, etc.), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, and / or a mixed reality device), an image capturing device (e.g., a camera), a vehicle, a display device, a server computer (e.g., communicating with a vehicle computing system), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television set, a digital media player, a video streaming device, or any other suitable computing device, or part of such devices.

[0129] Figure 7A is a diagram illustrating an example of a P-frame decoding system 700 that can be used to perform one-way decoding of P-frames and / or two-way decoding of B-frames. The P-frame decoding system 700 is an illustrative example of the P-frame decoding system 624 of Figure 6. The P-frame decoding system 700 includes a motion prediction system 702, a deformation engine 704, and a residual prediction system 706. The motion prediction system 702 and the residual prediction system 706 can include any type of machine learning system (e.g., using one or more neural networks). In one illustrative example, the motion prediction system 702 includes a motion autoencoder, and the residual prediction system 706 includes a residual autoencoder. While the P-frame decoding system 700 of Figure 7A is shown as including certain components, those skilled in the art will understand that the P-frame decoding system 700 can include fewer or more components than those shown in Figure 7A.

[0130] For a given time t, the P-frame decoding system 700 receives the input frame (represented as frame). ) and reference frame Reference frame The previously reconstructed frame generated before time t (such as the hat symbol ""). (As shown in the image). For example, the input frame. It is the current frame at time t, and the reference frame. It can be a past frame reconstructed or generated at time tn (e.g., at time t in the input frame). (in terms of previous time or sequence), where n is an integer value greater than or equal to 1. In an illustrative example, refer to the frame. It can be immediately following the input frame in time or sequence. Previous frames, in this case, reference frames It can be represented as In some cases, reference frames are received from the decoded image buffer (DPB) of the decoding system. .

[0131] In some cases, the input frame It can be a P-frame or a reference frame. It can be an I-frame, P-frame, or B-frame. For example, a reference frame. The reference frame can be reconstructed or generated by an I-frame decoding system (e.g., the I-frame decoding system may be part of a device including a P-frame decoding system 700, or a device different from the device including the P-frame decoding system 700), reconstructed or generated by a P-frame decoding system 700 (or a P-frame decoding system of a device different from the device including the P-frame decoding system 700), or reconstructed or generated by a B-frame decoding system (e.g., B-frame decoding system 600 of FIG. 6, B-frame decoding system 720 of FIG. 7B, or a B-frame decoding system of a device different from the device including the P-frame decoding system 700). In some cases, the reference frame can be selected based on a set of picture (GoP) structures shown in FIG. 9 or FIG. 10 and / or based on a sequence scheme (e.g., the sequence scheme shown in FIG. 11) or a layering scheme (e.g., the layering scheme shown in FIG. 12), or based on any other scheme. .

[0132] Motion prediction system 702 receives reference frames and input frame The motion prediction system 702 can determine the reference frame. pixels and input frame The motion between pixels is represented (e.g., by vectors, such as optical flow motion vectors). The motion prediction system 702 can then encode the motion and, in some cases, decode it into the input frame. Predicted movement For example, the encoder network 705 of the motion prediction system 702 can determine the current frame. and reference frame The motion between. The encoder network 705 can encode the motion into a latent representation (represented as latent). In some examples, the encoder network 705 can map motion to a potential source. The latent code. In some cases, the encoder network 705 can perform entropy decoding on the latent code to obtain the latent code. Converted to a bitstream. In some examples, encoder network 705 can quantize the latent... (For example, before performing entropy decoding on the latent code). Quantized latent... It can include residuals Quantitative representation. In some cases, potential It may include neural network data representing the quantized code (e.g., a startup graph or feature map of neural network nodes). The encoder network 705 may store potential... , will potential The decoder network 703 sends data to the motion prediction system 702, and / or can potentially... Send to a device that can decode potential Another device or system. Upon receiving a potential... At that time, decoder network 703 can detect potential... Decoding (e.g., inverse entropy decoding, dequantization, and / or reconstruction) is performed to generate a reference frame. pixels and input frame Predicted motion between pixels The encoder network 705 and decoder network 703 can be trained and optimized using training data (e.g., training images or frames) and one or more loss functions, as described below.

[0133] In some cases, the predicted motion This can include optical flow data (e.g., an optical flow graph including one or more displacement vectors), dynamic convolutional data (e.g., a matrix or kernel used for data convolution), or block-based motion data (e.g., motion vectors for each block), as described below with respect to Figure 8. For example, in some cases, the predicted motion... It may include an optical flow graph. In an illustrative example, the optical flow graph may include an input frame. The motion vector for each pixel (e.g., the first motion vector for the first pixel, the second motion vector for the second pixel, and so on). The motion vector represents the current frame. The pixels in the reference frame The movement of the corresponding pixels in the image.

[0134] The morphing engine 704 of the P-frame decoding system can obtain the motion prediction output by the motion prediction system 702. For example, the morphing engine 704 can retrieve motion predictions from storage. Alternatively, motion predictions can be received from the motion prediction system 702. The 704 Transformation Engine can use motion prediction. To deform (e.g., by performing motion compensation) the reference frame The pixels cause the generation of deformed frames. For example, reference frame The pixels can be moved to a new location or based on the reconstructed motion estimation data. Modifications were made. Optical flow was used to determine the reference frame. pixels and input frame In an illustrative example of motion between pixels, the morphing engine 704 can use a reference frame. The amount indicated by the motion vectors in the pixel-moving optical flow map produces deformed frames. As mentioned above, other motion-based techniques can also be used to determine motion predictions. , with deformed reference frame (For example, dynamic convolutional data, block-based motion data, etc.).

[0135] As mentioned above, in order to generate deformed frames The P-frame decoding system 700 can predict... and Movement between (e.g., optical flow or other motion) and use motion prediction Deformation reference frame To perform motion compensation. However, based on motion prediction Frame prediction It may not be accurate enough to capture the input frame. Represented as reconstructed frames For example, the input frame The scene being depicted may contain one or more occluded areas, over-illumination, under-illumination, or factors that could affect frame prediction. Not accurate enough to be used as a reconstruction frame Other effects.

[0136] The residual prediction system 706 can be used for correction. Predict and generate representation frames Reconstructed frames As shown in Figure 7A, it can be obtained from the input frame. Subtract the predicted frame (determined using subtraction operation 708) from the middle. Determine the residual For example, once the deformation engine 704 determines the predicted frame... The P-frame decoding system 700 can determine the predicted frame. and input frame The difference between them (using subtraction operation 708) is used to determine the residual. The encoder network 707 of the residual prediction system 706 can predict the residuals. Encoding to represent residuals potential In some examples, encoder network 707 can detect residuals. Mapping to what can be used as a potential The latent code. In some cases, encoder network 707 can perform entropy decoding on the latent code to obtain the latent code. Converted to a bitstream. In some examples, encoder network 707 can also quantize the latent... (For example, before performing entropy decoding). The potential of quantization. It can include residuals Quantitative representation. In some cases, potential It may include neural network data representing the quantized code (e.g., a startup graph or feature map of neural network nodes). Encoder network 707 may store potential... , will potential The decoder network 709, sent to the residual prediction system 706, and / or can potentially... Send to the device capable of decoding the potential Another device or system. Upon receiving a potential... At that time, decoder network 709 can detect potential... Decoding (e.g., inverse entropy decoding, dequantization, and / or reconstruction) is performed to generate the prediction residuals. The encoder network 707 and decoder network 709 can be trained and optimized using training data (e.g., training images or frames) and one or more loss functions, as described below.

[0137] Predicted residuals It can be used with frame prediction from the morphing engine 704. Used together to generate an input frame representing time t. Reconstructed frames For example, the P-frame decoding system 700 can predict residuals. Add (using addition operation 710) to frame prediction To generate reconstructed frames In some cases, the decoder network 709 of the residual prediction system 706 can predict the residuals. Add to frame prediction Reconstructing frames This can also be called a decoded frame. It can output the reconstructed current frame. For storage (e.g., in a decoded image buffer (DPB) or other storage device), transmission, display, for further processing (e.g., as a reference frame in further inter-frame prediction, for post-processing, etc.), and / or for any other purpose.

[0138] As mentioned above, in some cases, the P-frame decoding system 700 can decode the latent data representing motion (latent... ) and represent the residuals (potential) in the bit stream The potential data is sent to another device for decoding. For example, the other device may include devices capable of decoding from the potential data. Decoding Predictive Motion From potential Decoding prediction residual And it can be based on predicted motion and predicted residuals Generate reconstructed frames (For example, by predicting residuals) Add to using predictive motion Frame prediction generated The video decoder (e.g., motion prediction system 702 and residual prediction system 706, as described above, or configured to decode potential) and Another decoder).

[0139] In some examples, training data and one or more loss functions can be used to train and optimize the motion prediction system 702 and / or the residual prediction system 706. In some cases, the motion prediction system 702 and / or the residual prediction system 706 can be trained end-to-end (where all neural network components are trained in the same training process). In some aspects, the training data may include training images or frames. In some cases, the loss function Loss can be used to train the motion prediction system 702 and / or the residual prediction system 706 by processing the training images or frames. In one example, Loss can be defined as Loss = D + βR, where D is the frame (e.g., the input frame). ) and its reconstructed frame (e.g., D(frame) , frame The distortion between )) is β, which is a hyperparameter controlling the bit rate (bits per pixel), and R is used to convert the residual (e.g., residual) into the distortion between )) and )). Converted to a compressed bitstream (e.g., potential) The number of bits in the P-frame decoding system 700. In some cases, distortion D can be calculated based on peak signal-to-noise ratio (PSNR), structural similarity index measurement (SSIM), multi-scale SSIM (MS-SSIM), etc. Using training data and a loss function, the parameters (e.g., weights, biases, etc.) of the motion prediction system 702 and / or residual prediction system 706 can be adjusted until the P-frame decoding system 700 achieves the desired decoding result.

[0140] Figure 7B is a diagram illustrating an example of a B-frame decoding system 720 that can be used to perform bidirectional decoding using the P-frame decoding system 700 of Figure 7A. The B-frame decoding system 720 also includes a frame interpolation engine 722. Although the B-frame decoding system 720 is shown as including certain components, those skilled in the art will understand that the B-frame decoding system 720 may include fewer or more components than those shown in Figure 7B.

[0141] Interpolation engine 722 can obtain two reference frames, labeled as follows: and As shown in the figure, the P-frame decoding system 700 obtains the input frame at time t. Similar to those described above regarding Figure 7A. Input frame It can be in time or in sequence in the reference frame Then in the reference frame Previous frames. In some examples, the input frame. It can be the current frame at time t, or the reference frame. It can be a past frame at time t-n (e.g., an input frame at time t in time or sequence). (Previously), and reference frame This could be a future frame at time t-n (e.g., in time or sequentially within the input frame). (Then), where n and m can represent the same or different values. In an illustrative example, refer to the frame. It can be the input frame immediately following time t-1. Previous frames, and reference frames It can be immediately following the input frame at time t+1. The following frames.

[0142] In some cases, the input frame It can be a B-frame, and a reference frame. and Each of these can be an I-frame, P-frame, or B-frame. For example, a reference frame. It can be an I-frame, P-frame, or B-frame, with a reference frame. It can be an I-frame, P-frame, or B-frame. In some cases, the reference frame... and These can be frames of the same type, such as B-frames, P-frames, or I-frames. In other cases, the reference frame... and reference frame It can be a different type of frame. For example, a reference frame. It can be an I-frame or a reference frame. It can be a P-frame. In some cases, the reference frame can be selected based on a set of pictures (GoP) structure shown in Figure 9 or Figure 10 and / or based on an order scheme (e.g., the order scheme shown in Figure 11) or a layering scheme (e.g., the layering scheme shown in Figure 12), or based on any other scheme. and .

[0143] Interpolation engine 722 can interpolate two reference frames. and Perform interpolation operations to generate interpolation reference frames. For example, in some cases, interpolation is performed to generate frames that can be used as input frames. The reference frame is the entire reference frame (e.g., not just a portion of the frame, such as a pixel block). Interpolation reference frames can be used. The data is stored and / or output to the P-frame decoding system 700. The frame interpolation engine 722 can use a reference frame. and reference frame Perform any type of interpolation to generate an interpolated reference frame. Examples of interpolation techniques that can be performed by the frame interpolation engine 722 may include, but are not limited to: optical flow-based frame interpolation, gradient-shifting methods, phase-based methods, kernel-based methods, and / or any other frame interpolation techniques. In an illustrative example, to generate an interpolated reference frame... The frame interpolation engine 722 can interpolate reference frames. and reference frame The frame interpolation engine 722 calculates motion estimation information (e.g., optical flow information, such as an optical flow graph, also known as a motion vector graph). The frame interpolation engine 722 can interpolate the motion estimation information (e.g., the flow graph) to determine the interpolation reference frame. The associated motion estimate (e.g., optical flow) at time t. The frame interpolation engine 725 can use interpolated motion estimate information (e.g., interpolated optical flow) and a reference frame. and To generate interpolation reference frames (For example, at least in part through a deformed reference frame) and ).

[0144] In some examples, frame interpolation engine 722 can determine motion estimation information (e.g., optical flow information) pixel by pixel. For example, frame interpolation engine 722 can use a reference frame... (or reference frame) Each pixel in a subset of pixels is mapped to a reference frame. (or reference frame) The position of the corresponding pixel in the corresponding subset of pixels in the reference frame, and vice versa (e.g., in both directions). In one example, motion estimation information (e.g., optical flow information) for each pixel may include information used to indicate the pixel's position in the reference frame. and The vector of movement between (e.g., pixels from a reference frame) Move to reference frame and / or pixels from reference frame Move to reference frame In some cases, it can be based on a reference frame. and An optical flow map is generated by using optical flow vectors between frames. This optical flow map can include the optical flow vector of each pixel in the frame, where each vector indicates the movement of the pixel between frames.

[0145] In an illustrative example, the optical flow vector of a pixel can be a displacement vector (e.g., indicating horizontal and vertical displacement, such as x and y displacement), showing the movement of the pixel from the first frame to the second frame. In some cases, the frame interpolation engine 722 can calculate a reference frame. Two-dimensional (2D) flow graphs (e.g., indicating horizontal and vertical displacement) and reference frames The 2D flow graph. In some respects, the frame interpolation engine 722 can merge 2D flow graphs to generate reference frames for interpolation. A three-dimensional (3D) flow graph. This 3D flow graph may include horizontal and vertical dimensions indicating horizontal and vertical displacements respectively, and frame dimensions including frame indices identifying the frames to which the horizontal and vertical displacements are applied. For example, if the frame size identifies a reference frame... The horizontal and vertical dimensions can then indicate the reference frame. Horizontal and vertical displacement of mid-pixels (e.g., (How many pixels should the pixels in the image move based on the optical flow information).

[0146] The frame interpolation engine 722 can use the interpolated optical flow (e.g., 3D flow graph) to interpolate the reference frame. and reference frame Transform to input frame A specific time step (e.g., time t) can be used to fuse deformed frames to generate an interpolated reference frame. The following section describes further details of this example implementation of the frame interpolation engine using Figure 7C.

[0147] Use interpolated reference frames and input frame As input, the P-frame decoding system 700 can perform the same or similar operations as those described above with respect to Figure 7A to generate the reconstructed frame. For example, the encoder network 705 of the motion prediction system 702 can determine the input frame. and interpolation reference frame The motion between, and can be encoded into the latent In the motion prediction system 702, the decoder network 703 can detect potential... Decode to generate the predicted motion. The 704 transforming engine can use predicted motion. To modify the interpolation reference frame The pixels (e.g., by performing motion compensation) are used to generate frame predictions. As described above with reference to FIG7A, the encoder network 707 of the residual prediction system 706 can obtain the residuals. (determined as frame prediction) With input frame (the difference between them), and can also convert the residuals Encoding to latent In the residual prediction system 706, the decoder network 709 can perform latent... Decode the data to generate the prediction residuals. Then, the B-frame decoding system 720 can predict the frames. Add to prediction residuals To generate reconstructed frames .

[0148] Figure 7C is a diagram illustrating the system flow of an exemplary frame interpolation engine 725 that can be used by a B-frame decoding system. The frame interpolation engine 725 is an illustrative example of the frame interpolation engine 722 of Figure 7B. In the example of Figure 7C, the frame interpolation engine 725 includes an optical flow estimation layer 730, an optical flow prediction layer 732, a 2D deformation layer 734, a thinning layer 736, and a 3D deformation layer 738.

[0149] As shown in Figure 7C, the reference frame and reference frame The input is fed into the optical flow estimation layer 730. In some examples, the optical flow estimation layer 730 may include one or more neural networks or other machine learning systems. For example, the optical flow estimation layer 730 may be implemented using one or more optical flow estimation neural networks. The optical flow estimation layer 730 may use a reference frame. and reference frame To estimate relative to the reference frame and reference frame The bidirectional motion information 740 (e.g., forward optical flow and backward optical flow). The bidirectional motion information 740 may include information from the reference frame. to reference frame The optical flow or optical flow map (denoted as f 0→1), and from the reference frame to reference frame Optical flow or optical flow diagram (labeled as f 1→0).

[0150] For example, bidirectional motion information 740 may include an optical flow graph f0→1, which references the frame. (or reference frame) The position of each pixel in a subset of pixels is mapped to the reference frame. (or reference frame) The position of the corresponding pixel in the corresponding pixel subset (in the corresponding pixel subset). The bidirectional motion information 740 may also include an optical flow map f1→0, which will reference the frame. (or reference frame) The position of each pixel in a subset of pixels is mapped to the reference frame. (or reference frame) The position of the corresponding pixel in the corresponding subset of pixels in the reference frame. In some cases, the motion information 740 for each pixel may include an indication of the pixel's position in the reference frame. and The vector of movement between them. In some examples, it can be based on a reference frame. and The optical flow vectors between pixels are used to generate optical flow maps f0→1 and f1→0. The optical flow maps can include optical flow vectors for each pixel in a frame, where each vector indicates the movement of the pixel between frames.

[0151] The optical flow estimation layer 730 can provide bidirectional motion information 740 (e.g., f0→1 and f1→0) to the optical flow prediction layer 732. The optical flow prediction layer 732 can obtain time information 742 indicating time t in the time interval from 0 to 1 (e.g., t...). , where 0 represents the reference frame Time step, 1 represents the reference frame The time step, and time t corresponds to the frame to be interpolated (e.g., the interpolated frame). The time step also corresponds to the input frame (e.g., the input frame in Figure 7B). The time step is determined by the time step. The optical flow prediction layer 732 can use bidirectional motion information 740 and temporal information 742 to predict the 2D optical flow map 744. In some examples, the optical flow estimation layer 730 can be used for a reference frame. and reference frame The optical flow at time t is determined by interpolating the bidirectional motion information 740 (e.g., f0→1 and f1→0). In some cases, the optical flow at time t can be a linear interpolation between f0→1 and f1→0.

[0152] In some examples, the 2D optical flow graph 744 may include a time frame from time t to the reference frame. (For example, to the reference frame) Optical flow plot with a time step of 0 (e.g., ), and from time t to the reference frame (For example, to the reference frame) Optical flow plot with time step 1) (e.g., For example, the 2D optical flow map 744 may include a time-to-reference frame diagram. Reverse 2D flow graph of motion information of time series (e.g., time series 0 in the example shown in Figure 7C). and has a time frame from time t to the reference frame. Forward 2D flow graph of motion information of time series (e.g., time series 1 in the example shown in Figure 7C) .

[0153] In an illustrative example, the optical flow prediction layer 732 can predict the 2D optical flow map 744 according to Equations 1 and 2, as shown below: [Mode(] [1] [)] [Mode(] [2] [)]

[0154] In Equations 1 and 2 above, t represents the reference frame. Time step and reference frame Interpolated frames between time steps (e.g., time steps between 0 and 1) (corresponding to the input frame) The time step, f0→1, indicates the time step from the reference frame. to reference frame The optical flow, f 1→0, indicates the optical flow from the reference frame. to reference frame Optical flow, Indicates the identifier from t to the reference frame. 2D optical flow plots of the reverse motion at associated time steps (e.g., time step 0 in the example shown in Figure 7C). Indicates the identifier from t to the reference frame. 2D optical flow map of forward motion at associated time steps (e.g., time step 1 in the example shown in Figure 7C).

[0155] The 2D deformable layer 734 can obtain a 2D optical flow map 744 and a reference frame. and reference frame Furthermore, it can generate deformed frame 746. Deformed frame 746 may include a frame corresponding to the reference frame. The deformed frame (W x 0). The 2D deformed layer 734 can be used to transform the 2D optical flow map. Applied to reference frame This generates a deformed frame W x 0. The deformed frame 746 may also include a corresponding reference frame. The deformed frame (W x 1). The 2D deformed layer 734 can be used to transform the 2D optical flow map. Applied to reference frame To generate a deformed frame W x 1. The 2D deformable layer 734 can implement one or more deformable functions to transform the reference frame. Deform to time step t, and use the reference frame. Transform to time step t.

[0156] In some examples, to generate the deformed frame W x 0, the 2D deformed layer 734 can be used as a reference frame. Calculated 2D optical flow map , to reference frame The pixels in the image are moved to their respective positions corresponding to and / or relative to time t. Similarly, to generate a deformed frame W x 1, the 2D deformed layer 734 can be used as a reference frame. Calculated 2D optical flow map , reference frame The pixels in the frame are moved to positions corresponding to and / or relative to time t. In some cases, the deformed frame 746 may include a reference frame. 2D frames (which take into account the vertical and / or horizontal displacement of pixels (and / or pixel positions) at time t) and reference frames The 2D frame (which takes into account the vertical and / or horizontal displacement of pixels (and / or pixel positions) at time t).

[0157] The refinement layer 736 can obtain a 2D optical flow map 744, a deformed frame 746, and a reference frame. and As input, and can be used to perform thinning operations. In some examples, the thinning layer 736 may be an optional component of the frame interpolation engine 725. In some cases, the thinning layer 736 may include one or more neural networks or other machine learning systems. In an illustrative example, the thinning layer 736 may include one or more interpolation convolutional neural networks (CNNs). In some examples, the thinning operations performed by the thinning layer 736 may include: thinning and / or correcting values ​​in the 2D optical flow map 744. For example, in some cases, corresponding to the reference frame... Time step 0 and corresponding to the reference frame Some or all of the movement within a time step of 1 may not be linear. The thinning layer 736 can detect and correct for this nonlinear motion. The thinning layer 736 can generate a thinned 2D optical flow that explains this nonlinearity. Additionally or alternatively, the thinning layer 736 can thin or correct any deformed frame 746 (and / or reference frame). and Artifacts in any of (746 and / or the reference frame), refine or correct any deformed frames 746 (and / or the reference frame). and Any part (e.g., edge) of any of the frames 746 (and / or reference frames) is refined or corrected to resolve any deformed frames 746 (and / or reference frames). and Occlusion in any of them, and / or performing image data and / or with deformed frame 746 and / or reference frame and Any further refinement of the associated optical flow information.

[0158] In some examples, the refinement operation may also include: merging the 2D optical flow map 744 into the frame used for interpolation. In the 3D optical flow map 748. For example, as previously described, the 2D optical flow map 744 may include a reference frame. 2D optical flow maps (e.g., ) and for reference frames 2D optical flow maps (e.g., The refinement layer 736 can incorporate 2D optical flow maps 744 (e.g., 2D optical flow maps). and 2D optical flow map This generates a 3D optical flow map 748. The 3D optical flow map 748 can be used to generate a specific frame (e.g., a reference frame). and reference frame This is mapped to the corresponding optical flow information. For example, a 3D optical flow map 748 may include two spatial dimensions (e.g., X and Y axes or vertical and horizontal dimensions) and a third dimension (e.g., the frame dimension) that includes frame information. The frame information in the third dimension can identify the frames (e.g., reference frames) to which motion information (e.g., vertical and / or horizontal displacement) applies in the two spatial dimensions. or reference frame ).

[0159] For example, if the third dimension (e.g., the frame dimension) identifies the reference frame The corresponding value in the spatial dimensions (e.g., on the X and Y axes or the vertical and horizontal dimensions) can indicate the reference frame. Horizontal and / or vertical displacement of mid-pixels (e.g., reference frame) The number of pixels in the image should be determined based on optical flow information in the spatial dimension (how much they should move). Similarly, if a third dimension (e.g., the frame dimension) identifies the reference frame... The corresponding value in the spatial dimensions (e.g., on the X and Y axes or the vertical and horizontal dimensions) can indicate the reference frame. Horizontal and / or vertical displacement of mid-pixels (e.g., reference frame) The number of pixels should be determined based on optical flow information in the spatial dimension (how much they should move).

[0160] In some cases, the thinning layer 736 can use the 2D optical flow map 744 to merge the 2D optical flow map 744 into the 3D optical flow map 748. In some examples, the thinning layer 736 can use the thinned 2D optical flow generated by refining the 2D optical flow map 744 to merge the 2D optical flow map 744 into the 3D optical flow map 748. In some cases, the thinning layer 736 can use a reference frame. and The 3D optical flow map 748 is generated by taking the deformed frame 746 and / or the 2D optical flow map 744 (or the refined 2D optical flow) as input.

[0161] The 3D deformable layer 738 can use the 3D optical flow map 748 and the reference frame. and (and / or deformed frame 746) to generate a time step t corresponding to (the input frame) Interpolation reference frame (corresponding to the time step) For example, the 3D deformable layer 738 can use a 3D optical flow map 748 to modify the reference frame. and (Or deformed frame 746) is deformed and the deformed frame is merged (e.g., deformed reference frame). and Or deformed frame 746), to generate interpolation reference frame. In some cases, the 3D deformation layer 738 can implement one or more deformation functions. In an illustrative example, the following formula can be used to generate interpolated frames: Î t= 0 g(I0, Ft→0)+(1 – 0) g(I1, Ft→1), [Mode(] [3] [)]

[0162] Where g(·,·) is the backward deformation function (which can be implemented using bilinear interpolation in some cases), and the parameters are... 0 controls the contributions of the two input images I0 and I1, and depends on temporal consistency and occlusion inference, while The term represents element-wise multiplication (which results in a multiplication based on...). Content-aware weights of the input image with zero parameters.

[0163] In some cases, one or more loss functions can be used to train and / or optimize the optical flow estimation layer 730, optical flow prediction layer 732, 2D deformable layer 734, thinning layer 736, and / or 3D deformable layer 738. In some cases, the optical flow estimation layer 730, optical flow prediction layer 732, 2D deformable layer 734, thinning layer 736, and 3D deformable layer 738 can be trained end-to-end using one or more loss functions. In an illustrative example, the loss described above (Loss = D + βR) can be used.

[0164] While Figure 7C is described in relation to the use of optical flow interpolation, it should be noted that optical flow interpolation is used herein as an illustrative example for explanatory purposes. Those skilled in the art will recognize that the frame interpolation engine 722 of Figure 7B and / or the frame interpolation engine 725 of Figure 7C can implement any other frame interpolation technique.

[0165] Figure 8 illustrates the steps that can be performed to determine a reference frame (e.g., from the reference frame). to reference frame A graph of different types of motion estimates between motion information (or vice versa). The motion estimates in Figure 8 can be used to determine interpolation reference frames (e.g., as described with respect to Figures 7B and 7C) and / or to deform reference frames (e.g., by the deformation engine 704 of Figures 7A and 7B). In Figure 8, term x represents a reference frame from which motion can be estimated, term f represents the motion estimate, and term y represents the deformed frame that can be calculated as follows: y = f(x).

[0166] One type of motion estimation is a block-based motion estimation technique 802. Block-based motion estimation can be performed on a block-by-block basis. For example, for each block in frame y, motion estimation f defines the position of the corresponding block in frame x. In an illustrative example, motion estimation f may include motion vectors indicating the displacement (e.g., horizontal and vertical displacement) of a block in frame y relative to its corresponding block in frame x. By determining the similarity between blocks (e.g., similarity of pixel values), it can be determined that a block from frame x corresponds to a block in frame y.

[0167] Another type of motion estimation that can be performed is optical flow motion estimation technique 804. Optical flow motion estimation can be performed on a pixel-by-pixel basis. For example, for each pixel in frame y, the motion estimate f defines the position of the corresponding pixel in frame x. The motion estimate f for each pixel can include vectors indicating the movement of the pixel between frames. In some cases, an optical flow map (also called a motion vector map) can be generated based on the calculation of optical flow vectors between frames. The optical flow map can include optical flow vectors for each pixel in a frame, where each vector indicates the movement of the pixel between frames. In an illustrative example, the optical flow vector of a pixel can be a displacement vector (e.g., indicating horizontal and vertical displacements, such as x-displacement and y-displacement), showing the movement of the pixel from the first frame to the second frame.

[0168] In some cases, optical flow maps can include vectors representing fewer than all pixels in a frame. For example, dense optical flow can be computed between frames to generate an optical flow vector for each pixel in the frame, which can then be included in the dense optical flow map. In some examples, each optical flow map can include a 2D vector field, where each vector is a displacement vector showing the movement of a point from one frame to another.

[0169] As described above, optical flow vectors or optical flow maps can be computed between frames in a frame sequence. Two frames can include two directly adjacent frames captured consecutively, or two frames separated by a certain distance in the frame sequence (e.g., within two frames of each other, within three frames of each other, or any other suitable distance). In an illustrative example, pixels in frame x... You can move a certain distance or displacement within frame y. .

[0170] Another type of motion estimation that can be performed is dynamic convolution (also known as adaptive convolution) motion estimation technique 806. Dynamic convolutional motion estimation can be performed pixel by pixel. For example, for each pixel in frame y, the motion estimation f defines a linear combination of the pixel neighborhood in frame x. As shown in Figure 8, for a pixel at a certain pixel position in frame y, a kernel or matrix is ​​defined around the pixel neighborhood surrounding the pixel at the same pixel position in frame x. The exemplary dynamic convolutional motion estimation in Figure 8 is performed on the pixel at position (2, 7) in frame y (corresponding to (column number, row number)), where the top-left pixel position in frame y is at position (0, 0). A kernel is defined around the pixel at position (2, 7) in frame x. For example, as shown, a kernel of size 5x5 is applied to the pixel neighborhood surrounding the pixel at position (2, 7) in frame x.

[0171] Each entry in the kernel or matrix can include a number (e.g., a set of 5x5 numbers in the example of Figure 8). The set of numbers in the kernel can describe how to combine pixels from the pixels surrounding that pixel location in frame x, where that pixel location corresponds to the same pixel location of the pixel being mapped and / or reconstructed in frame y.

[0172] Figures 9 and 10 are diagrams illustrating different examples of a Group of Pictures (GoP) structure (corresponding to the order of frames within a GoP). Figure 9 shows an example of I-frame, B-frame, and I-frame (IBI) decoding techniques. For example, IBI techniques can be used in random access modes in conventional decoding systems (e.g., operating according to one or more video decoding standards such as HEVC or VVC). As shown, for the first GoP (labeled GoP1), the decoding system (e.g., decoding system 600 of Figure 6) can decode (e.g., encode / compress and / or decode / decompress) the first and last frames into I-frames, and can compress the intermediate frames into B-frames. As indicated by the arrows in the column labeled "Ref," the two I-frames are used as reference frames for B-frame compression. The column labeled "Order" indicates that the I-frames are decoded first, followed by the intermediate B-frames. In the second GoP (labeled GoP2), the B-frames can be decoded using the reference frames from GoP1, so it is not necessary to decode the first frame in GoP2 into an I-frame. The last frame (I-frame) in GoP1 can be used as a reference frame to decode B-frames in GoP2. The B-frame decoding system 600 in Figure 6 or the B-frame decoding system 720 in Figure 7B can be used to decode B-frames. For example, the first I-frame in GoP1 in Figure 9 can be the reference frame in Figure 7B. The last I-frame in GoP1 of Figure 9 can be the reference frame in Figure 7B. The B frame in GoP1 of Figure 9 can be the reconstructed frame in Figure 7B. In this case, two I-frames from GoP1 can be used to generate interpolated frames. ).

[0173] Figure 10 is a diagram illustrating an example of I-frame, B-frame, and P-frame (IBP) decoding techniques. For example, because the B-frame decoding system described herein (e.g., B-frame decoding system 720 of Figure 7B) uses a P-frame decoding system (e.g., P-frame decoding system 700 of Figures 7A and 7B), when the P-frame decoding system is used for unidirectional and bidirectional prediction, P-frames can be generated using the same components without adding additional complexity if the parameters (e.g., weights, biases, etc.) are common. In the example of Figure 10, the last frame of each GoP does not necessarily have to be an I-frame (as is the case in the IBI GoP structure of Figure 9), but can be decoded into a P-frame. The decoding order in Figure 10 is also different from that in Figure 9. For example, the first frame of GoP1 in Figure 1 is first decoded (e.g., encoded / compressed and / or decoded / decompressed) into an I-frame. Using the I-frame as a reference frame, the last frame in GoP1 can be decoded into a P-frame (e.g., by the P-frame decoding system 700 of Figure 7A). Then, the I-frames and P-frames in GoP1 can be used to decode intermediate frames into B-frames (e.g., using the B-frame decoding system 600 of Figure 6 or the B-frame decoding system 720 of Figure 7B). For GoP2, the last frame of GoP1 can be used as a reference frame (e.g., by the P-frame decoding system 700 of Figure 7A) to decode the last frame of GoP2 into a P-frame. The last frame of GoP1 and the last frame of GoP2 can be used as reference frames (e.g., by the B-frame decoding system 600 of Figure 6 or the B-frame decoding system 720 of Figure 7B) to decode intermediate frames of GoP2 into B-frames.

[0174] Figures 11 and 12 are diagrams illustrating the frame decoding order within a GoP. For example, once the decoding system (e.g., an I-frame decoding system and / or a P-frame decoding system) has determined two reference frames for the GoP (e.g., the first and last frames in the GoP of Figures 11 and 12), a B-frame decoding system (e.g., B-frame decoding system 600 of Figure 6 or B-frame decoding system 720 of Figure 7B) can begin decoding intermediate frames of the GoP into B-frames. The example decoding orders in Figures 11 and 12 define examples of the order in which those B-frames are decoded (e.g., encoded / compressed and / or decoded / decompressed). In one illustrative example, the first and last frames of the GoP of Figures 11 and 12 may include I-frames, similar to those shown in Figure 9. In another illustrative example, the first frame of the GoP of Figures 11 and 12 may include I-frames, and the last frame may include P-frames, similar to those shown in Figure 10.

[0175] Figure 11 illustrates an example of a sequential scheme for bidirectional prediction. In the sequential scheme of Figure 11, B-frames are decoded sequentially from left to right (e.g., by the B-frame decoding system 600 of Figure 6 or the B-frame decoding system 720 of Figure 7B). As shown, with the first and last frames of the GoP used as reference frames, bidirectional prediction is first used to decode B-frame number 2. Next, with the previous B-frame (number 2) and the last frame of the GoP used as reference frames, bidirectional prediction is used to decode B-frame number 3. After B-frame number 3, B-frame number 4 is decoded, then B-frame number 5, and so on.

[0176] Figure 12 illustrates an example of a hierarchical scheme for bidirectional frame prediction. In the hierarchical scheme of Figure 12, once two initial reference frames (the first and last frames of the GoP) are available, the decoding system (e.g., the B-frame decoding system 600 of Figure 6 or the B-frame decoding system 720 of Figure 7B) can use these two initial reference frames in bidirectional prediction to decode the fifth frame (the intermediate frame with sequence number 2) into the first B frame. Then, the decoding system can use the reference frames indicated by the "Ref" column in Figure 12 to decode the third or seventh frame (both with sequence number 3) in the GoP. For example, the first frame of the GoP (which can be an I-frame) and the fifth frame of the GoP (which can be a B-frame) can be used as reference frames to decode the third frame (sequence number 3), and the last frame of the GoP (which can be an I-frame or a P-frame) and the fifth frame of the GoP (B-frame) can be used as reference frames to decode the seventh frame (sequence number 3). Then, the frame labeled number 4 can be decoded in any order using the reference frames indicated by the "Ref" column in Figure 12.

[0177] Figures 13A and 13B are graphs illustrating the performance of the B-frame decoding system 600 of Figure 6 (e.g., implemented using the B-frame decoding system 720 of Figure 7B) relative to a baseline decoding system that does not use an interpolated reference frame. All three decoding systems illustrated in the graphs of Figures 13A and 13B are neural network-based decoding systems trained using the same training conditions (e.g., the same training dataset, the same loss function, and the same number of training iterations, etc.). These graphs represent performance by plotting the rate (expressed as bits per pixel) as a function of peak signal-to-noise ratio (PSNR). Line 1302 shows the performance of the B-frame decoding system 720 of Figure 7B when using the hierarchical decoding order of Figure 12. Line 1304 shows the performance of the B-frame decoding system 720 when using the sequential decoding order of Figure 11. Line 1306 shows the performance of the baseline decoding system. The upper left corner of these graphs (lower rate, higher PSNR) indicates better performance. As shown in Figures 13A and 13B, line 1302 (B-frame decoding system 720 using layered decoding) represents the highest performance, followed by line 1304 (B-frame decoding system 720 using sequential decoding), and then line 1306 (baseline decoding system).

[0178] Figures 14A through 14F are graphs illustrating the performance of the B-frame decoding system 600 of Figure 6 (e.g., implemented using the B-frame decoding system 720 of Figure 7B) based on the frame rate per frame (as shown in the graph labeled as rate vs. frame index), the PSNR per frame (as shown in the graph labeled as PSNR vs. frame index), and the multi-scale structural similarity (MS-SSIM) per frame (as shown in the graph labeled as MS-SSIM vs. frame index). The x-axis of these graphs includes the frame index representing the frame order, and the y-axis includes various performance metrics (e.g., rate, PSNR, and MS-SSIM). In the examples of Figures 14A through 14F, the number of frames in each GoP is 12, and these graphs show the average of all GoPs used to generate the result video sequence.

[0179] Using intra-frame prediction, the first frame in each GoP sequence (with frame index 0) is generated as an I-frame (e.g., as shown in the GoP structures of Figures 9 and 10). Points 1402 in Figure 14A, 1404 in Figure 14B, 1406 in Figure 14C, 1408 in Figure 14D, 1410 in Figure 14E, and 1412 in Figure 14E represent the metrics (rate, PSNR, or MS-SSIM) of the I-frame. For example, point 1402 in Figure 14A shows the rate (number of bits) of compression used for that I-frame. Using unidirectional prediction, the last frame in each GoP sequence (with frame index 11) is generated as a P-frame (e.g., generated by the P-frame decoding system 700 of Figure 7A). Points 1414 in Figure 14A, 1416 in Figure 14B, 1418 in Figure 14C, 1420 in Figure 14D, 1422 in Figure 14E, and 1424 in Figure 14E represent metrics (rate, PSNR, or MS-SSIM) for P-frames. For example, as shown in Figure 14A, fewer bits are used for P-frames compared to the bits used for I-frames.

[0180] The frames between the first and last frames are decoded into B-frames (e.g., performed by the B-frame decoding system 720 of Figure 7B). Two sets of results for B-frames are shown, one using the sequential decoding order of Figure 11 and the other using the hierarchical decoding order of Figure 12. Referring to Figure 14A as an example, line 1426 represents the result using the sequential decoding order, where the frame with frame index 1 is decoded into a B-frame first, followed by the frame with frame index 2, and so on. Line 1428 in Figure 15A represents the result using the hierarchical decoding order, where the frame with frame index 5 is decoded into a B-frame first, followed by the frame with frame index 2, followed by the frame with frame index 8, and so on. As shown in Figure 14A, fewer bits are used for subsequent B-frames decoded due to the smaller distance from the reference frame to the B-frame, as shown in Figure 12. For example, using I-frames (the first frame of GoP) and P-frames (the last frame of GoP) as reference frames, bidirectional prediction is performed on the frame with frame index 5, while using I-frames and B-frames with frame index 2 as reference frames, bidirectional prediction is performed on the frame with frame index 1. The smooth lines in Figures 14B-F represent the corresponding results using the layered decoding order of Figure 12, and the jagged lines in Figures 14B-F represent the corresponding results using the sequential decoding order of Figure 11.

[0181] Figure 15 is a flowchart illustrating an example of a process 1500 for performing bidirectional prediction of frames using the techniques described herein. At block 1502, process 1500 includes obtaining a first reference frame and a second reference frame. In one illustrative example, the first reference frame may include reference frame 0 (Ref 0) of Figure 6, and the second reference frame may include reference frame 1 (Ref 1) of Figure 6. In another illustrative example, the first reference frame may include reference frame 0 (Ref 0) of Figures 7B and / or 7C. The second reference frame may include reference frame 1 of Figure 7B and / or Figure 7C. ).

[0182] At box 1504, process 1500 includes generating a third reference frame at least partially by performing interpolation between the first and second reference frames. For example, process 1500 may include determining a first set of motion information representing pixel motion from the first reference frame to the third reference frame. Process 1500 may also include determining a second set of motion information representing pixel motion from the second reference frame to the third reference frame. In some aspects, the first set of motion information and the second set of motion information are determined based on pixel motion between the first and second reference frames. In some cases, the first set of motion information includes a first optical flow map, while the second set of motion information includes a second optical flow map. Referring to the description of FIG7C as an illustrative example, the first motion information may include the information defined above using equation (1). The second motion information may include the information defined above using equation (2). Processing 1500 may include generating first deformation information by performing a deformation function on a first reference frame, at least in part, using a first set of motion information. In some aspects, the deformation function includes a bilinear interpolation function, such as that described above with respect to FIG7C. Processing 1500 may also include generating second deformation information by performing a deformation function on a second reference frame, at least in part, using a second set of motion information. Processing 1500 may include generating a third reference frame based on the first deformation information and the second deformation information.

[0183] At box 1506, process 1500 includes performing unidirectional inter-frame prediction on the input frame based on a third reference frame. For example, to perform unidirectional inter-frame prediction, process 1500 may include estimating motion between the input frame and the third reference frame. In an illustrative example, predicting motion may include the predicted motion of FIG7B. In some cases, the estimated motion between the input frame and the third reference frame includes optical flow data (e.g., determined using optical flow motion estimation technique 804 shown in FIG8). In some aspects, the estimated motion between the input frame and the third reference frame includes dynamic convolutional data (e.g., determined using dynamic convolutional motion estimation technique 806 shown in FIG8). In some aspects, the estimated motion between the input frame and the third reference frame includes block-based motion data (e.g., determined using block-based motion estimation technique 802 shown in FIG8). Processing 1500 may further include: generating a deformed frame by deforming one or more pixels of the third reference frame, at least in part, based on the estimated motion. For example, referring to FIG7B as an illustrative example, the deforming engine 704 may be based on predicted motion. The motion vectors (or other motion data) included are used to deform and interpolate the reference frame. Pixels.

[0184] In some examples, to further perform unidirectional inter-frame prediction, process 1500 may include determining a residual, at least in part, by determining the difference between the input frame and the deformed frame. Process 1500 may also include using the residual to generate a prediction residual. Referring to FIG7B as an illustrative example, the P-frame decoding system 700 may generate a residual. As frame t prediction and input frame The difference between them. The residual prediction system 706 can use the techniques described above to generate predicted residuals. .

[0185] In some aspects, a first neural network (e.g., a first autoencoder or other type of neural network) is used to estimate motion, and a second neural network (e.g., a second autoencoder or other type of neural network) is used to generate prediction residuals. In an illustrative example, the first neural network may include the motion prediction system 702 of FIG. 7B, and the second neural network may include the residual prediction system 706 of FIG. 7B. In some examples, processing 1500 may include generating a reconstructed frame representing the input frame based on the deformed frame and the predicted residuals. The reconstructed frame includes bidirectional prediction frames. Referring again to FIG. 7B as an illustrative example, the P-frame decoding system 720 can predict frame t by using the predicted residuals. (by interpolating reference frames) (Generated by deformation) and the predicted residual Add them together to generate the reconstructed frame t ( In some cases, processing 1500 may also include storing the reconstructed frame in memory (e.g., a decoded picture buffer (DPB) or other memory). In some examples, the stored reconstructed frame can be used for inter-frame prediction of other frames.

[0186] As described above, the P-frame decoding system described herein (e.g., the P-frame decoding system 700 of FIG7A) can use interpolated reference frames (e.g., interpolated reference frames). This can be used to perform bidirectional prediction, or it can perform unidirectional prediction. For example, to perform unidirectional prediction, processing 1500 may include: obtaining a fourth reference frame (e.g., reference frame 1500). Processing 1500 may further include estimating motion between the additional input frame and the fourth reference frame (e.g., using motion prediction system 702). In an illustrative example, the estimated motion may include the predicted motion shown in Figure 7A. Processing 1500 may include generating additional deformed frames (e.g., using deforming engine 704) by deforming one or more pixels of the fourth reference frame, at least in part, based on estimated motion. Referring again to FIG7A as an illustrative example, the additional deformed frames may include frame prediction. Processing 1500 may include: generating a reconstructed frame representing an additional input frame based on an additional deformed frame and an additional prediction residual, in which case the reconstructed frame representing the additional input frame includes a unidirectional prediction frame. Referring to FIG7A as an illustrative example, the P-frame decoding system 720 can generate a reconstructed frame representing an additional input frame based on an additional deformed frame and an additional prediction residual, in which case the reconstructed frame representing the additional input frame includes a unidirectional prediction frame. (via deformed reference frame) (Generate) Add to the prediction residual To generate reconstructed frame t ( In some cases, processing 1500 may include storing a reconstructed frame representing another input frame in memory (e.g., a decoded picture buffer (DPB) or other memory). In some examples, the stored reconstructed frame representing the other input frame can be used for inter-frame prediction of other frames.

[0187] Figure 16 is a flowchart illustrating another example of a process 1600 for performing bidirectional prediction of frames using the techniques described herein. In some cases, at least a portion of the process 1600 may be performed by the decoder network 703 of the motion prediction system 702 and the decoder network 709 of the residual prediction system 706. At block 1602, the process 1600 includes: obtaining an interpolation reference frame and motion information representing the motion between the input frame and the interpolation reference frame. In an illustrative example, the interpolation reference frame may include the interpolation reference frame of Figure 7B. In some respects, motion information is obtained from a bitstream. For example, motion information may include the potential shown in Figure 7B. At box 1604, processing 1600 includes: estimating the motion between the input frame and the interpolated reference frame based on motion information. For example, the decoder network 703 of the motion prediction system 702 can be based on latent... Estimate the input frame and interpolation reference frame The estimated motion may include the predicted motion shown in Figure 7B. .

[0188] At box 1606, processing 1600 includes: generating a deformed frame by deforming one or more pixels of the interpolated reference frame, at least in part, based on estimated motion. For example, referring to FIG7B as an illustrative example, the deforming engine 704 may be based on the motion included in the predicted motion. The motion vectors (or other motion data) in the interpolation reference frame The pixels are deformed.

[0189] At box 1608, processing 1600 includes: generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual. This reconstructed frame includes a bidirectional prediction frame. Referring again to FIG7B as an illustrative example, the P-frame decoding system 720 can decode frames t by predicting... (using deformed interpolation reference frames) (Generation) and prediction residuals Add to generate the reconstructed frame t ( In some cases, processing 1500 may also include storing the reconstructed frame in memory (e.g., a decoded picture buffer (DPB) or other memory). In some examples, the stored reconstructed frame can be used for inter-frame prediction of other frames.

[0190] In some examples, process 1600 may include obtaining a first reference frame and a second reference frame. Process 1600 may also include generating an interpolated reference frame, at least in part, by performing interpolation between the first and second reference frames. The techniques described herein (e.g., those described with respect to FIG7C and / or FIG15) can be used to generate interpolated reference frames.

[0191] In some examples, processing 1600 may include obtaining residuals from the bitstream. For example, the decoder network 709 of the residual prediction system 706 may obtain potential residuals from the bitstream. .potential Residual Processing 1600 may also include generating predictive residuals based on the obtained residuals. For example, referring to Figure 7B as an illustrative example, the decoder network 709 can use latent... Used as input to generate predicted residuals .

[0192] In some respects, similar to that described above with respect to process 1500, process 1600 may include: obtaining a fourth reference frame, estimating motion between the other input frames and the fourth reference frame, and generating an additional deformed frame by deforming one or more pixels of the fourth reference frame, at least in part, based on the estimated motion. Process 1600 may include: generating a reconstructed frame representing the additional input frame based on the additional deformed frame and additional prediction residuals, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

[0193] In some examples, processing 1500 and / or 1600 may include training the neural network system using a loss function (e.g., Loss) for one or more training iterations. In some cases, the loss function may determine the loss between one or more reconstructed B-frames and one or more frames represented by the reconstructed B-frames. In some examples, the first loss function may be equal to D + βR, as previously described.

[0194] In some examples, the processes described herein (e.g., processes 1500, 1600, and / or other processes described herein) may be performed by a computing device or apparatus (e.g., a computing device having the computing device architecture 1700 shown in FIG. 17). This computing device may include any suitable device, such as an autonomous vehicle computer, a robotic device, a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device, a server device, a video gaming device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a camera device, a set-top box device, and / or any other computing device with the resource capability to perform the processes described herein, including process 800. In some examples, the computing device may include a mobile device, a wearable device, an XR device, a personal computer, a laptop computer, a video server, a television, a camera, a set-top box, a video game console, or other devices. In some examples, processing 1500 and / or processing 1600 can be performed by a computing device having a computing device architecture 1700 that implements the B-frame decoding system 600 of Figure 6 and / or the B-frame decoding system 720 of Figure 7.

[0195] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers, or combinations of transmitters and receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processing described herein. In some examples, a computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP) based data or other types of data.

[0196] Components of a computing device can be implemented using circuitry. For example, these components may include and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), neural processing units (NPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or other suitable electronic circuitry), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0197] Processes 1500 and 1600 are shown as logic flowcharts, whose operations represent a series of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, these operations represent computer-executable instructions stored on one or more computer-readable storage media, which, when executed by one or more processors, perform the described operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which these operations are described should not be construed as restrictive, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0198] Additionally, the processes described herein (including processes 1500 and 1600) can be executed under the control of one or more computer systems configured with executable instructions, and these processes can be implemented as code that executes jointly on one or more processors, code that executes via hardware, or a combination thereof (e.g., executable instructions, one or more computer programs, or one or more application programs). As described above, the code can be stored, for example, on a computer-readable or machine-readable storage medium in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0199] Figure 17 illustrates an exemplary computing device architecture 1700 that can implement the various technologies described herein. In some examples, the computing device may include a mobile device, wearable device, XR device, personal computer, laptop computer, video server, video game console, robotic device, set-top box, television, camera, server, or other device. For example, computing device architecture 1700 may implement the B-frame decoding system 600 of Figure 6, the P-frame decoding system 700 of Figure 7A, and / or the B-frame decoding system 720 of Figure 7B. Components of computing device architecture 1700 that communicate electrically with each other using a connection 1705 such as a bus are shown. The exemplary computing device architecture 1700 includes a processing unit (CPU or processor) 1710 and a computing device connection 1705 that couples various computing device components, including computing device memory 1715 (e.g., read-only memory (ROM) 1720 and random access memory (RAM) 1725), to processor 1710.

[0200] The computing device architecture 1700 may include a cache of high-speed memory that is directly connected to, closely adjacent to, or integrated as part of the processor 1710. The computing device architecture 1700 may copy data from memory 1715 and / or storage device 1730 to cache 1712 for fast access by the processor 1710. In this way, the cache can provide performance improvements, thereby avoiding latency for the processor 1710 while waiting for data. These and other modules may control or be configured to control the processor 1710 to perform various actions. Other computing device memory 1715 may also be available. Memory 1715 may include various different types of memory with different performance characteristics. The processor 1710 may include any general-purpose processor and hardware or software services stored in storage device 1730 (e.g., service 1 1732, service 2 1734, and service 3 1736) and is configured to control the processor 1710 and dedicated processors in which software instructions are incorporated into the processor design. The processor 1710 can be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0201] To enable users to interact with the computing device architecture 1700, input device 1745 can represent any number of input devices, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 1735 can also be one or more of a variety of output devices known to those skilled in the art, such as a monitor, projector, television, speaker equipment, etc. In some instances, multi-mode computing devices allow users to provide multiple types of input to communicate with computing device architecture 1700. Communication interface 1740 typically controls and manages user input and computing device output. There are no limitations on operation on any particular hardware device; therefore, the basic functionality described herein can be easily replaced with improved hardware or firmware devices as they are developed.

[0202] Storage device 1730 is non-volatile memory and may be a hard disk or other type of computer-readable media capable of storing data accessible by a computer, such as magnetic tape, flash memory card, solid-state storage device, digital universal disk, magnetic tape cassette, random access memory (RAM) 1725, read-only memory (ROM) 1720, and combinations thereof. Storage device 1730 may include services 1732, 1734, and 1736 for controlling processor 1710. Other hardware or software modules are contemplated. Storage device 1730 may be connected to computing device connection 1705. In one aspect, a hardware module performing a specific function may include software components for performing that function, incorporating necessary hardware components (e.g., processor 1710, connection 1705, output device 1735, etc.) stored in a computer-readable medium.

[0203] The term "device" is not limited to one or a specific number of physical objects (e.g., a smartphone, a controller, a processing system, etc.). As used herein, a device can include any electronic device having one or more portions that can implement at least some of the contents of this disclosure. Although the specification and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. Furthermore, the term "system" is not limited to multiple components or a particular embodiment. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a particular configuration, type, or number of objects.

[0204] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity, in some instances, the techniques described herein may be presented as comprising individual functional blocks, including devices, device components, methods or routines embodied in software, or combinations of hardware and software. Other components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0205] The various embodiments described above are processes or methods, which are depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While a flowchart can describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Furthermore, the order of these operations can be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0206] The processes and methods described above can be implemented using computer-executable instructions stored in or available from a computer-readable medium. For example, such instructions may include instructions and data that cause or configure a general-purpose computer, special-purpose computer, or processing device to perform a specific function or group of functions. The computer resources may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions (e.g., assembly language), firmware, source code, etc.

[0207] The term "computer-readable media" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media can include non-transitory media capable of storing data therein, but does not include carrier waves and / or transient electronic signals propagated wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as flash memory, memory or storage devices, optical storage media such as magnetic disks or optical discs, flash memory, USB devices equipped with non-volatile memory, network storage devices, optical discs (CDs) or digital versatile discs (DVDs), any suitable combinations thereof, etc. Computer-readable media may store code and / or machine-executable instructions that can represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments can be coupled to another code segment or hardware circuitry by transmitting and / or receiving information, data, arguments, parameters, or memory contents. Information, variables, parameters, data, etc., can be transmitted, forwarded, or transferred through any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0208] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0209] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of forms. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program products) used to perform the necessary tasks may be stored on computer-readable or machine-readable media. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small-form-factor personal computers, personal digital assistants, rack-mount devices, stand-alone devices, and so on. The functionality described herein may also be embodied in peripheral devices or add-in cards. As a further example, this functionality may also be implemented on a circuit board between different chips or different processes executed in a single device.

[0210] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0211] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative embodiments of this application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed differently in other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by prior art. Various features and aspects of the above-described applications may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications beyond those described herein without departing from the scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods have been described in a specific order. It should be understood that in alternative embodiments, the methods may be performed in a different order than that described.

[0212] Those skilled in the art will understand that the symbols or terms less than (“<”) and greater than (“>”) used herein can be replaced with less than or equal to (“>”). ") and greater than or equal to (" The symbol 」) is used without departing from the scope of this specification.

[0213] When a component is described as being "configured" to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0214] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0215] A statement or other language that refers to "at least one" and / or "one or more" of a set indicates that one member of that set or multiple members of that set (in any combination) satisfy the claims. For example, a statement that refers to "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, a statement that refers to "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The use of "at least one" and / or "one or more" of a set does not limit the set to the items listed in that set. For example, a statement that refers to "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0216] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0217] The techniques described herein can also be implemented using electronic hardware, computer software, firmware, or any combination thereof. These techniques can be implemented using any of a variety of devices, such as general-purpose computers, wireless communication handheld devices, or integrated circuit devices with multiple uses (including applications in wireless communication handheld devices and other devices). Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable storage medium comprising code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable storage medium can form part of a computer program product, which may include encapsulation material. Computer-readable media may include memory or data storage media, such as random access memory (RAM) like synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and so on. Alternatively or additionally, these technologies may be implemented at least in part by computer-readable communication media that carries or transmits program code in the form of instructions or data structures (e.g., propagated signals or waves), and the program code can be accessed, read, and / or executed by a computer.

[0218] The code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated logic or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, it may also be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, several microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0219] Illustrative examples of this disclosure include:

[0220] Aspect 1: A system for processing video data, the system comprising: a bidirectional decoding engine including an interpolation engine and a unidirectional decoding engine, wherein: the interpolation engine is configured to obtain a first reference frame and a second reference frame, and to generate a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame; the unidirectional decoding engine is configured to perform unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0221] Aspect 2: The system according to Aspect 1, wherein the one-way decoding engine includes a motion engine and a deformation engine, and wherein: the motion engine is configured to estimate motion between the input frame and the third reference frame; the deformation engine is configured to at least partially deform one or more pixels of the third reference frame based on the estimated motion to generate a deformed frame.

[0222] Aspect 3: The system according to Aspect 2, wherein the one-way decoding engine further includes a residual engine, and wherein: the residual engine is configured to: determine a residual at least in part by determining the difference between the input frame and the deformed frame; and use the residual to generate a prediction residual.

[0223] Aspect 4: The system according to aspect 3 further includes a memory and at least one processor coupled to the memory, wherein the one-way decoding engine is configured to: generate a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame; and wherein the at least one processor is configured to store the reconstructed frame in the memory.

[0224] Aspect 5: The system according to aspect 4, wherein the at least one processor includes a neural processing unit (NPU).

[0225] Aspect 6: The system according to any one of Aspects 3 to 5, wherein the motion engine includes a first neural network and the residual engine includes a second neural network.

[0226] Aspect 7: The system according to aspect 6, wherein the first neural network includes a first autoencoder, and the second neural network includes a second autoencoder.

[0227] Aspect 8: A system according to any one of Aspects 1 to 7, wherein the one-way decoding engine is configured to: obtain a fourth reference frame; estimate motion between an additional input frame and the fourth reference frame; generate an additional deformed frame by deforming one or more pixels of the fourth reference frame at least in part based on the estimated motion; and generate a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes the one-way prediction frame.

[0228] Aspect 9: A system according to any one of Aspects 1 to 8, wherein, in order to generate the third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame, the interpolation engine is configured to: determine a first set of motion information representing pixel motion from the first reference frame to the third reference frame; determine a second set of motion information representing pixel motion from the second reference frame to the third reference frame; generate first deformation information by performing a deformation function on the first reference frame at least in part using the first set of motion information; generate second deformation information by performing the deformation function on the second reference frame at least in part using the second set of motion information; and generate the third reference frame based on the first deformation information and the second deformation information.

[0229] Aspect 10: The system according to aspect 9, wherein the first set of motion information and the second set of motion information are determined based on pixel motion between the first reference frame and the second reference frame.

[0230] Aspect 11: The system according to any one of Aspects 9 or 10, wherein the first set of motion information includes a first optical flow map, and the second set of motion information includes a second optical flow map.

[0231] Aspect 12: The system according to any one of Aspects 9 to 11, wherein the deformation function comprises a bilinear interpolation function.

[0232] Aspect 13: The system according to any one of aspects 2 to 12, wherein the estimated motion between the input frame and the third reference frame includes optical flow data.

[0233] Aspect 14: The system according to any one of Aspects 2 to 12, wherein the estimated motion between the input frame and the third reference frame comprises dynamic convolutional data.

[0234] Aspect 15: The system according to any one of Aspects 2 to 12, wherein the estimated motion between the input frame and the third reference frame comprises block-based motion data.

[0235] Aspect 16: The system according to any one of Aspects 1 to 15, wherein the system is one of a mobile device, an extended reality device, or a television.

[0236] Aspect 17: The system according to any one of aspects 1 to 16 further includes at least one of a display and a camera configured to capture one or more video frames.

[0237] Aspect 18: A method for processing video data, the method comprising: obtaining a first reference frame and a second reference frame; generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame; and performing unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0238] Aspect 19: The method according to aspect 18 further includes: estimating motion between the input frame and the third reference frame; and generating a deformed frame by deforming one or more pixels of the third reference frame based at least in part on the estimated motion.

[0239] Aspect 20: The method according to aspect 19 further includes: determining a residual at least in part by determining the difference between the input frame and the deformed frame; and using the residual to generate a prediction residual.

[0240] Aspect 21: The method according to aspect 20 further includes: generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame; and storing the reconstructed frame in memory.

[0241] Aspect 22: The method according to any one of Aspects 20 or 21, wherein the motion is estimated using a first autoencoder, and wherein the prediction residual is generated using a second autoencoder.

[0242] Aspect 23: The method according to any one of aspects 18 to 22 further includes: obtaining a fourth reference frame; estimating motion between an additional input frame and the fourth reference frame; generating an additional deformed frame by deforming one or more pixels of the fourth reference frame at least in part based on the estimated motion; and generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

[0243] Aspect 24: The method according to any one of aspects 18 to 23 further includes: determining a first set of motion information representing pixel motion from the first reference frame to the third reference frame; determining a second set of motion information representing pixel motion from the second reference frame to the third reference frame; generating first deformation information by performing a deformation function on the first reference frame at least in part using the first set of motion information; generating second deformation information by performing the deformation function on the second reference frame at least in part using the second set of motion information; and generating the third reference frame based on the first deformation information and the second deformation information.

[0244] Aspect 25: According to the method of aspect 24, wherein the first set of motion information and the second set of motion information are determined based on pixel motion between the first reference frame and the second reference frame.

[0245] Aspect 26: The method according to any one of Aspects 24 or 25, wherein the first set of motion information includes a first optical flow map, and the second set of motion information includes a second optical flow map.

[0246] Aspect 27: The method according to any one of aspects 24 to 26, wherein the deformation function comprises a bilinear interpolation function.

[0247] Aspect 28: The method according to any one of aspects 19 to 27, wherein the estimated motion between the input frame and the third reference frame includes optical flow data.

[0248] Aspect 29: The method according to any one of aspects 19 to 27, wherein the estimated motion between the input frame and the third reference frame comprises dynamic convolutional data.

[0249] Aspect 30: The method according to any one of aspects 19 to 27, wherein the estimated motion between the input frame and the third reference frame includes block-based motion data.

[0250] Aspect 31: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the following operations: obtaining a first reference frame and a second reference frame; generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame; and performing unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0251] Aspect 32: The non-transitory computer-readable medium according to aspect 31 further includes, when executed by the one or more processors, instructions that cause the one or more processors to perform any of the operations described in aspects 19 to 30.

[0252] Aspect 33: An apparatus for processing video data, comprising: means for obtaining a first reference frame and a second reference frame; means for generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame; and means for performing unidirectional inter-frame prediction on an input frame based on the third reference frame.

[0253] Aspect 34: The apparatus according to aspect 33 further includes a component for performing the operations described in any one of aspects 19 to 30.

[0254] Aspect 35: A system for processing video data, the system comprising: a memory; and at least one processor coupled to the memory, the at least one processor being configured to: obtain an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; estimate motion between the input frame and the interpolation reference frame based on the motion information; generate a deformed frame by deforming one or more pixels of the interpolation reference frame at least in part based on the estimated motion; and generate a reconstructed frame representing the input frame based on the deformed frame and a prediction residual, the reconstructed frame including a bidirectional prediction frame.

[0255] Aspect 36: The system according to aspect 35, wherein the at least one processor is configured to: obtain a first reference frame and a second reference frame; and generate the interpolated reference frame at least in part by performing interpolation between the first reference frame and the second reference frame.

[0256] Aspect 37: The system according to aspect 35, wherein the at least one processor is configured to: obtain residuals from a bit stream; and generate prediction residuals based on the obtained residuals.

[0257] Aspect 38: The system according to aspect 37, wherein the motion information is obtained from the bit stream.

[0258] Aspect 39: The system according to aspect 35, wherein the at least one processor is configured to: obtain a fourth reference frame; estimate motion between an additional input frame and the fourth reference frame; generate an additional deformed frame by deforming one or more pixels of the fourth reference frame, at least in part, based on the estimated motion; and generate a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

[0259] Aspect 40: A method for processing video data, the method comprising: obtaining an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; estimating motion between the input frame and the interpolation reference frame based on the motion information; generating a deformed frame by deforming one or more pixels of the interpolation reference frame at least in part based on the estimated motion; and generating a reconstructed frame representing the input frame based on the deformed frame and a prediction residual, the reconstructed frame including a bidirectional prediction frame.

[0260] Aspect 41: The method according to aspect 40 further includes: obtaining a first reference frame and a second reference frame; and generating the interpolated reference frame at least in part by performing interpolation between the first reference frame and the second reference frame.

[0261] Aspect 42: The method according to aspect 40 further includes: obtaining residuals from the bit stream; and generating prediction residuals based on the obtained residuals.

[0262] Aspect 43: According to the method of aspect 42, wherein the motion information is obtained from the bit stream.

[0263] Aspect 44: The method according to aspect 40 further includes: obtaining a fourth reference frame; estimating motion between an additional input frame and the fourth reference frame; generating an additional deformed frame by deforming one or more pixels of the fourth reference frame, at least in part, based on the estimated motion; and generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

[0264] Aspect 45: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the following operations: obtaining an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; estimating motion between the input frame and the interpolation reference frame based on the motion information; generating a deformed frame by deforming one or more pixels of the interpolation reference frame, at least in part, based on the estimated motion; and generating a reconstructed frame representing the input frame, the reconstructed frame including a bidirectional prediction frame, based on the deformed frame and a prediction residual.

[0265] Aspect 46: The non-transitory computer-readable medium according to aspect 45 further includes, when executed by the one or more processors, instructions that cause the one or more processors to perform the operations according to any one of aspects 41 to 44.

[0266] Aspect 47: An apparatus for processing video data, comprising: means for obtaining an interpolation reference frame and motion information representing motion between an input frame and the interpolation reference frame; means for estimating motion between the input frame and the interpolation reference frame based on the motion information; means for generating a deformed frame by deforming one or more pixels of the interpolation reference frame at least in part based on the estimated motion; and means for generating a reconstructed frame representing the input frame based on the deformed frame and a prediction residual, the reconstructed frame including a bidirectional prediction frame.

[0267] Aspect 48: The apparatus according to aspect 47 further includes a component for performing the operations according to any one of aspects 41 to 44.

[0268] 100: Image Processing System 102: Central Processing Unit (CPU) 104: Graphics Processing Unit (GPU) 106: Digital Signal Processor (DSP) 108: Neural Processing Unit (NPU) 110: Connector Block 112: Multimedia Processor 114: Sensor 116: Image Signal Processor (ISP) 118: Storage memory block 120: Storage device 200: Deep Convolutional Networks (DCN) 202: Fully Connected Neural Networks 204: Locally Connected Neural Networks 206: Convolutional Neural Networks 208: Connection Strength 210: value 212: value 214: value 216: value 218: First set of feature maps 220: Second set of feature maps 222: Output 224: First eigenvector 226: Image 228: Second eigenvector 230: Image capturing device 232: Convolutional Layer 350: Deep Convolutional Network 352: Input data 354A: Convolutional Block 354B: Convolutional Block 356: Convolutional Layer (CONV) 358: Normalized Layer (LNorm) 360: Max Pooling Layer 362A: Layer 362B: Layer 364: Logistic Regression (LR) Layer 366: Category Score 400:P frame decoding system 401: Input Frame 402: Sport Compression Engine 403: Reference Frame 404: Motion Compensation Engine 406: Residual Compression Engine 500:B Frame Decoding System 501: Input Frame 502: Sport Compression Engine 504: Motion Compensation Engine 506: Residual Compression Engine 510:B-frame decoding system 511: Input Frame 512: Frame Interpolation Engine 516: Residual Compression Engine 600:B Frame Decoding System 611: Coarse prediction of input frame 613: Interpolation Reference Frame 615: Input Frame 622: Frame Interpolation Engine 624: P-frame decoding system 700:P frame decoding system 702: Motion Prediction System 703: Decoder Network 704: Transforming Engine 705: Encoder Network 706: Residual Prediction System 707: Encoder Network 708: Subtraction Operation 709: Decoder Network 710: Addition Operation 720:B-frame decoding system 722: Frame Interpolation Engine 725: Frame Interpolation Engine 730: Optical Flow Estimation Layer 732: Optical Flow Prediction Layer 734:2D Deformation Layer 736: Refinement Layer 738: 3D Deformation Layer 740: Two-way sports information 742: Time Information 744:2D Optical Flow Map 746: Deformed Frame 748: 3D Optical Flow Map 802: Block-based motion estimation techniques 804: Optical Flow Motion Estimation Techniques 806: Dynamic Convolution Motion Estimation Technique 1302: Line 1304: Line 1402: point 1404: points 1406: points 1408: points 1410: points 1412: point 1414: points 1416: points 1418: points 1420: points 1422: points 1424: points 1426: Line 1428: Line 1500: Processing 1502: Box 1504: Box 1506: Box 1600: Processing 1602: Box 1604: Box 1606: Box 1608: Frame 1700: Computing Device Architecture 1705: Connection 1710: Processing unit (CPU or processor) 1712: Cache 1715: Storage Memory 1720: Read-only memory (ROM) 1725: Random Access Memory (RAM) 1730: Storage devices 1732: Service 1 1734: Service 2 1735: Output device 1736: Service 3 1740: Communication Interface 1745: Input devices

Claims

1. A system for processing video data, the system comprising: At least one processor, implemented by circuitry, is configured to: obtain a first reference frame and a second reference frame; The processor generates a third reference frame at least partially by performing interpolation between the first and second reference frames; performs unidirectional inter-frame prediction on the input frame based on the third reference frame, wherein, in order to generate the third reference frame at least partially by performing interpolation between the first and second reference frames, the at least one processor is further configured to: determine a first set of motion information representing pixel motion from the first reference frame to the third reference frame; determine a second set of motion information representing pixel motion from the second and third reference frames; perform a deformation function on the first reference frame at least partially using the first set of motion information to generate first deformation information; perform the deformation function on the second reference frame at least partially using the second set of motion information to generate second deformation information; and generate the third reference frame based on the first deformation information and the second deformation information.

2. The system according to claim 1, wherein, The at least one processor is further configured to: estimate motion between the input frame and the third reference frame; and generate a deformed frame by deforming one or more pixels of the third reference frame, at least in part, based on the estimated motion.

3. The system according to claim 2, wherein, The at least one processor is further configured to: determine the residual at least in part by determining the difference between the input frame and the deformed frame; And use the residuals to generate prediction residuals.

4. The system according to claim 3 further includes a memory, the at least one processor being coupled to the memory and further configured to: generate a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame; and wherein, The at least one processor is further configured to store the reconstructed frame in the memory.

5. The system according to claim 4, wherein, The at least one processor includes a neural processing unit (NPU).

6. The system according to claim 3, wherein, The at least one processor further includes a first neural network and a second neural network.

7. The system according to claim 6, wherein, The first neural network includes a first autoencoder, and the second neural network includes a second autoencoder.

8. The system according to claim 1, wherein, The at least one processor is further configured to: obtain a fourth reference frame; estimate motion between an additional input frame and the fourth reference frame; generate an additional deformed frame by deforming one or more pixels of the fourth reference frame, at least in part, based on the estimated motion; and generate a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

9. The system according to claim 1, wherein, The at least one processor is further configured to: determine the first set of motion information and the second set of motion information based on the pixel motion between the first reference frame and the second reference frame.

10. The system according to claim 1, wherein, The first set of motion information includes a first optical flow map, and the second set of motion information includes a second optical flow map.

11. The system according to claim 1, wherein, The deformation function includes a bilinear interpolation function.

12. The system according to claim 2, wherein, The estimated motion between the input frame and the third reference frame includes optical flow data.

13. The system according to claim 2, wherein, The estimated motion between the input frame and the third reference frame includes dynamic convolutional data.

14. The system according to claim 2, wherein, The estimated motion between the input frame and the third reference frame includes block-based motion data.

15. The system according to claim 1, wherein, The system is one of a mobile device, an extended reality device, or a television.

16. The system according to claim 1, further comprising: At least one of a display or camera configured to capture one or more video frames.

17. A method for processing video data, the method comprising: Obtain the first reference frame and the second reference frame; A third reference frame is generated at least partially by performing interpolation between the first and second reference frames; a one-way inter-frame prediction is performed on an input frame based on the third reference frame; a first set of motion information representing pixel motion from the first reference frame to the third reference frame is determined; a second set of motion information representing pixel motion from the second reference frame to the third reference frame is determined; a deformation function is performed on the first reference frame at least partially using the first set of motion information to generate first deformation information; the deformation function is performed on the second reference frame at least partially using the second set of motion information to generate second deformation information; and the third reference frame is generated based on the first deformation information and the second deformation information.

18. The method according to claim 17 further includes: Estimate the motion between the input frame and the third reference frame; And at least in part by deforming one or more pixels of the third reference frame based on the estimated motion, to generate a deformed frame.

19. The method according to claim 18, further comprising: The residual is determined at least in part by determining the difference between the input frame and the deformed frame; And use the residuals to generate prediction residuals.

20. The method according to claim 19, further comprising: Based on the deformed frame and the prediction residual, a reconstructed frame representing the input frame is generated, the reconstructed frame including a bidirectional prediction frame; And to store the reconstructed frame in memory.

21. The method according to claim 19, wherein, The motion is estimated using a first autoencoder, and the predicted residual is generated using a second autoencoder.

22. The method according to claim 17 further includes: Obtain the fourth reference frame; Estimate the motion between the additional input frame and the fourth reference frame; At least in part, one or more pixels of the fourth reference frame are deformed based on the estimated motion to generate an additional deformed frame; and based on the additional deformed frame and an additional prediction residual, a reconstructed frame representing the additional input frame is generated, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

23. The method according to claim 17, wherein, The first set of motion information and the second set of motion information are determined based on the pixel motion between the first reference frame and the second reference frame.

24. The method according to claim 17, wherein, The first set of motion information includes a first optical flow map, and the second set of motion information includes a second optical flow map.

25. The method according to claim 17, wherein, The deformation function includes a bilinear interpolation function.

26. The method according to request item 18, wherein, The estimated motion between the input frame and the third reference frame includes optical flow data.

27. The method according to request item 18, wherein, The estimated motion between the input frame and the third reference frame includes dynamic convolutional data.

28. The method according to request item 18, wherein, The estimated motion between the input frame and the third reference frame includes block-based motion data.

29. A non-transitory computer-readable medium having instructions thereon that, when executed by one or more processors, cause the one or more processors to perform the following operations: obtaining a first reference frame and a second reference frame; generating a third reference frame at least partially by performing interpolation between the first reference frame and the second reference frame; and performing unidirectional inter-frame prediction on an input frame based on the third reference frame, wherein the non-transitory computer-readable medium further comprises instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: determining a first set of motion information representing pixel motion from the first reference frame to the third reference frame; determining a second set of motion information representing pixel motion from the second reference frame to the third reference frame; performing a warp function on the first reference frame at least partially using the first set of motion information to generate first warp information; performing the warp function on the second reference frame at least partially using the second set of motion information to generate second warp information; and generating the third reference frame based on the first warp information and the second warp information.

30. The non-transitory computer-readable medium according to claim 29, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: estimate motion between the input frame and the third reference frame; and generate a deformed frame by deforming one or more pixels of the third reference frame, at least in part, based on the estimated motion.

31. The non-transitory computer-readable medium according to claim 30, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: determine a residual at least in part by determining the difference between the input frame and the deformed frame; and generate a prediction residual using the residual.

32. The non-transitory computer-readable medium according to claim 31, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame; and storing the reconstructed frame in memory.

33. The non-transitory computer-readable medium according to claim 31, wherein, The motion is estimated using a first autoencoder, and the predicted residual is generated using a second autoencoder.

34. The non-transitory computer-readable medium according to claim 29, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: obtaining a fourth reference frame; estimating motion between an additional input frame and the fourth reference frame; generating an additional deformed frame by deforming one or more pixels of the fourth reference frame at least in part based on the estimated motion; and generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

35. An apparatus for processing video data, the apparatus comprising: Components used to obtain the first reference frame and the second reference frame; A component for generating a third reference frame at least partially by performing interpolation between the first and second reference frames; a component for performing unidirectional inter-frame prediction on an input frame based on the third reference frame; a component for determining a first set of motion information representing pixel motion from the first reference frame to the third reference frame; a component for determining a second set of motion information representing pixel motion from the second and third reference frames; a component for generating first deformation information by performing a deformation function on the first reference frame at least partially using the first set of motion information; and a component for generating second deformation information by performing the deformation function on the second reference frame at least partially using the second set of motion information. And a component for generating the third reference frame based on the first deformation information and the second deformation information.

36. The apparatus of claim 35 further comprises: means for estimating motion between the input frame and the third reference frame; and means for generating a deformed frame by deforming one or more pixels of the third reference frame, at least in part, based on the estimated motion.

37. The apparatus of claim 36 further comprises: a component for determining a residual at least in part by determining the difference between the input frame and the deformed frame; and a component for generating a predicted residual using the residual.

38. The apparatus of claim 37, further comprising: means for generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame; and means for storing the reconstructed frame in memory.

39. The apparatus of claim 37, wherein the member for estimating motion is configured to estimate motion using a first autoencoder, and wherein the prediction residual is generated using a second autoencoder.

40. The apparatus of claim 35, further comprising: means for obtaining a fourth reference frame; means for estimating motion between an additional input frame and the fourth reference frame; means for generating an additional deformed frame by deforming one or more pixels of the fourth reference frame at least in part based on the estimated motion; and means for generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

41. A system for processing video data, the system comprising: One memory; and at least one processor coupled to the memory, the at least one processor being configured to: obtain an interpolation reference frame and a potential representation of motion between an input frame and the interpolation reference frame; process the potential representation of motion to determine a predicted motion between the input frame and the interpolation reference frame; generate a deformed frame by deforming one or more pixels of the interpolation reference frame at least in part based on the predicted motion; and generate a reconstructed frame representing the input frame based on the deformed frame and a prediction residual, the reconstructed frame including a bidirectional prediction frame.

42. The system according to claim 41, wherein the at least one processor is configured to: obtain a first reference frame and a second reference frame; and generate the interpolated reference frame at least in part by performing interpolation on the first reference frame and the second reference frame.

43. The system according to claim 41, wherein the at least one processor is configured to: obtain residuals from a self-biological stream; and generate the predicted residuals based on the obtained residuals.

44. The system according to claim 43, wherein the potential representation of the motion is obtained from the bit stream.

45. The system of claim 41, wherein the at least one processor is configured to: obtain a fourth reference frame; estimate motion between an additional input frame and the fourth reference frame; generate an additional deformed frame by deforming one or more pixels of the fourth reference frame at least partially based on the predicted motion; and generate a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

46. ​​A method for processing video data, the method comprising: Obtain the interpolation reference frame and the potential representation of motion between the input frame and the interpolation reference frame; The process involves processing the latent representation of the motion to determine a predicted motion between the input frame and the interpolated reference frame; generating a deformed frame by deforming one or more pixels of the interpolated reference frame at least in part based on the predicted motion; and generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame.

47. The method according to claim 46 further includes: Obtain the first reference frame and the second reference frame; The interpolated reference frame is generated at least in part by interpolating the first reference frame with the second reference frame.

48. The method according to claim 46 further includes: Self-segmented flow obtains residuals; And the predicted residual is generated based on the obtained residual.

49. The method according to claim 48, wherein the potential representation of the motion is obtained from the bit stream.

50. The method according to claim 46 further includes: Obtain the fourth reference frame; Estimate the motion between the other input frame and the fourth reference frame; At least in part, one or more pixels of the fourth reference frame are deformed based on the predicted motion to generate an additional deformed frame; and a reconstructed frame representing the additional input frame is generated based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

51. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the following operations: obtaining an interpolation reference frame and a latent representation of motion between an input frame and the interpolation reference frame; processing the latent representation of motion to determine a predicted motion between the input frame and the interpolation reference frame; generating a deformed frame by deforming one or more pixels of the interpolation reference frame at least in part based on the predicted motion; and generating a reconstructed frame representing the input frame based on the deformed frame and a prediction residual, the reconstructed frame including a bidirectional prediction frame.

52. The non-transitory computer-readable medium according to claim 51, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: obtaining a first reference frame and a second reference frame; and generating the interpolated reference frame at least in part by performing interpolation on the first reference frame and the second reference frame.

53. The non-transitory computer-readable medium according to claim 51, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: obtain a residual from the bitstream; and generate the predicted residual based on the obtained residual.

54. The method according to claim 53, wherein the potential representation of the motion is obtained from the bit stream.

55. The non-transitory computer-readable medium according to claim 51, further comprising, when executed by the one or more processors, instructions that cause the one or more processors to perform the following operations: obtaining a fourth reference frame; estimating motion between an additional input frame and the fourth reference frame; generating an additional deformed frame by deforming one or more pixels of the fourth reference frame at least in part based on the predicted motion; and generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

56. An apparatus for processing video data, the apparatus comprising: Components for obtaining an interpolation reference frame and a potential representation of motion between the input frame and the interpolation reference frame; The components include: a component for processing the latent representation of the motion to determine the predicted motion between the input frame and the interpolation reference frame; a component for generating a deformed frame by deforming one or more pixels of the interpolation reference frame at least in part based on the predicted motion; and a component for generating a reconstructed frame representing the input frame based on the deformed frame and the prediction residual, the reconstructed frame including a bidirectional prediction frame.

57. The apparatus according to claim 56 further includes: A component for obtaining a first reference frame and a second reference frame; and a component for generating the interpolated reference frame by at least partially interpolating the first reference frame and the second reference frame.

58. The apparatus according to claim 56, further comprising: Components used to obtain residuals from autogenous flow; And components for generating the predicted residual based on the obtained residual.

59. The apparatus according to claim 58, wherein the potential representation of the motion is obtained from the bit stream.

60. The apparatus according to claim 56 further includes: Components used to obtain the fourth reference frame; A component used to estimate the motion between the additional input frame and the fourth reference frame; The means for generating an additional deformed frame by deforming one or more pixels of the fourth reference frame at least in part based on the predicted motion; and the means for generating a reconstructed frame representing the additional input frame based on the additional deformed frame and an additional prediction residual, wherein the reconstructed frame representing the additional input frame includes a unidirectional prediction frame.

Images