Video coding using camera motion compensation and object motion compensation

Abstract

Systems and techniques are provided for coding (e.g., encoding and / or decoding) video data using camera motion information. For example, a decoding device may obtain an encoded video data frame associated with an input frame, the encoded video data frame including camera information and a residual associated with generating the video data. A camera motion compensation frame may be generated based on a reference frame and the camera information. Optical flow information associated with object motion determined based at least on the input frame and the reference frame may be generated. A motion compensation frame may be generated by normalizing the camera motion compensation frame based on the optical flow information. A reconstructed input frame may be generated based on the motion compensated frame and the residual.

Description

Technical Field

[0001] The present disclosure generally relates to video coding (eg, encoding and / or decoding video data). For example, aspects of the present disclosure relate to systems and techniques for performing camera motion compensation and object motion compensation for video coding. Background Art

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes large amounts of data to meet the growing demands for video quality, performance, and features. For example, consumers of video data generally expect high-quality video with high fidelity, high resolution, high frame rate, etc. As a result, the large amounts of video data that are generally required to meet these demands place a burden on communication networks and devices that process and store video data.

[0003] Video coding techniques can be used to compress video data. One example goal of video coding is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. As evolving video services become available and demand for large amounts of video data continues to increase, coding techniques with better performance and efficiency are needed. Summary of the invention

[0004] In some examples, systems and techniques for decoding (e.g., encoding and / or decoding) video data using a machine learning architecture are described. For example, the systems and techniques may use one or more machine learning networks (e.g., neural networks) to perform video decoding for rendered video data and / or other video data based on depth information and / or camera pose information associated with the rendered video data. According to at least one illustrative example, a device for decoding video data is provided, the device comprising at least one memory (e.g., the at least one memory is configured to store data, such as virtual content data, one or more images, etc.) and one or more processors coupled to the at least one memory (e.g., the one or more processors are implemented in a circuit). The one or more processors are configured and capable of: obtaining an encoded video data frame associated with an input frame, the encoded video data frame including camera information and residuals associated with generating the video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with an object motion determined based at least on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; and generating a reconstructed input frame based on the motion compensated frame and the residual.

[0005] In another example, a method for decoding video data is provided, the method comprising: obtaining an encoded video data frame associated with an input frame, the encoded video data frame comprising camera information and a residual associated with generating the video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with an object motion determined based at least on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; and generating a reconstructed input frame based on the motion compensated frame and the residual.

[0006] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: obtain an encoded video data frame associated with an input frame, the encoded video data frame including camera information and residuals associated with generating the video data; generate a camera motion compensated frame based on a reference frame and the camera information; generate optical flow information associated with an object motion determined based on at least the input frame and the reference frame; generate a motion compensated frame by warping the camera motion compensated frame based on the optical flow information; and generate a reconstructed input frame based on the motion compensated frame and the residual.

[0007] In another example, an apparatus for decoding video data is provided. The apparatus includes: a component for obtaining an encoded video data frame associated with an input frame, the encoded video data frame including camera information and residuals associated with generating the video data; a component for generating a camera motion compensation frame based on a reference frame and the camera information; a component for generating optical flow information associated with object motion determined based on at least the input frame and the reference frame; a component for generating a motion compensation frame by warping the camera motion compensation frame based on the optical flow information; and a component for generating a reconstructed input frame based on the motion compensation frame and the residuals.

[0008] In another example, a device for encoding video data is provided, the device comprising at least one memory (e.g., the at least one memory is configured to store data, such as virtual content data, one or more images, etc.) and one or more processors coupled to the at least one memory (e.g., the one or more processors are implemented in a circuit). The one or more processors are configured and capable of: obtaining an input frame of video data and camera information associated with generating the input frame of video data; generating a camera motion compensation frame based on a reference frame and the camera information; generating optical flow information associated with an object motion determined based on at least the input frame and the reference frame; generating a motion compensation frame by warping the camera motion compensation frame based on the optical flow information; determining a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensation frame; and generating an encoded video data frame associated with the input frame of video data, the encoded video data frame including the camera information and the residual.

[0009] In another example, a method for encoding video data is provided, the method comprising: obtaining an input frame of video data and camera information associated with the input frame generating the video data; generating a camera motion compensation frame based on a reference frame and the camera information; generating optical flow information associated with an object motion determined based at least on the input frame and the reference frame; generating a motion compensation frame by regularizing the camera motion compensation frame based on the optical flow information; determining a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensation frame; and generating an encoded video data frame associated with the input frame of video data, the encoded video data frame comprising the camera information and the residual.

[0010] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: obtain an input frame of video data and camera information associated with generating the input frame of the video data; generate a camera motion compensated frame based on a reference frame and the camera information; generate optical flow information associated with an object motion determined based at least on the input frame and the reference frame; generate a motion compensated frame by warping the camera motion compensated frame based on the optical flow information; determine a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensated frame; and generate an encoded video data frame associated with the input frame of the video data, the encoded video data frame including the camera information and the residual.

[0011] In another example, a device for encoding video data is provided. The device includes: a component for obtaining an input frame of video data and camera information associated with generating the input frame of video data; a component for generating a camera motion compensation frame based on a reference frame and the camera information; a component for generating optical flow information associated with an object motion determined based at least on the input frame and the reference frame; a component for generating a motion compensation frame by regularizing the camera motion compensation frame based on the optical flow information; a component for determining a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensation frame; and a component for generating an encoded video data frame associated with the input frame of video data, the encoded video data frame including the camera information and the residual.

[0012] In some aspects, the apparatus may include or may be part of a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a television, a video game console, or other device. In some aspects, the apparatus includes a mobile device (e.g., a mobile phone or a so-called "smart phone"). In some aspects, the apparatus also includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more images and / or one or more videos including video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to send the reconstructed video frame to at least one device on a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing devices or components.

[0013] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0014] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0016] Figure 1An example image processing system is illustrated that can implement various techniques described herein according to some examples;

[0017] Figure 2A An example of a fully connected neural network according to some examples is illustrated;

[0018] Figure 2B An example of a locally connected neural network according to some examples is illustrated;

[0019] Figure 2C An example of a convolutional neural network according to some examples is illustrated;

[0020] Figure 2D illustrates a detailed example of a deep convolutional network (DCN) designed to recognize features from an image according to some examples;

[0021] Figure 3 is a block diagram illustrating another example DCN according to some examples;

[0022] Figure 4 is a diagram illustrating an example of a system including a device operable to perform image and / or video coding (e.g., encoding and decoding) using a neural network-based system according to some examples;

[0023] Figure 5 is a diagram illustrating an example of a motion estimation technique according to some examples;

[0024] Figure 6 is a diagram illustrating an example neural network-based video codec (e.g., a coder / decoder) that may be used to perform video decoding for rendered content according to some examples;

[0025] Figure 7 is a diagram illustrating an example of a neural video coding system that may be used to perform unidirectional decoding and / or bidirectional decoding according to some examples;

[0026] Figure 8 is a diagram illustrating an example of a machine learning based video coding system including a motion compensation subnetwork and an object motion compensation subnetwork according to some examples;

[0027] Fig. 9 is a diagram illustrating another example of a machine learning based video coding system including a motion compensation subnetwork and an object motion compensation subnetwork according to some examples;

[0028] Fig.10 is a diagram illustrating another example of a machine learning based video coding system including a motion compensation subnetwork and an object motion compensation subnetwork according to some examples;

[0029] Fig.11is a flowchart illustrating an example of a process for processing video data according to some examples; and

[0030] Fig.12 An example computing system is illustrated that can be used to implement various aspects described herein. DETAILED DESCRIPTION

[0031] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0032] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0033] Digital video data can include large amounts of data, especially as the demand for high-quality video data continues to grow. For example, consumers of video data generally expect increasingly higher quality video, with high fidelity, high resolution, high frame rate, etc. However, the large amounts of video data that are typically required to meet such demands can place a significant burden on communication networks and devices that process and store video data.

[0034] Various techniques may be used to decode video data. In some cases, video coding may be performed according to a specific video coding standard and / or scheme. Example video coding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding, Versatile Video Coding (VVC), etc. One goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. As the demand for video services grows and new video services become available, coding techniques with better efficiency and performance are needed.

[0035] Video coding may use a prediction method (such as intra-frame prediction or inter-frame prediction) that exploits redundancy present in video frames or other sequences of images or frames. Intra-frame prediction is performed using data within a single video frame and is based on the spatial characteristics of the frame. Frames that are decoded using intra-frame prediction are called I-frames. Inter-frame prediction of frames is performed based on the temporal characteristics of the frame relative to other frames. For example, inter-frame prediction of video frames may be performed in the following manner: identifying areas of other video frames that include changes relative to the video frame and areas that include redundancy relative to the video frame (e.g., background areas that remain mostly unchanged). Redundancy may be removed, thereby generating a residual of a video frame. The residual may be further encoded (e.g., using entropy coding), and the result may be included in a bitstream that is stored, transmitted, or otherwise output.

[0036] Examples of inter prediction include unidirectional prediction (Uni-directional prediction) and bidirectional prediction (Bi-directional prediction). Unidirectional prediction involves using a single reference frame when performing inter prediction of a frame. Frames coded using unidirectional prediction are called predicted frames (P frames). Bidirectional prediction involves using two reference frames when performing inter prediction of a frame. Frames coded using bidirectional prediction are called bidirectional predicted frames (B frames).

[0037] In some cases, machine learning systems can be used to perform video encoding (compression) and decoding (decompression). Generally speaking, machine learning (ML) is a subset of artificial intelligence (AI). ML systems include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences. An example of an ML system is a neural network (also called an artificial neural network), which may include a group of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in various applications and / or devices, such as image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc.

[0038] Individual nodes in a neural network can simulate biological neurons by obtaining input data and performing operations on the data. The results of the operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​limit the way in which the input data is related to the output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be added. The sum of the products can be tuned by an optional bias, and an activation function can be applied to the result to generate an output signal or "output activation" (sometimes referred to as an activation map or feature map) of the node. The weight values ​​can initially be determined by the iterative flow of training data through the network. For example, weight values ​​can be established during the training phase, where the network learns how to identify a specific category by the typical input data characteristics of a specific category. In one example, the network can be trained to learn a specific task by adapting the values ​​of parameters associated with neurons (e.g., activation parameters and / or weights, biases, etc.), adding and / or removing neurons or even neuron layers, adding or removing edges between neurons, etc.

[0039] There are different types of neural networks, such as autoencoders, convolutional neural networks (CNNs), recurrent neural networks (RNNs), multilayer perceptron (MLP) neural networks, etc. A convolutional neural network may include a collection of artificial neurons that each have a receptive field (e.g., a spatially local region of an input space) and collectively tile an input space. Convolutional neural networks have numerous applications. For example, CNNs are widely used in the fields of pattern recognition and classification. RNNs work on the following principle: save the output of a layer and feed the output back to the input to help predict the result of the layer. In an MLP neural network, data can be fed to an input layer, and one or more hidden layers provide an abstraction level for the data. The output layer can then be predicted based on the abstract data. MLPs may be particularly suitable for classification prediction problems that assign categories or labels to inputs.

[0040] In a layered neural network architecture (called a deep neural network when there are multiple hidden layers), the outputs of the first layer of artificial neurons become the inputs of the second layer of artificial neurons, the outputs of the second layer of artificial neurons become the inputs of the third layer of artificial neurons, and so on. Convolutional neural networks can be trained to recognize hierarchical structures of features. The computations in a convolutional neural network architecture can be distributed across a set of processing nodes, which can be configured in one or more computation chains. These multi-layer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.

[0041] In some examples, machine learning-based P-frame and B-frame decoding systems may be used to perform unidirectional prediction and bidirectional prediction, respectively. In some cases, such systems may include a neural network architecture (e.g., one or more deep neural networks, such as one or more autoencoders). An example of a machine learning-based P-frame decoding system may perform motion compression and motion compensation on a current frame and a reference frame to determine a prediction of motion between the current frame and the reference frame. Motion prediction may be used to modify pixels of a reference frame (e.g., by moving pixels of a reference frame according to a motion vector included in the motion prediction) to produce a prediction of the current frame. The residual portion of the P-frame decoding system may generate a prediction residual representing the difference between the prediction and the current frame. The predicted residual may be combined with the prediction to generate a reconstructed current frame.

[0042] Described herein are systems, methods (also referred to as processes), devices, and computer-readable media (collectively, "systems and techniques") for performing decoding of video data based on using rendering information and / or additional information associated with the video data to perform camera motion compensation and object motion compensation. As used herein, the term decoding may refer to encoding (e.g., compression), decoding (e.g., decompression), or both encoding and decoding. For example, the systems and techniques described herein may perform video decoding of frames of video content (e.g., frames of rendered and / or game content) based on using separate motion prediction systems to perform camera motion compensation and to perform object motion compensation. In some cases, the systems and techniques may perform camera motion compensation based on rendering information associated with one or more frames of the video content.

[0043] In some aspects, the rendering information associated with one or more frames of the currently encoded and / or decoded video content may include the current camera pose p t (e.g., the camera pose associated with the current frame t), the reference camera pose p t-1 (e.g., the camera pose associated with the previous frame t-1) and the depth map D associated with the current frame t t . In some cases, the camera pose information may include position and orientation information of the camera relative to a reference coordinate system of a scene depicted in the video content (e.g., a scene captured by the camera). A depth map may be an image that includes information associated with the distance of the surface of a scene object from a given viewpoint (e.g., the viewpoint of a camera that captured the scene). In some examples, the depth map may have the same pixel size as the corresponding frame of video data for which the depth map was generated, wherein some (or all) of the pixels included in the depth map are associated with depth values. The depth value associated with a given pixel may indicate the distance from the camera viewpoint to the surface of the scene object depicted by the given pixel.

[0044] In examples where the frames of the encoded and / or decoded video content are frames of rendered video data, camera pose information and depth information may be generated in association with the rendered video frames (e.g., generated when the video frames are initially generated, prior to encoding or decoding). For example, based on the camera pose and depth information that the rendering engine has used to initially generate the rendered frames, the camera pose and depth information may be obtained from one or more rendering engines associated with generating the rendered frames. In some aspects, the camera pose and depth information associated with the frames of the rendered game content may be obtained from one or more buffers associated with a rendering or game engine used to generate the game content. For example, the camera pose and depth information associated with the frames of the rendered game content generated by the game engine and / or the cloud game server may be obtained from a deferred rendering buffer and / or a geometry buffer (G-buffer) associated with the game engine. In some examples, the camera pose and depth information may be stored in a deferred rendering buffer or a G-buffer in various other types of rendering information (e.g., motion information, optical flow maps, normal maps, albedo maps, etc.).

[0045] In some examples, the systems and techniques described herein may be used to perform video decoding based on separate camera motion compensation and object motion compensation for non-rendered video content (eg, natural video content). t and p t-1 The depth information may be obtained from a camera used to capture non-rendered or natural video content, may be determined using one or more post-processing algorithms or machine learning networks, etc. In some cases, the depth information may be obtained from a camera that includes one or more depth sensors and captures depth information and video content simultaneously (e.g., such that the video content is captured in conjunction with a corresponding depth map for one or more frames of the captured video content). In some aspects, the depth information may be obtained from a stereo camera that includes two or more cameras for determining a depth map.

[0046] In some examples, a depth map prediction system and a three-dimensional (3D) regularization engine may be used to generate an initial frame prediction. It uses the input camera pose p t , reference camera pose p t-1 and the depth map D t In some cases, the initial frame prediction It can also be called a camera motion compensation frame. The 3D warping engine can receive the camera pose p of the current frame t based on the camera pose p of the current frame t. t , the reference camera pose p of the previous frame t-1 t-1 , previously decoded (e.g., reconstructed) frames and the predicted (e.g., reconstructed) depth map for the current frame t Takes as input to generate initial frame predictions Camera pose pt and p t-1 Together with the depth map D t The neural video encoder may be generated by a rendering engine (e.g., a cloud gaming engine) associated with the neural video encoder. In some examples, the neural video encoder may be included in the same server or other computing device as the rendering engine used to generate the frames of rendered video data being encoded (e.g., which may also be the server or other computing device used to generate the camera pose p t and p t-1 Together with the depth map D t The same rendering engine as .

[0047] Additional aspects of the systems and techniques will be described with respect to the accompanying figures. Figure 1 An example implementation of an image processing system 100 is illustrated, which in some cases can be used to implement the systems and techniques described herein. The image processing system 100 may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, image data, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed in multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102 and / or from the memory block 118.

[0048] The image processing system 100 may also include additional processing blocks for performing specific functions, such as a GPU 104; a DSP 106; a connectivity block 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.; and / or a multimedia processor 112, which may, for example, detect image features. In some examples, the NPU 108 may be implemented in the CPU 102, the DSP 106, and / or the GPU 104. In some cases, the image processing system 100 may also include one or more sensors 114, one or more image signal processors (ISPs) 116, and / or a storage device 120.

[0049] In some examples, the image processing system 100 may implement an ARM instruction set architecture for one or more processors. In one aspect of the present disclosure, the instructions loaded into the CPU 102 may include code for searching a lookup table (LUT) for a stored multiplication result corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit for the multiplication product is detected. In addition, the instructions loaded into the CPU 102 may include code for storing a calculated multiplication product of an input value and a filter weight when a lookup table miss for the multiplication product is detected.

[0050] The image processing system 100 may be part of a computing device or multiple computing devices. In some examples, the image processing system 100 may be part of an electronic device (or multiple devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a surveillance camera, etc.), a phone system (e.g., a smart phone, a cellular phone, a conferencing system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smart watch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a system on a chip (SOC), a digital media player, a game console, a video streaming device, a server, a drone, a vehicle computer, an Internet of Things (IoT) device, or any other suitable electronic device.

[0051] In some implementations, the CPU 102, GPU 104, DSP 106, NPU 108, connectivity block 110, multimedia processor 112, one or more sensors 114, ISP 116, memory block 118, and / or storage 120 may be part of the same computing device. For example, in some cases, the CPU 102, GPU 104, DSP 106, NPU 108, connectivity block 110, multimedia processor 112, one or more sensors 114, ISP 116, memory block 118, and / or storage 120 may be integrated into a smartphone, a laptop, a tablet, a smart wearable device, a video game system, a server, and / or any other computing device. In other implementations, CPU 102, GPU 104, DSP 106, NPU 108, connectivity block 110, multimedia processor 112, one or more sensors 114, ISP 116, memory block 118, and / or storage 120 may be part of two or more separate computing devices.

[0052] The image processing system 100 and / or its components may be configured to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video decoding) using techniques according to various aspects of the present disclosure discussed herein. By using a deep learning architecture and the techniques described herein to perform video compression and / or decompression, various aspects of the present disclosure may improve the efficiency of video compression and / or decompression on a device and / or reduce associated resource requirements and usage. For example, a device using the video decoding techniques described herein may more efficiently compress video data, may reduce the amount of data sent to a target device in compressed video data, and the destination device may efficiently receive and decompress compressed video data. In some examples, the deep learning architecture and techniques described herein may reduce the amount of data exchanged between decoding devices or components (such as encoders and decoders) to decode video content. The reduced amount of data sent for video decoding may reduce latency, increase performance, and reduce the cost or burden of computing resources (such as, for example, bandwidth, memory, storage, power, computing, hardware, etc.).

[0053] As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of the neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. The node may include one or more weights to indicate the importance of the node of one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, where the early layers are used to determine simple and low-level features of the input, while the later layers establish a hierarchy of more complex and abstract features.

[0054] A deep learning architecture can learn a hierarchy of features. For example, if presented with visual data, the first layer can learn to recognize features (such as edges) in the input stream. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize features such as shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Still higher layers can learn to recognize common visual objects and / or spoken phrases.

[0055] Neural networks can be designed to have a variety of connectivity patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0056] The connections between neural network layers can be fully connected or partially connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In the fully connected neural network 202, a neuron in a first hidden layer may communicate its output to each neuron in a second hidden layer, such that each neuron in the second layer will receive an input from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first hidden layer may be connected to a limited number of neurons in a second hidden layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strengths may have different values ​​(e.g., 210, 212, 214, and 216). The locally connected connectivity patterns may produce spatially different receptive fields in higher layers because higher layer neurons in a given area may receive input that is tuned through training to properties of a limited portion of the total input to the network.

[0057] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is shown. The convolutional neural network 206 can be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful. According to aspects of the present disclosure, the convolutional neural network 206 can be used to perform one or more aspects of video compression and / or decompression.

[0058] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2DAn example of a DCN 200 designed to recognize features from an image 226 input from an image capture device 230 (such as a camera or image sensor) is illustrated. In some examples, the DCN 200 of the current example can be trained to identify visual features in the image 226, such as, for example, one or more objects or signs in the image 226.

[0059] In some examples, DCN 200 can be trained by supervised learning. During training, an image (such as image 226) can be presented to DCN 200, and a forward pass can then be calculated to produce output 222. DCN 200 can include a feature extraction portion and a classification portion. Upon receiving image 226, convolution layer 232 can apply a convolution kernel (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolution kernel of convolution layer 232 can be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolution kernels are applied to image 226 at convolution layer 232. Convolution kernels can also be referred to as filters or convolution filters.

[0060] The first set of feature maps 218 may be subsampled by a maximum pooling layer (not shown) to generate a second set of feature maps 220. The maximum pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (such as 14×14) is smaller than the size of the first set of feature maps 218 (such as 28×28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0061] exist Figure 2D In the example of , the second set of feature maps 220 are convolved to generate a first feature vector 224. In addition, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of the DCN 200 is the probability that the image 226 includes one or more features.

[0062] In this example, the probability of "logo" and "60" in output 222 is higher than the probability of other numbers (such as "30", "40", "50", "70", "80", "90", and "100") in output 222. Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth value of image 226 (e.g., "logo" and "60"). The weights of DCN 200 can then be adjusted so that the output 222 of DCN 200 is closer to the target output.

[0063] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease when the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0064] In practice, the error gradient of the weights can be calculated over a small number of examples so that the calculated gradient is close to the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, new images can be presented to the DCN, and a forward pass through the network can produce an output 222 that can be considered an inference or prediction of the DCN.

[0065] A Deep Belief Network (DBN) is a probabilistic model that includes multiple layers of hidden nodes. A DBN can be used to extract a hierarchical representation of a training data set. A DBN can be obtained by stacking layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn probability distributions without information about the category to which each input should be classified, RBMs are often used for unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of inputs from the previous layer and the target class) and can be used as a classifier.

[0066] A deep convolutional network (DCN) is a network that is a convolutional network configured with additional pooling and normalization layers. DCN can achieve high performance on many tasks. DCN can be trained using supervised learning, where both the input target and the output target are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0067] The DCN may be a feed-forward network. In addition, as described above, the connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN may be used for fast processing. For example, the computational burden of the DCN may be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0068] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then the convolutional network trained on this input can be thought of as three-dimensional, where two spatial dimensions are along the axes of the image and the third dimension captures color information. The output of the convolutional connection can be thought of as forming a feature map in a subsequent layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as rectification, max(0,x)). The values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0069] Figure 3 is a block diagram illustrating an example of a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. Figure 3 As shown, the deep convolution network 350 includes convolution blocks 354A and 354B. Each of the convolution blocks 354A and 354B may be configured with a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0070] The convolution layer 356 may include one or more convolution filters that can be applied to the input data 352 to generate a feature map. Although only two convolution blocks 354A, 354B are shown, the present disclosure is not limited thereto, and any number of convolution blocks (e.g., blocks 354A, 354B) may be included in the deep convolution network 350 according to design preferences. The normalization layer 358 may normalize the output of the convolution filter. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximum pooling layer 360 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0071] For example, the parallel filter bank of the deep convolutional network can be loaded onto the CPU 102 or GPU 104 of the image processing system 100 to achieve high performance and low power consumption. In some examples, the parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the image processing system 100. The deep convolutional network 350 can access other processing blocks that may exist on the image processing system 100.

[0072] The deep convolutional network 350 may include one or more fully connected layers, such as layer 362A (labeled “FC1”) and layer 362B (labeled “FC2”). The deep convolutional network 350 may include a logistic regression (LR) layer 364. Each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 has weights (not shown) to be updated between them. The output of each of these layers (e.g., 356, 358, 360, 362, 364) can be used as an input to a subsequent layer of these layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) supplied at the first convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is a probability that the input data includes a feature from a set of features.

[0073] Another type of neural network is an autoencoder. An autoencoder can be trained (e.g., using training data and one or more loss functions) to receive an input and generate a version of that input at its output (e.g., essentially copying its input to its output). An autoencoder can be trained to learn efficient data decoding in an unsupervised manner. For example, given an image of an object, an autoencoder can first encode the image into a lower dimensional latent representation, and can then decode the latent representation back into the image of the object. An autoencoder can learn (through training) to compress the input data while minimizing reconstruction error.

[0074] As previously mentioned, digital video data may include large amounts of data, which may place a significant burden on communication networks and equipment that processes and stores video data. For example, recording uncompressed video content typically results in large file sizes that increase substantially as the resolution of the recorded video content increases. In an illustrative example, an uncompressed 16-bit per channel video recorded in a 1080p / 24 format (e.g., a resolution of 1920 pixels wide and 1080 pixels high, where 24 frames are captured per second) may take up 12.4 megabytes per frame, or 297.6 megabytes per second. An uncompressed 16-bit per channel video recorded at a 4K resolution of 24 frames per second may take up 49.8 megabytes per frame, or 1195.2 megabytes per second.

[0075] Network bandwidth is another constraint where large video files can be problematic. For example, video content is often delivered over wireless networks (e.g., via LTE, LTE-Advanced, New Radio (NR), WiFi™, Bluetooth™, or other wireless networks) and can make up a large portion of consumer Internet traffic. Therefore, it is desirable to reduce the amount of bandwidth used to deliver video content in these networks.

[0076] Because uncompressed video content may produce large files that may involve considerable memory for physical storage and considerable bandwidth for transmission, video coding techniques may be utilized to compress and decompress such video content, as further described herein.

[0077] In order to reduce the size of video content - and therefore reduce the amount of storage involved in storing the video content and the amount of bandwidth involved in delivering the video content - various video coding techniques may be performed according to a particular video coding standard and / or scheme such as HEVC, AVC, MPEG, VVC, etc. Video coding may use prediction methods (such as inter-frame prediction or intra-frame prediction) that exploit redundancy present in a video image or sequence. One goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. As the demand for video services grows and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0078] The encoding device may encode the video data according to the video coding standard to generate an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") may include a series of one or more decoded video sequences. The encoding device may generate a decoded representation of the picture by dividing each picture into multiple slices. The slice is independent of other slices, so that the information in the slice can be decoded without relying on data from other slices in the same picture. The slice includes one or more slice segments, which include independent slice segments and (if present) one or more dependent slice segments that depend on the previous slice segment. In HEVC, the slice is divided into coding tree blocks (CTBs) of luminance samples and chrominance samples. The CTB of the luminance sample and one or more CTBs of the chrominance sample together with the syntax for the sample are referred to as a coding tree unit (CTU). CTU may also be referred to as a "tree block" or a "maximum coding unit" (LCU). CTU is the basic processing unit for HEVC encoding. CTU can be split into multiple coding units (CUs) of different sizes. A CU contains an array of luma samples and an array of chroma samples called coding blocks (CBs).

[0079] Luma CB and chroma CB can be further split into prediction blocks (PB). PB is a sample block of luma component or chroma component, which uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled). Luma PB and one or more chroma PBs together with associated syntax form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) are signaled in the bitstream for each PU, as well as inter prediction for luma PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. CB may also be divided into one or more transform blocks (TB). TB represents a square block of samples of a color component, to which a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to decode the prediction residual signal. Transform unit (TU) represents a TB of luma samples and chroma samples and corresponding syntax elements. Transform decoding is described in more detail below.

[0080] According to the HEVC standard, a TU may be used to perform a transform. The size of a TU may be set based on the size of a PU within a given CU. A TU may have the same size as a PU or be smaller than a PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT may correspond to TUs. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. These transform coefficients may then be quantized by an encoding device.

[0081] Once the picture of the video data is partitioned into CUs, the encoding device predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture using, for example, DC prediction to find an average value for the PU, using plane prediction to adapt the plane surface to the PU, using directional prediction to infer from adjacent data, or using any other suitable prediction type. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated prediction for image sample blocks. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to decode a picture area.

[0082] In some examples, a slice type is assigned to one or more slices of a picture. Slice types may include I slices, P slices, and B slices. An I slice (intra, independently decodable) is a slice of a picture that is coded only by intra prediction and is therefore independently decodable because an I slice requires only data within a frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and with unidirectional inter prediction. Each prediction unit or prediction block within a P slice is coded with intra prediction or inter prediction. When inter prediction is applied, the prediction unit or prediction block is predicted by one reference picture, and thus the reference sample comes from one reference region of one frame. A B slice (bidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and inter prediction (e.g., either bidirectional prediction or unidirectional prediction). A prediction unit or prediction block of a B slice can be bidirectionally predicted from two reference pictures. Each picture can contribute a reference region, and the sample sets of the two reference regions can be weighted (e.g., with equal weights or with different weights) to generate a prediction signal for a bidirectionally predicted block. As described above, slices of one picture are coded independently. In some cases, a picture may be coded as only one slice.

[0083] After performing prediction using intra prediction and / or inter prediction, the encoding device may perform transformation and quantization. For example, after prediction, the encoding device may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current pixel block (PU) being decoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., using inter prediction or intra prediction), the encoding device may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantizes the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented by a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of the pixel values.

[0084] Any residual data that may remain after prediction is performed may be transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some aspects, TUs may be used for transform and quantization processes implemented by an encoding device. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TU to produce serialized transform coefficients for entropy coding.

[0085] The encoding device may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0086] Once quantization is performed, the decoded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction mode, motion vector, block vector, etc.), segmentation information, and any other suitable data (such as other syntax data). The different elements of the decoded video bitstream can then be entropy encoded by the encoding device. In some examples, the encoding device may scan the quantized transform coefficients using a predefined scanning order to produce a serialized vector that can be entropy encoded. In some examples, the encoding device may perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoding device may entropy encode the vector. For example, the encoding device may use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another suitable entropy coding technique.

[0087] The encoding device may store the encoded video bitstream and / or may send the encoded video bitstream data to a receiving device that may include a decoding device via a communication link. The decoding device may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more decoded video sequences that constitute the encoded video data. The decoding device may then rescale the encoded video bitstream data and perform an inverse transform on it. The residual data is then passed to the prediction level of the decoding device. The decoding device then predicts a pixel block (e.g., PU) using intra-frame prediction, inter-frame prediction, IBC, and / or other types of prediction. In some examples, the prediction is added to the output (residual data) of the inverse transform. The decoding device may output the decoded video to a video target device, which may include a display or other output device for displaying the decoded video data to a consumer of the content.

[0088] Video coding systems and techniques defined by various video coding standards (e.g., the HEVC video coding techniques described above) may be able to preserve much of the information in the original video content and may be defined a priori based on signal processing and information theory concepts. However, in some cases, machine learning (ML)-based image and / or video systems may provide benefits over non-ML-based image and video coding systems such as end-to-end neural network-based image and video coding (E2E-NNVC) systems. As described above, many E2E-NNVC systems are designed as a combination of an autoencoder subnetwork (encoder subnetwork) and a second subnetwork responsible for learning a probabilistic model on quantized latent data for entropy coding. This architecture can be viewed as a combination of a transform plus quantization module (encoder subnetwork) and an entropy modeling subnetwork module.

[0089] Figure 4A system 400 is depicted that includes a device 402 configured to perform image and / or video encoding and decoding using an E2E-NNVC system 410. The device 402 is coupled to a camera 407 and a storage medium 414 (e.g., a data storage device). In some implementations, the camera 407 is configured to provide image data 408 (e.g., a video data stream) to the processor 404 for encoding by the E2E-NNVC system 410. In some implementations, the device 402 may be coupled to and / or may include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 402 may be coupled to a microphone and / or other input device (e.g., a keyboard, a mouse, a touch input device such as a touch screen and / or a touchpad, and / or other input devices). In some examples, the camera 407, the storage medium 414, the microphone, and / or other input devices may be part of the device 402.

[0090] Device 402 is also coupled to a second device 490 via a transmission medium 418, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 418 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The transmission medium 418 may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 418 may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from a source device to a receiving device. The wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi, or a wireless network). TM , Radio Frequency (RF), UWB, WiFi Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, Ethernet based on coaxial cable, digital signal line (DSL), etc.). The wired network and / or wireless network may be implemented using various equipment (such as base stations, routers, access points, bridges, gateways, switches, etc.). The encoded video bitstream data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to a receiving device.

[0091] The device 402 includes one or more processors (referred to herein as "processors") 404 coupled to a memory 406, a first interface ("I / F 1") 412, and a second interface ("I / F 2") 416. The processor 404 is configured to receive image data 408 from a camera 407, from the memory 406, and / or from a storage medium 414. The processor 404 is coupled to the storage medium 414 via the first interface 412 (e.g., via a memory bus), and to a transmission medium 418 via a second interface 416 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0092] The processor 404 includes an E2E-NNVC system 410. The E2E-NNVC system 410 includes an encoder portion 462 and a decoder portion 466. In some implementations, the E2E-NNVC system 410 may include one or more autoencoders. The encoder portion 462 is configured to receive input data 470 and process the input data 470 to generate output data 474 based at least in part on the input data 470.

[0093] In some specific implementations, the encoder portion 462 of the E2E-NNVC system 410 is configured to perform lossy compression on the input data 470 to generate output data 474, so that the output data 474 has fewer bits than the input data 470. The encoder portion 462 can be trained to compress the input data 470 (e.g., an image or video frame) without using motion compensation based on any previous representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 462 may compress the video frame using only video data from the video frame without using any data from the previously reconstructed frame. The video frame processed by the encoder portion 462 may be referred to herein as an intra-predicted frame (I frame). In some examples, conventional video coding techniques (e.g., according to HEVC, VVC, MPEG-4 or other video coding standards) may be used to generate I frames. In such examples, the processor 404 may include or be coupled to a video decoding device (e.g., an encoding device) configured to perform block-based intra-frame prediction (such as intra-frame prediction described above with respect to the HEVC standard). In such examples, the processor 404 may not include the E2E-NNVC system 410.

[0094] In some implementations, the encoder portion 462 of the E2E-NNVC system 410 may be trained to compress input data 470 (e.g., video frames) using motion compensation based on a previous representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 462 may use video data from a video frame and compress the video frame using data from a previously reconstructed frame. The video frames processed by the encoder portion 462 may be referred to herein as intra-predicted frames (P frames). Motion compensation may be used to determine data for the current frame by describing how pixels from a previously reconstructed frame move to a new position in the current frame and residual information.

[0095] As shown, the encoder portion 462 of the E2E-NNVC system 410 may include a neural network 463 and a quantizer 464. The neural network 463 may include one or more transformers, one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures that generate intermediate data 472. The intermediate data 472 is input to the quantizer 464.

[0096] The quantizer 464 is configured to perform quantization and, in some cases, entropy coding on the intermediate data 472 to produce output data 474. The output data 474 may include quantized (and, in some cases, entropy coded) data. The quantization operation performed by the quantizer 464 may result in the generation of a quantization code (or data representing a quantization code generated by the E2E-NNVC system 410) based on the intermediate data 472. The quantization code (or data representing a quantization code) may also be referred to as a latent code or latent data (latent) (denoted as z). The entropy model applied to the latent data may be referred to as a "prior" in this article. In some examples, the quantization and / or entropy coding operations may be performed using existing quantization and entropy coding operations performed when encoding and / or decoding video data according to an existing video coding standard. In some examples, the quantization and / or entropy coding operations may be performed by the E2E-NNVC system 410. In some illustrative examples, E2E-NNVC system 410 may be trained using supervised training, where during training, residual data is used as input and quantization codes and entropy codes are used as known outputs (labels).

[0097] The decoder portion 466 of the E2E-NNVC system 410 is configured to receive output data 474 (e.g., directly from the quantizer 464 and / or from the storage medium 414). The decoder portion 466 may process the output data 474 to generate a representation 476 of the input data 470 based at least in part on the output data 474. In some examples, the decoder portion 466 of the E2E-NNVC system 410 includes a neural network 468, which may include one or more transformers, one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more LSTM networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures.

[0098] The processor 404 is configured to transmit the output data 474 to at least one of the transmission medium 418 or the storage medium 414. For example, the output data 474 may be stored in the storage medium 414 to be later retrieved and decoded (or decompressed) by the decoder portion 466 to generate a representation 476 of the input data 470 as reconstructed data. The reconstructed data may be used for various purposes, such as for playback of video data that has been encoded / compressed to generate the output data 474. In some implementations, the output data 474 may be decoded at another decoder device (e.g., in the device 402, in the second device 490, or in another device) that matches the decoder portion 466 to generate a representation 476 of the input data 470 as reconstructed data. For example, the second device 490 may include a decoder that matches (or substantially matches) the decoder portion 466, and the output data 474 may be transmitted to the second device 490 via the transmission medium 418. The second device 490 may process the output data 474 to generate a representation 476 of the input data 470 as reconstructed data.

[0099] Components of system 400 may include, and / or may be implemented using, electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include, and / or may be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

[0100] Although system 400 is shown as including certain components, one of ordinary skill will appreciate that system 400 may include more than Figure 4More or fewer components than those shown. For example, system 400 may also include input devices and output devices (not shown), or may be part of a computing device that includes input devices and output devices. In some implementations, system 400 may also include or may be part of a computing device that includes: one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) that communicate with and / or are electrically connected to the one or more memory devices, one or more wireless interfaces for performing wireless communications (e.g., including one or more transceivers and baseband processors for each wireless interface), one or more wired interfaces for performing communications via one or more hardwired connections (e.g., serial interfaces such as universal serial bus (USB) inputs, and / or other wired interfaces), and / or Figure 4 Other components not shown.

[0101] In some implementations, the system 400 may be implemented locally by and / or included in a computing device. For example, the computing device may include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses or other AR device), a wearable device, a server (e.g., in a software as a service (SaaS) system or other server-based system), a television, and / or any other computing device having resource capabilities to perform the techniques described herein. In one example, the E2E-NNVC system 410 may be incorporated into a portable electronic device that includes: a memory 406 coupled to a processor 404 and configured to store instructions executable by the processor 404, and a wireless transceiver coupled to an antenna and coupled to the processor 404 and operable to send output data 474 to a remote device.

[0102] Figure 5 is an example that may be performed to determine between reference frames (e.g., from a reference frame To reference frame Schematic diagram of examples of different types of motion estimation for motion information of (or vice versa). Figure 5, term x represents a reference frame from which motion may be estimated, term f represents a motion estimate, and term y represents a regular frame which may be calculated as follows: y=f(x). One type of motion estimation is a block-based motion estimation technique 502. Block-based motion estimation may be performed on a block-by-block basis. For example, for each block in frame y, motion estimate f defines the position of the corresponding block in frame x. In an illustrative example, motion estimate f may include a motion vector indicating a displacement (e.g., horizontal and vertical displacement) of a block in frame y relative to a corresponding block in frame x. A block from frame x may be determined to correspond to a block in frame y by determining a similarity (e.g., similarity of pixel values) between the blocks.

[0103] Another type of motion estimation that may be performed is an optical flow motion estimation technique 504. Optical flow motion estimation may be performed on a pixel-by-pixel basis. For example, for each pixel in frame y, a motion estimate f defines the position of the corresponding pixel in frame x. The motion estimate f for each pixel may include motion information, such as a vector (e.g., a motion vector) indicating the movement of the pixel between frames. In some cases, an optical flow map (e.g., also referred to as a motion vector map) may be generated based on the calculation of the optical flow vectors between frames. The optical flow map may include an optical flow vector for each pixel in a frame, where each vector indicates the movement of the pixel between frames. In an illustrative example, the optical flow vector for a pixel may be a displacement vector (e.g., indicating horizontal and vertical displacements, such as x and y displacements) showing the movement of the pixel from a first frame to a second frame.

[0104] In some cases, an optical flow map may include vectors for fewer than all pixels in a frame. For example, a dense optical flow between frames may be calculated to generate an optical flow vector for each pixel in the frame, which may be included in the dense optical flow map. In some examples, each optical flow map may include a 2D vector field, where each vector is a displacement vector showing the movement of a point from a first frame to a second frame.

[0105] As described above, an optical flow vector or optical flow map may be calculated between frames of a frame sequence. The two frames may include two directly adjacent frames that are consecutively captured frames or two frames that are separated by a certain distance in the frame sequence (e.g., within two frames of each other, within three frames of each other, or any other suitable distance). In an illustrative example, a pixel I(x, y, t) in frame x may be moved by a certain distance or displacement (Δx, Δy) in frame y.

[0106] In some cases, a machine learning-based video decoding system (e.g., such as a neural video system or codec) may be used to encode and / or decode video data including frames of rendered content. For example, the rendered content may be generated by a cloud gaming system and / or server, an AR device, a VR device, etc. In some aspects, the rendered content may be generated based on one or more user inputs, as described in more depth below.

[0107] Figure 6 6 is a diagram illustrating an example of a neural video decoding system 600 that can be used to encode and / or decode rendered content associated with a cloud gaming system. A cloud gaming system may include one or more servers that run a video game locally (e.g., on a server) and stream output video frames to a user device that is remote from the cloud gaming server. By running the video game on a remote (e.g., cloud) server or other suitable hardware processing device, a user can play the video game on a smaller and more portable device (such as a smartphone) that might not otherwise have sufficient computing power to run the video game locally.

[0108] For example, Figure 6 As depicted in , a cloud gaming server may include a game engine for running a video game or otherwise performing underlying processing, computations, and the like associated with running a video game. The game engine may receive one or more user input commands for a currently running video game. For example, the user input commands may be obtained at a client device (such as a user's smartphone) or other user device connected to the cloud gaming server for a cloud gaming session. In some examples, the user input commands may include movement or directional input (e.g., using a joystick or hardware controller coupled to a client device, using an on-screen or software controller element provided by a client device, etc.), although it should be noted that various other user input commands may also be received and utilized by the cloud gaming server.

[0109] Based on the user input command, the game engine included in the cloud game server can calculate the effect or result of the user input command in the video game environment, and can update the current game state accordingly. In an illustrative example, the game engine can generate one or more updated frames of rendered video data based on the user input command and the change to the current game state. The one or more updated frames of rendered video data can be written or otherwise stored in one or more buffers included in the cloud game server.

[0110] In some cases, one or more buffers may include a deferred shading buffer and / or a geometry buffer (G-buffer). For example, a G-buffer may be used to store a screen space representation of geometry and material information (e.g., generated by an intermediate rendering pass in a deferred shading rendering pipeline associated with a game engine). In some examples, the G-buffer may include rendering information associated with one or more frames of cloud gaming video data. The rendering information may include, but is not limited to, motion information, depth information, normal information, albedo information, and the like. In some cases, the rendering information may be stored in the G-buffer as a graph. For example, the rendering information associated with a frame of cloud gaming video data may include, but is not limited to, a motion graph (e.g., an optical flow graph), a depth graph, a normal graph, an albedo graph, and the like.

[0111] A neural video encoder included in a cloud gaming server may be used to encode frames of cloud gaming video data. Figure 6 As illustrated, the neural video encoder may obtain frames of cloud gaming video data from a G-buffer (eg, generated by a gaming engine in response to user input commands) and encode each frame into a bitstream.

[0112] A bitstream associated with an encoded frame of cloud gaming video data may be sent to a client or user device, where the bitstream is decoded by a neural video decoder. In some examples, a reconstructed frame of cloud gaming video data (e.g., decoded from a bitstream by a client device neural video decoder) may be provided to a renderer included in the client device, where the renderer then renders and displays the reconstructed frame to the user. Based on the display of the reconstructed frame, the user perceives the flow of time within the video game and provides subsequent user input based on the updated game state—the subsequent user input is sent to the game engine of the cloud gaming server, and the above example process may be repeated.

[0113] Latency can be an important performance factor for cloud gaming systems. For example, increased latency can be perceived by users as lag. In some cases, a latency of 100 milliseconds (ms) or less may be required for fast-paced video games, and a latency of 150ms or less may be required for slower-paced games. In some cases, the latency threshold associated with decoding video frames of rendered content (such as cloud gaming content) may be lower than the latency threshold associated with streaming live video from a server to a remote client device. For example, based at least in part on the fact that live video streams are mainly unidirectional, live video streams can tolerate latency of up to 500ms. The maximum latency threshold associated with decoding video frames of rendered content can be the maximum end-to-end latency that a user can tolerate, and can be affected by the speed of the game engine, the speed of the server-side neural video encoder, the speed of the communication downlink (e.g., for sending encoded bitstreams) from the cloud gaming server to the client device, the speed of the neural video decoder, and / or the speed of the communication uplink (e.g., for sending user input commands) from the client device to the cloud gaming server.

[0114] In some examples, frames of rendered content (e.g., such as cloud gaming content) may be encoded and decoded based on or using a neural network-based predictive frame (P-frame) decoding system, where the current frame is encoded / decoded based on information determined for a previous frame (e.g., information determined in a previous time step). In some aspects, a neural network-based P-frame decoding system (such as Figure 7) to encode and decode frames of rendered content. As illustrated, the example neural P frame decoding system 700 includes a motion prediction system 702, a warping engine 704, and a residual prediction system 706. The motion prediction system 702 and the residual prediction system 706 may include any type of machine learning system (e.g., using one or more neural networks and / or other machine learning models, architectures, networks, etc.).

[0115] In some aspects, the motion prediction system 702 may include one or more autoencoders. In one illustrative example, the encoder network 705 and the decoder network 703 of the motion prediction system 702 may be implemented as autoencoders (e.g., also referred to as “motion autoencoders” or “motion AEs”). In some cases, the motion prediction system 702 may be used to implement optical flow-based motion estimation techniques, such as Figure 5 , and the motion autoencoder may be implemented as an optical flow autoencoder (e.g., also referred to as a "flow autoencoder" or "flow AE"). In some aspects, the residual prediction system 706 may include one or more autoencoders. In an illustrative example, the encoder network 707 and the decoder network 709 of the residual prediction system 706 may be implemented as an autoencoder (e.g., also referred to as a "residual autoencoder" or "residual AE"). Although Figure 7 The example P frame decoding system 700 is shown as including certain components, but a person of ordinary skill will understand that the example P frame decoding system 700 may include more Figure 7 There are more components than those shown.

[0116] In one illustrative example, for a given time t, the neural P frame decoding system 700 may receive an input frame X t and reference frame In some aspects, the reference frame may be a previously reconstructed frame generated before time t (e.g., at time t-1) (e.g., as determined by the hat operator Input frame X t and reference frame may be associated with or otherwise obtained from the same video data sequence (e.g., as consecutive frames, etc.). t can be the current frame at time t, and the reference frame It can be temporally or sequentially adjacent to the input frame X t In some cases, the reference frame may be received from a decoded picture buffer (DPB) of the example neural P frame decoding system 700. In some cases, the input frame X t Can be a P frame and a reference frame It can be an I frame, a P frame, or a B frame. For example, a reference frame It may have been previously reconstructed or generated by an I frame decoding system (e.g., which may be a part of a device including the P frame decoding system 700 or a device different from the device including the P frame decoding system 700), by the P frame decoding system 700 (or a P frame decoding system of a device different from the device including the P frame decoding system 700), or by a B frame decoding system (e.g., which may be a part of a device including the P frame decoding system 700 or a device different from the device including the P frame decoding system 700).

[0117] like Figure 7 As depicted, a motion prediction system 702 (eg, a motion autoencoder, an optical flow autoencoder, etc.) receives a reference frame and the current (ie, input) frame X t As input, the motion prediction system 702 may determine the reference frame The pixels and input frame X t The motion prediction system 702 may then encode and in some cases decode the determined motion as a vector for the input frame X. t Prediction of movement

[0118] For example, the encoder network 705 of the motion prediction system 702 may be used to determine the current frame X t and reference frame In some aspects, the encoder network 705 may encode the determined motion information into a latent representation (eg, represented as latent data z m For example, in some cases, the encoder network 705 may map the determined motion information to a latent code, which may be used as the latent data z m The encoder network 705 may additionally or alternatively be coupled to z m The associated latent code performs entropy decoding to convert the latent data z m In some examples, the encoder network 705 can quantize the latent data z m (eg, before performing entropy coding on the latent code). The quantized latent data may include the latent data z m In some cases, the latent data z m Neural network data (e.g., activation maps or feature maps of neural network nodes) representing one or more quantized codes may be included.

[0119] In some aspects, the encoder network 705 may store latent data z m , the latent data z mThe latent data z may be transmitted to a decoder network 703 included in the motion prediction system 702, and / or the latent data z may be transmitted to a decoder network 703 included in the motion prediction system 702. m Transmit to the submersible data m Decoding another device or system. m When the decoder network 703 can decode the latent data z m Decoding (e.g., inverse entropy decoding, inverse quantization, and / or reconstruction) to generate a reference frame The pixels and input frame X t The predicted motion between pixels For example, the decoder network 703 may decode the latent data z m Decode to generate optical flow map This involves the reference frame Some (or all) of the pixels included in are mapped to the input frame X t The encoder network 705 and the decoder network 703 may be trained and optimized using training data (e.g., training images or frames) and one or more loss functions, as will be described in more depth below.

[0120] In one illustrative example, the encoder network 705 and the decoder network 703 may be included in a motion prediction autoencoder, such as an optical flow autoencoder. The optical flow autoencoder may include a method for quantizing the latent data z m One or more components that convert the quantized latent data into a bitstream (eg, generated as an output of the encoder network 705 of the optical flow autoencoder). The bitstream generated from the quantized latent data may be provided as input to the decoder 703 of the optical flow autoencoder.

[0121] In some examples, the predicted motion The information or data may include optical flow information or data (e.g., an optical flow map including one or more motion vectors), dynamic convolution data (e.g., a matrix or kernel for data convolution), or block-based motion data (e.g., a motion vector for each block). In one illustrative example, the predicted motion An optical flow map may be included. In some cases, as previously described, an optical flow map Can include input frame X t The motion vector may represent a motion vector for each pixel of the current frame X (eg, a first motion vector for a first pixel, a second motion vector for a second pixel, etc.). The motion vector may represent a motion vector for each pixel of the current frame X. t The pixels in the reference frame The motion information determined by the corresponding pixels in (e.g., the motion information determined by the encoder network 705).

[0122] The warping engine 704 of the neural P-frame decoding system 700 may obtain an optical flow map generated as an output of the motion prediction system 702 (e.g., generated as an output of the decoder network 703 / optical flow autoencoder described above) For example, the warping engine 704 may retrieve the optical flow map from the storage device Or the optical flow map may be directly received from the motion prediction system 702 The regularization engine 704 may use an optical flow map To regularize (e.g., by performing motion compensation) the reference frame pixels, resulting in a regular frame In some aspects, the regular frame Also called motion compensated frame (For example, by including The corresponding motion vector in the reference frame is used to regularize For example, the regularization engine 704 may generate pixels based on the pixels included in the optical flow map. The motion vectors (and / or other motion information) in the reference frame The pixels are moved to new positions to generate motion compensated frames

[0123] As mentioned above, in order to generate a regular frame The neural P frame decoding system 700 can perform motion compensation by predicting the input frame X t and reference frame The optical flow between And then by using the optical flow map Regular reference frame To generate motion compensated frames However, in some cases, based on the optical flow map The generated frame prediction (e.g., motion compensated frame ) may not be accurate enough to convert the input frame X t Reconstructed frame For example, there may be an input frame X t One or more occluded areas in the depicted scene, over-lighting, under-lighting, and / or causing motion compensated frames Not accurate enough to be used as a reconstructed input frame other effects.

[0124] The residual prediction system 706 may be used to correct or otherwise improve the motion compensation frame For example, the residual prediction system 706 may generate one or more residuals that the neural P frame decoding system 700 may then associate with the motion compensated frame Combined to generate a more accurate reconstructed input frame (e.g., more accurately represent the underlying input frame X t The reconstructed input frame In one illustrative example, Figure 7 As depicted, the neural P frame decoding system 700 may be implemented by taking an input frame X t Subtract the predicted (eg, motion compensated) frame (eg, using subtraction operation 708) to determine the residual r t For example, in motion compensated prediction frames After being determined by the regularization engine 704, the P frame decoding system 700 may determine the motion compensated prediction frame and input frame X t The residual r is determined by calculating the difference between t .

[0125] In some aspects, the encoder network 707 of the residual prediction system 706 may convert the residual r t Encoded into latent data z r , where the latent data z r Residual r t For example, the encoder network 707 can convert the residual r t Mapped to the latent data z r In some cases, the encoder network 707 may convert the latent data z into a latent code by performing entropy decoding on the latent code. r In some examples, the encoder network 707 may additionally or alternatively quantize the latent data z r (e.g., before performing entropy coding). Quantize the latent data z r The residual r t In some cases, the latent data z r The encoder network 707 may include neural network data (e.g., activation maps or feature maps of neural network nodes) representing one or more quantized codes. r , sends or otherwise provides latent data z to the decoder network 709 of the residual prediction system 706 r , and / or the latent data z r Transmit to the submersible data r Decoding another device or system. r When the decoder network 709 can decode the latent data z r Decoding (eg, inverse entropy coding, inverse quantization, and / or reconstruction) is performed to generate a prediction (eg, decoded) residual In some examples, encoder network 707 and decoder network 709 may be trained and optimized using training data (eg, training images or frames) and one or more loss functions, as described below.

[0126] In one illustrative example, the encoder network 707 and the decoder network 709 may be included in a residual prediction autoencoder. The residual autoencoder may include a method for quantizing the latent data z r (For example, where the latent data z r The quantized latent data z is generated as an output of the encoder network 707 of the residual autoencoder and converts the quantized latent data into one or more components of a bitstream. r The generated bitstream may be provided as input to the decoder 709 of the residual autoencoder.

[0127] Prediction residuals (eg, generated by the decoder network 709 and / or a residual autoencoder for implementing the residual prediction system 706) may be associated with the motion compensated prediction frame (e.g., by the warping engine 704 using an optical flow map generated by the decoder network 703 and / or an optical flow autoencoder used to implement the motion prediction system 702) to generate) to generate an input frame X representing the input frame at time t t The reconstructed input frame

[0128] For example, the neural P frame decoding system 700 may convert the prediction residual and motion compensated prediction frames Add (e.g., using addition operation 710) or otherwise combine them to generate a reconstructed input frame In some cases, the decoder network 709 of the residual prediction system 706 may use the prediction residual Added to motion compensated frame prediction In some examples, the input frame has been reconstructed It may also be referred to as a decoded frame and / or a reconstructed current frame. May be output for storage (e.g., in a decoded picture buffer (DPB) or other storage device), transmission, display, for further processing (e.g., as a reference frame in further inter-frame prediction, for post-processing, etc.) and / or for any other purpose.

[0129] In one illustrative example, the P frame decoding system 700 may send a representation of an optical flow map or other motion information (eg, latent data z) in one or more bitstreams to another device for decoding. m ) and the latent data representing the residual information (e.g., the latent data z r) of the latent data. In some cases, the other device may include a m and z r In one illustrative example, another device may include a video decoder that implements one or more portions of the P-frame decoding system 700, the motion prediction system 702 (e.g., an optical flow autoencoder), and / or the residual prediction system 706 (e.g., a residual autoencoder), as described above.

[0130] Other devices or video decoders may use the latent data z generated as an output of the decoder network 703 included in the motion prediction system 702 (eg, generated as an output of an optical flow autoencoder including the decoder network 703). m Optical flow map Other devices or video decoders may use the latent data z generated as an output of the decoder network 709 included in the residual prediction system 706 (e.g., generated as an output of a residual autoencoder including the decoder network 709) r To the residual Other devices or video decoders can then use the optical flow map and residual to generate a decoded (eg, reconstructed) input frame

[0131] For example, when the video decoder implements an architecture that is the same as or similar to the architecture of the P-frame decoding system 700 described above, the video decoder may include a regularization engine (e.g., the same as or similar to the regularization engine 704) that receives the decoded optical flow map and reference frame As input, the video decoder regularization engine can be based on the reference frame motion vectors and / or other motion information determined by some (or all) pixels of To regularize the reference frame Video decoder regularization engine outputs motion compensated frame predictions (e.g., as described above with respect to the output of the regularization engine 704). The video decoder may then convert the decoded residual and motion compensated frame prediction Add or otherwise combine them to generate a decoded (e.g., reconstructed) input frame (eg, as described above with respect to the output of addition operation 710).

[0132] In some examples, the motion prediction system 702 and / or the residual prediction system 706 may be trained and / or optimized using training data and one or more loss functions. In some cases, the motion prediction system 702 and / or the residual prediction system 706 may be trained in an end-to-end manner (e.g., where all neural network components are trained during the same training process). As previously mentioned, in some examples, the motion prediction system 702 may be implemented as an optical flow autoencoder including an encoder network 705 and a decoder network 703; and the residual prediction system 706 may be implemented as a residual autoencoder including an encoder network 707 and a decoder network 709. In some aspects, the training data may include a plurality of training images and / or training frames. In some cases, a loss function (e.g., Loss) may be used to perform training based on the motion prediction system 702 and / or the residual prediction system 706 processing the training images or frames.

[0133] In one example, the loss function (eg, Loss) may be given as Loss=D+βR, where D is the weight of a given frame (eg, such as input frame X t ) and its corresponding reconstructed frame (e.g., ). For example, the distortion D can be determined as D(X t , ). β is a hyperparameter that can be used to control the bit rate (e.g., bits per pixel), and R is a parameter used to convert the residual (e.g., residual r t ) into a compressed bit stream (e.g., latent data z r ) bits. In some examples, the distortion D may be calculated based on one or more of a peak signal-to-noise ratio (PSNR), a structural similarity index metric (SSIM), a multi-scale SSIM (MS-SSIM), etc. In some aspects, using one or more training data sets and one or more loss functions, parameters (e.g., weights, biases, etc.) of the motion prediction system 702 and / or the residual prediction system 706 may be tuned until the example neural P frame decoding system 700 achieves a desired video decoding result.

[0134] In some examples, one or more of the encoder networks, decoder networks, autoencoder networks, and / or codecs described herein may be implemented based on or using a scale space flow (SSF) architecture. In some aspects, the SSF architecture may be enhanced by one or more motion compensation steps. For example, enhancing the SSF architecture by at least one motion compensation step may better use available rendered content (e.g., one or more depth maps, one or more optical flow maps, etc.). In some examples, the SSF architecture may be enhanced by one or more motion compensation steps to correct (e.g., compensate) camera motion. Correcting or compensating for camera motion may improve rate distortion and decoding efficiency of the enhanced SSF architecture and / or one or more codecs implemented using the systems and techniques described herein. In some aspects, region of interest decoding (ROI decoding) may be implemented based on one or more ROI-aware losses and / or adjustments to one or more encoder networks on a ROI mask.

[0135] In some examples, the systems and techniques described herein can perform or apply camera motion compensation to input including rendered content. In some aspects, the systems and techniques described herein can be used with a TartanAir dataset (e.g., an SSF framework analyzed on the same TartanAir dataset). In some cases, the use of ROI coding may be associated with the greater coding flexibility of the systems and techniques described herein. For example, one or more depth maps may be used as a proxy for one or more ROI maps. In some aspects, the use of ROI coding may be associated with rate savings compared to non-ROI implementations (e.g., SSF) while maintaining the perceived quality of the coded input and / or output. In some examples, one or more GameCodec parameters may be suitable for a specific gaming environment and may be associated with additional BD rate gains. In some cases, the BD rate gain associated with adapting the GameCodec parameters to the player environment may be based at least in part on the scene type associated with a given gaming environment. In some examples, systems and techniques may be used to implement instance adaptive coding.

[0136] In some examples, a rate-distortion (RD) loss may be used. to train a specific SSF-based implementation, where the parameter β is used to weigh the rate of distortion (e.g., ). In some aspects, systems and techniques can perform training based on a modified ROI-aware objective, such as using a loss function

[0137] D ROI is the ground truth frame sequence x and its reconstruction The mean squared error distortion metric between , weighted by a sequence of importance maps m whose values ​​are in (0,1) (e.g., where N represents the total number of elements in the sequence):

[0138] In some examples, There can be a generalized loss definition that supports the case where the importance mask is continuous rather than binary (e.g., based on using a depth map as a surrogate for importance, together with a model of the human visual system where sharpness decreases exponentially from focus and results in a continuous importance map with peaks near the estimated focus and close objects).

[0139] In some aspects, the temporally coherent Perlin noise mask m of the same dimension as the frame sequence x may be used. p The noise mask m is sampled to perform training. p can be rescaled to (0,1) and used to define the importance mask as m=1-α·m p , where α is a scalar in (0,1) that controls the focus on the ROI. In some cases, if α is 0, the importance map m can be all ones, and Degradation to non-ROI situation

[0140] In some examples, the evaluation phase of training can use an importance map based on the depth mask and a hyperparameter α, which can be used to control or adjust the amount of difference that should exist between foreground and background regions in terms of image quality. For example, to represent a sequence of depth maps z, an importance map m can be used, where where z min and z max are the minimum and maximum values ​​of the depth of the associated game engine, respectively, and γ is a hyperparameter that controls the power-law transformation of the depth map. In some cases, γ can be adjusted at test time to control the drop-off between foreground and background (e.g., slower if γ<1 and steeper if γ>1).

[0141] In some aspects, the learned codec can be more easily fine-tuned to the application domain (e.g., relative to a standard codec). In the example of cloud gaming, the systems and techniques can use one or more models, neural networks, machine learning networks, autoencoders, etc., which are fine-tuned on different game scenes in the dataset. When evaluating unseen sequences from the same game, an improvement in RD performance can be observed. In some aspects, the updated model weights can be sent to update the receiver side codec before starting to send (e.g., representing at most tens of megabytes).

[0142] As previously mentioned, in some examples, frames of rendered video content (e.g., such as cloud gaming content) may be encoded and decoded using a neural video coding system the same as or similar to the example neural P-frame coding system 700 described above. For example, a current frame of cloud gaming content (or other rendered video content) may be decoded based on determining an optical flow map or other motion information for the current frame in conjunction with a residual map.

[0143] In some aspects, the motion between consecutive frames of video content can be based on object motion and camera motion. Figure 7 In the example of , a single motion prediction system 702 is used to predict or otherwise determine both object motion and camera motion. In some examples, using a single motion prediction system to perform camera-based motion compensation and object-based motion compensation may reduce the performance and / or accuracy of the example neural P-frame decoding system 700, such as for decoding video content that includes significant camera motion over a short period of time. For example, frames of cloud gaming content may include significant camera motion over a short period of time (e.g., based on rapid user input, viewpoint manipulation, etc. while playing the game). In some aspects, the systems and techniques described herein may be used to encode and decode frames of video content (including frames of rendered video content) based on decomposing camera motion compensation and object motion compensation, as will be described in more depth below.

[0144] Figure 8 8 is a diagram illustrating an example of a machine learning based video decoding system 800 including a depth map prediction system 822 and an object motion prediction system 802. In one illustrative example, the depth map prediction system 822 can be used to compensate for camera motion using rendering information associated with a currently encoded or decoded frame of rendered video data (e.g., a frame of rendered cloud gaming content), as described in more depth below. In some aspects, the object motion prediction system 802 can be used to compensate for object motion using optical flow based regularization (e.g., as described above with respect to Figure 7 In some cases, the object motion prediction system 802 may be used in conjunction with the object motion prediction system 702 illustrated in FIG. Figure 7 The object motion prediction system 802 may be the same as or similar to the object motion prediction system 702 illustrated in FIG. 802 . For example, the object motion prediction system 802 may be implemented as an optical flow autoencoder including an encoder network 805 and a decoder network 803 .

[0145] As previously described, video content (e.g., computer-generated or animated visual content, AR content, VR content, gaming content, etc.) including rendered frames of video data may include relatively greater motion between consecutive frames compared to natural video content (e.g., video content captured by a camera). In some cases, rendered frames of video data may include relatively greater camera motion and relatively greater object motion compared to natural content video data.

[0146] In the example of a rendered frame of game content (e.g., generated by a Figure 6 In some embodiments, the associated camera motion between consecutive frames may be relatively high based on the frequent and / or large movements made by a user playing a video game, where the user movements or inputs are associated with corresponding changes in the camera point of view (POV) and / or field of view (FOV). Additionally, in the example of rendered frames of game content, the associated object motion between consecutive frames may be relatively high based on factors such as rendered frames of game content corresponding to large and / or complex scenes (e.g., which may include large-sized objects, highly detailed or granular objects, various visual and / or granular effects such as light, rain, smoke, etc.).

[0147] In one illustrative example, the systems and techniques described herein may perform video decoding of frames of video content (e.g., frames of rendered and / or game content) based on using separate motion prediction systems to perform camera motion compensation and to perform object motion compensation. For example, the depth map prediction system 822 may be used to perform camera motion compensation based on rendering information associated with one or more frames of video content, as will be explained in more depth below.

[0148] In one illustrative example, the rendering information associated with one or more frames of the currently encoded / decoded video content may include the current camera pose p t (e.g., the camera pose associated with the current frame t), the reference camera pose p t-1 (e.g., the camera pose associated with the previous frame t-1) and the depth map D associated with the current frame t t . The camera pose information may include position and orientation information of the camera relative to a reference coordinate system of a scene depicted in the video content (e.g., a scene captured by the camera). A depth map may be an image that includes information associated with a distance of a surface of a scene object from a given viewpoint (e.g., a viewpoint of a camera that captured the scene). In some examples, the depth map may have the same pixel size as a corresponding frame of video data for which the depth map was generated, wherein some (or all) of the pixels included in the depth map are associated with depth values. The depth value associated with a given pixel may indicate a distance from the camera viewpoint to a surface of a scene object depicted by the given pixel.

[0149] In examples where frames of video content that are decoded (e.g., encoded and / or decoded) using the example neural video decoding system 800 are frames of rendered video data, camera pose information and depth information may be generated in association with the rendered video frames (e.g., generated when the video frames were initially generated, prior to encoding or decoding). For example, based on the camera pose and depth information that the rendering engines used to initially generate the rendered frames, the camera pose and depth information may be obtained from one or more rendering engines associated with generating the rendered frames. In some aspects, the camera pose and depth information associated with the frames of rendered game content may be obtained from one or more buffers associated with a rendering or game engine used to generate the game content. For example, the camera pose and depth information associated with the frames of rendered game content may be obtained from a deferred rendering buffer and / or a geometry buffer (G-buffer) associated with the game engine. Figure 6 In some examples, the camera pose and depth information may be stored in a deferred rendering buffer or a G-buffer among various other types of rendering information (e.g., motion information, optical flow information such as an optical flow map, a normal map, an albedo map, etc.).

[0150] In some examples, the systems and techniques described herein may be used to perform video decoding based on separate camera motion compensation and object motion compensation for non-rendered video content (eg, natural video content). t and p t-1 The depth information may be obtained from a camera used to capture non-rendered or natural video content, may be determined using one or more post-processing algorithms or machine learning networks, etc. In some cases, the depth information may be obtained from a camera that includes one or more depth sensors and captures depth information and video content simultaneously (e.g., such that the video content is captured in conjunction with a corresponding depth map for one or more frames of the captured video content). In some aspects, the depth information may be obtained from a stereo camera that includes two or more cameras for determining a depth map.

[0151] In one illustrative example, the depth map prediction system 822 and the 3D warping engine 824 may be used to generate an initial frame prediction. It uses the input camera pose p t , reference camera pose p t-1 and the depth map D t In some aspects, the initial frame prediction It can also be called a camera motion compensation frame. As shown in the example, the 3D warping engine 824 can receive the camera pose p of the current frame t based on the camera pose p of the current frame t. t , the reference camera pose p of the previous frame t-1 t-1 , previously decoded (e.g., reconstructed) frames and the predicted (e.g., reconstructed) depth map for the current frame t Takes as input to generate initial frame predictions

[0152] Camera pose p t and p t-1 Together with the depth map D t Can be achieved by Figure 8 In one illustrative example, a neural video encoder of an example architecture is associated with a rendering engine (e.g., a cloud gaming engine) to generate. Figure 8 The neural video encoder of the example architecture of may be included in the same server or other computing device as the rendering engine used to generate the frames of rendered video data being encoded (e.g., which may also be the computer used to generate the camera pose p t and p t-1 Together with the depth map D t The same rendering engine as .

[0153] accomplish Figure 8 Some or all of the neural video decoders in the example architecture of may receive a camera pose p from a neural video encoder and / or a server (e.g., a rendering engine) associated with the neural video encoder. t and p t-1 In some examples, based on receiving a reference camera pose p t-1 As the current camera pose of the previous frame, the neural video decoder may only receive the current camera pose p associated with the currently coded frame of rendered video data. t (E.g., the neural video decoder may store the camera pose received from the neural video encoder for use in decoding future or subsequent frames of rendered video content).

[0154] In some aspects, systems and techniques may use the depth map prediction system 822 to perform depth map compression instead of sending the depth map D from the neural video encoder to the neural video decoder. t For example, the depth map D t The encoder network 825 may be provided as an input to the depth map prediction system 822, which may be implemented as an autoencoder in some cases. The encoder network 825 may generate a depth map D t The latent representation z c As output. In some aspects, the latent representation z c can be converted into a bit stream and sent to the implementation Figure 8 An example architecture for a neural video decoder.

[0155] For example, the depth map D t The latent representation z cmay be provided as an input to a decoder network 823 included in the depth map prediction system 822. In some aspects, both the neural video encoder and the neural video decoder may include the decoder network 823. c As input, the decoder network 823 can generate a predicted depth map for the current decoded frame t As output. In some examples, the predicted depth map It may also be called a "reconstructed depth map".

[0156] The 3D regularization engine 824 can be based on the camera pose p t and p t-1 And the reconstructed depth map Regularize previously reconstructed frames To generate the initial frame prediction (e.g., camera motion compensated frames). For example, the camera pose p t and p t-1 can be used to estimate the 3D camera transformation T(p) associated with the motion or movement between the pose of the camera at time t and the pose at time t-1 t ,p t-1 ). Based on the reconstructed depth map The inverse projection is performed using the camera projection matrix P (also referred to as “back projection”). The output of the inverse projection may be a set of 3D scene coordinates about the camera at time t.

[0157] The estimated 3D camera transformation T(p t ,p t-1 ) and the output of the inverse projection (e.g., a set of 3D scene coordinates about the camera at time t) may be provided as input to a 3D point transform, which generates as output a set of 3D scene coordinates about the camera at time t-1. A camera projection may be performed to transform the 3D scene coordinates about the camera at time t-1 into a set of 2D scene coordinates about the camera at time t-1. The camera projection into 2D scene coordinates may utilize the same camera projection matrix P used to perform the previous inverse projection.

[0158] Based on the camera projection of the 2D scene coordinates about the camera at time t-1, I can be determined t (e.g., a frame or scene captured by a camera at time t) and I t-1 (e.g., a frame or scene captured by a camera at time t-1). t with I t-1 The estimated 2D correspondence between 824 and 834 may be provided as input to a bilinear warping engine. In some aspects, the bilinear warping engine may be the same as or included in the 2D warping engine 824. t with I t-1In addition to the estimated 2D correspondence between the two frames, the bilinear warping engine may additionally receive the previously reconstructed frames As input (for example, Figure 8 824). Based on the t and I t-1 The estimated 2D correspondence between the previously reconstructed frame In some aspects, the camera compensation frame generated as output based on the above bilinear warping can be compared with the initial frame prediction generated by the 3D warping engine 824. same.

[0159] Camera motion compensated frames The additional warping engine 804 may then be provided. As illustrated, the additional warping engine 804 may also receive a predicted optical flow map associated with the object motion for the current frame t. (Example of optical flow information) as input. In some aspects, systems and techniques can be based on using the warping engine 804 to predict the object motion optical flow map Regularize camera motion compensation frames to perform cumulative warping (e.g., based on camera motion compensated frames that have been generated using previous warping operations performed by the 3D warping engine 824). of cumulative regularization).

[0160] In some aspects, the object motion prediction system 802 may be similar to the above with respect to Figure 7 The object motion prediction system 702 described above is the same or similar to the object motion prediction system 702, and the object motion optical flow map Can be compared with the same above Figure 7 Optical flow diagram of description The same or similar method is used to generate the object motion optical flow map. The camera motion compensation frame and the current decoded frame X t The encoder network 805 provided as input to the object motion prediction system 802 may be implemented as an optical flow autoencoder. The encoder network 805 may determine the optical flow between two input frames and generate a latent representation z of the determined optical flow map. p As described above with respect to the depth map prediction system 822, in some aspects, the object motion prediction system 802 may output the latent representation z p is converted into a bitstream sent from the neural video encoder to the neural video decoder. The latent data z p The bitstream representation of may be provided as input to a decoder network 803, which generates a representation of a camera motion compensated frame and the input (eg, currently decoded) frame Xt The predicted (e.g., reconstructed) optical flow map of the optical flow between as output.

[0161] In some aspects, the depth map D t The latent representation z c may be provided as an additional input to the object motion prediction system 802. For example, the object motion prediction system 802 may provide the depth map D t The latent representation z c and the latent representation z of the optical flow map generated by the encoder network 805 p Cascade. Based on the received depth map D t The latent representation z c (e.g., associated with camera motion) and the latent representation z of the optical flow map p (e.g., associated with object motion) as input, in some aspects the decoder network 803 can generate a reconstructed optical flow map To include object motion information but not camera motion information. For example, the decoder network 803 may be based on analyzing the depth map D t The latent representation z c (e.g., associated with camera motion) and from the reconstructed optical flow map Remove camera motion information.

[0162] The warping engine 804 can be based on the reconstructed object motion optical flow map Regularize camera motion compensation frames To generate the predicted motion compensated frame As output (for example, in the same way as previously relative to Figure 7 In one illustrative example, the motion compensated frame is predicted in the same or similar manner as described for the frame prediction output by the illustrated regularization engine 704. Motion compensation for camera motion is included (eg, based on a first warping performed by the 3D warping engine 824 ) and motion compensation for object motion is included (eg, based on a second warping performed by the warping engine 804 ).

[0163] Predicted motion compensated frame The predicted (eg, reconstructed) residual to improve, in some ways, it can be compared with the residual (The above is relative to Figure 7 In one illustrative example, the residual prediction system 806 (which may be the same as the residual prediction system 806) is performed in the same or similar manner as described in the description of the residual. Figure 7 The residual prediction system 706 illustrated in FIG. 8 (same as or similar to the residual prediction system 706) may receive a latent representation z generated by the object motion prediction system 802. pFor example, the decoder network 809 included in the residual prediction system 806 may convert the latent representation z p and the latent residual z r cascade, and the reconstructed residual can be generated based on the latent representations of the two cascades as output.

[0164] The addition operation 810 may be used to convert the predicted motion compensated frame and the reconstructed residual Add or otherwise combine them to produce the reconstructed current frame as output.

[0165] In one illustrative example, systems and techniques may use a camera based on the current camera pose p t , reference camera pose p t-1 and the depth map D t The generated rendered optical flow information (e.g., rendered optical flow map) is used to compensate for camera motion. For example, instead of using the camera pose information p t and p t-1 The depth map D is provided as input to the regularization engine 824 and t Instead of providing the encoder network 825 with the camera motion prediction system 822 as input, the systems and techniques may use these same three inputs (e.g., p t 、p t-1 and D t ) to generate a rendered optical flow map and provide the rendered optical flow map as input to the encoder network 825 of the camera motion prediction system 822.

[0166] Fig. 9 is a diagram illustrating an example of a machine learning-based video decoding system 900, the video decoding system utilizing a video decoding system based on (p t 、p t-1 and D t ) as input for generating camera motion compensated frames using the camera motion prediction system 922 and the warping engine 924 In one illustrative example, Fig. 9 The machine learning based video decoding system 900 described in Figure 8 The machine learning-based video decoding system 800 described in FIG. 8 is the same as or similar to the machine learning-based video decoding system 800 described in FIG. 8 . For example, the camera motion prediction system 922 and the regularization engine 924 can be used with Figure 8 The camera motion prediction system 822 and the regularization engine 824 illustrated in FIG. 8 are the same or similar, and the object motion prediction system 902 and the regularization engine 904 may be the same as or similar to the camera motion prediction system 822 and the regularization engine 824 illustrated in FIG. Figure 8 The object motion prediction system 802 and the regularization engine 804 illustrated in FIG. 8 are the same as or similar to each other.

[0167] exist Fig. 9 In the example, the rendered optical flow can be achieved by Fig. 9 Example architecture 900 of a neural video encoder based on p t 、p t-1 and D t The rendered optical flow map may be generated and / or may be generated by a rendering engine, a server, a cloud gaming server, or other computing device associated with the neural video encoder. In some examples, the neural video encoder and the computing device for generating the rendered optical flow may be included in the same device. In some aspects, the rendered optical flow map may include a current camera pose p t and the previous camera pose p t-1 And the current depth map D t The rendered optical flow may be provided as input to the encoder network 925 of the camera motion prediction system 922, which generates a latent representation z of the rendered optical flow. c In some aspects, based at least in part on the camera motion information included in the rendered optical flow map, the latent representation z c The camera motion information between frames t and t-1 may be included or otherwise associated with the camera motion information.

[0168] In some aspects, a neural video decoder implementing some or all of example architecture 900 may receive only latent data z from a corresponding neural video encoder (e.g., a neural video encoder also implementing some or all of example architecture 900). c To generate camera motion compensated frames In some cases, the latent data z c may be converted into a bitstream and sent to a decoder network 923 and / or a camera motion prediction system 922 included in a neural video decoder. The decoder network 923 may receive the latent data z c As input and generates a predicted (e.g., reconstructed) optical flow map for the current frame t As output. In some aspects, the optical flow map has been reconstructed The optical flow map may be associated with the motion information determined between the previous frame t-1 and the current frame t (e.g., the same or similar motion information as included in the rendered optical flow map determined at the neural video encoder side). As input, the warping engine 924 can generate a camera motion compensated frame Instead of receiving the camera pose information p from the neural video encoder (or associated rendering engine / server) t and p t-1 ,like Figure 8 .

[0169] In some aspects, based on utilizing the rendered optical flow as an input to the camera motion prediction system 922, fewer bits may be sent between the neural video encoder side and the neural video decoder side for each corresponding frame of video data (e.g., rendered video data and / or game content video data) because less bits are required than sending the depth map D t and the camera pose p t and p t-1 A potential representation of one or more of Figure 8 Compared with the case of only sending the latent representation z of the rendered optical flow map c Fewer bits are available.

[0170] In some examples, the above are respectively relative to Figure 8 and Fig. 9 The illustrated and described machine learning based video decoding systems 800 and 900 can be based on compensating frames for camera motion. and object motion compensated frames Perform cumulative warping to generate reconstructed frames (e.g., based on object motion compensation frames It is based on receiving the previously regularized camera motion compensation frame In some aspects, systems and techniques can generate reconstructed frames based on performing camera motion compensation and object motion compensation separately. No cumulative regularization between the camera motion compensation results and the object motion compensation results is performed.

[0171] For example, Fig.10 is an example of a method for generating a reconstructed frame based on performing optical flow compression, iterative flow estimation, and residual flow estimation (without performing cumulative regularization). 1000 is a diagram of an example of a machine learning based video decoding system 1000. In one illustrative example, Fig.10 The machine learning based video decoding system 1000 described in Fig. 9 The machine learning based video decoding system 900 and / or Figure 8 The camera motion prediction system 1022 and the regularization engine 1024 may be the same as or similar to one or more of the machine learning-based video decoding systems 800 illustrated in FIG. Fig. 9 and Figure 8 The camera motion prediction system 922 and / or 824 and one or more of the regularization engines 924 and / or 824 illustrated in FIG. 1 are the same or similar; the object motion prediction system 1002 and the regularization engine 1004 may be respectively the same as or similar to the camera motion prediction system 922 and / or 824 and / or the regularization engine 924 and / or 824 illustrated in FIG. Fig. 9 and Figure 8One or more of the object motion prediction systems 902 and / or 802 and the regularization engines 904 and / or 804 illustrated in are the same or similar; etc.

[0172] exist Fig.10 In the example of t and p t-1 And the depth map D t To generate the rendered optical flow, as above relative to Fig. 9 As illustrated, the camera motion prediction system 1022 can be used with respect to Fig. 9 The illustrated camera motion prediction system 922 receives the rendered optical flow map as input in the same or similar manner as described above. The camera motion prediction system 1022 may additionally generate a reconstructed camera motion optical flow map. As output, the reconstructed camera motion optical flow map is compared with the Fig. 9 The warping engine 1024 may receive a previously reconstructed (eg, reference) frame and the reconstructed camera motion optical flow map As input, it can be compared with the above Fig. 9 The same or similar to the unification engine described in the unification engine 924 illustrated in FIG.

[0173] like Fig.10 As depicted in FIG. 1 , the output generated by the warping engine 1024 (eg, camera motion compensated frame ) may be provided as input only to the object motion prediction system 1002 (e.g., rather than being provided as input to both the object motion prediction system 1002 and the warping engine 1004, as with respect to Fig. 9 and object motion prediction system 902 and warping engine 904). Fig.10 As shown in the example, the reconstructed camera motion optical flow map generated by the camera motion prediction system 1022 May additionally be provided as input to the object motion prediction system 1002 .

[0174] In one illustrative example, the object motion prediction system 1002 may receive a current decoding frame X t and camera motion compensated frames As input (for example, respectively with respect to the above Figure 8 and Fig. 9 The object motion prediction system 802 and the object motion prediction system 902 illustrated in the example are the same or similar to the object motion prediction system described in the example 1002). In addition, the object motion prediction system 1002 may receive the reconstructed camera motion optical flow map (e.g., from camera motion prediction system 1022) and a previously reconstructed reference frame as input.

[0175] In some respects, Fig.10 The example architecture 1000 illustrated in FIG. 1000 may be used to directly perform a second regularization (e.g., regularization performed by a regularization engine 1004) to obtain a frame of camera motion compensation after the regularization. Add object motion compensation on top of camera motion compensation / camera motion optical flow correction For example, the warping engine 1004 does not directly receive the camera motion compensated frames. And does not use object motion based optical flow The regularization is applied to the camera motion compensation frame (For example, Figure 8 and Fig. 9 In contrast, the camera motion compensates the frame Provided as input to the object motion prediction system 1002, it generates an object motion optical map including both object motion optical flow information and camera motion optical flow correction information. For example, the object motion prediction system 1002 may receive a previously reconstructed reference frame based on the received and camera motion compensated frames The additional input is used to generate a corrected camera motion compensation output (e.g., an object motion optical map associated with the camera motion optical flow correction information). part).

[0176] The output of the object motion prediction system 1002 (e.g., an object motion optical map including object motion optical flow information and camera motion optical flow correction information) ) can be provided as input to the regularization engine 1004. The regularization engine 1004 can be based on the object motion optical map Regularize previously reconstructed frames To generate the prediction for the current frame As output, instead of regularizing the camera motion compensated frame based on the object motion optical map which does not include the camera motion optical flow correction (As compared to Figure 8 and Fig. 9 In one illustrative example, the systems and techniques may utilize Fig.10 The example architecture 1000 illustrated in FIG. 1 is used to generate an initial camera motion compensated frame. Based on the object motion optical flow map including object motion optical flow information and camera motion optical flow correction information To correct the initial camera motion compensation frame And by using an optical map of object motion based on camera motion correction Regularize previously reconstructed frames To generate the prediction for the current frame

[0177] Fig.11 is a flow chart illustrating an example of a process 1100 for processing video data. At block 1102, the process 1100 includes obtaining an encoded video data frame associated with an input frame, the encoded video data frame including camera information and residuals associated with generating video data. For example, Figure 8 One or more of the encoder networks 825, 805, and / or 807 illustrated in obtain some (or all) of the encoded video data frames. In some cases, the camera information may include one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame.

[0178] For example, a camera projection matrix may be used to perform 3D normalization, such as by using Figure 8 3D warping engine 824 illustrated in FIG. In some examples, the camera pose information may include camera pose information from one or more times (such as time t and time t-1). In some examples, the depth information associated with the input frame may be a depth map, such as Figure 8 The depth map d shown in t The depth map may include depth information of a plurality of pixels included in the input frame.

[0179] At block 1104, process 1100 includes generating a camera motion compensated frame based on the reference frame and the camera information. Figure 8 3D warping engine 824 illustrated in the example generates a camera motion compensated frame. In some aspects, generating a camera motion compensated frame includes warping a reference frame based on a reconstructed depth map associated with an input frame and camera information. The camera information used to perform the warping may include camera pose information associated with the input frame and camera pose information associated with the reference frame.

[0180] In some examples, the reconstructed depth map may be generated based on a latent representation of a depth map associated with the input frame, wherein an autoencoder is used to generate the latent representation of the depth map. For example, the latent representation of the depth map may be generated by Figure 8 The depth map prediction system 822 illustrated in FIG. 8 is generated, where the encoder network 825 and the decoder network 823 are included in an autoencoder.

[0181] In some examples, a camera motion compensated frame may be generated based on warping a reference frame using a reconstructed optical flow map associated with the camera information. The reconstructed optical flow map may be generated based on camera motion information determined from camera pose information and a depth map associated with an input frame. For example, the camera pose information may include camera pose information associated with the input frame and may include camera pose information associated with the reference frame. In some examples, the reconstructed optical flow map may be generated based on a latent representation of an optical flow map generated using the camera pose information and the depth map.

[0182] For example, the latent representation of the optical flow map can be compared with Figure 8 805 as latent data generated using the object motion prediction system 802 and / or the encoder network 805. o In some cases, the optical flow information may be generated based on a latent representation of the optical flow associated with the input frame. The optical flow information may include object motion information determined between the input frame and the reference frame using an optical flow autoencoder. For example, the optical flow autoencoder may use each of the Figure 8 The object motion prediction system 802 and / or the combination of the encoder network 805 and the decoder network 803 illustrated in the example are implemented.

[0183] At block 1106, process 1100 includes generating optical flow information associated with the object motion determined based at least on the input frame and the reference frame. For example, the optical flow information may be used Figure 8 The object motion prediction system 802 illustrated in FIG. 1 may be generated by the object motion prediction system 802 illustrated in FIG. 1 and / or may be generated by the object motion prediction system 802 illustrated in FIG. Figure 8 In some cases, the optical flow information may be generated based on a latent representation of the optical flow associated with the input frame. The latent representation of the optical flow may be generated with Figure 8 The latent representation z instantiated in m In some examples, the latent representation of the optical flow associated with the input frame can be obtained by an optical flow encoder network and / or an encoder network included in an optical flow autoencoder (e.g., Figure 8 The latent representation of the optical flow can be generated by the optical flow decoder network and / or the decoder network included in the optical flow decoder (e.g., Figure 8 The optical flow information may include object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0184] At block 1108, process 1100 includes generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information. For example, the motion compensated frame may be composed of Figure 8 The warping engine 804 illustrated in FIG. 8 generates frame predictions based on receiving camera motion compensated frames (e.g., initial frame predictions) from the 3D warping engine 824 and optical flow information from the object motion prediction system 802 / decoder network 803 as inputs, each of which is described in detail in FIG. Figure 8 The motion compensated frame may include separate camera motion compensation and object motion compensation and may be used to generate a reconstructed input frame.

[0185] At block 1110, process 1100 includes generating a reconstructed input frame based on the motion compensated frame and the residual. For example, the motion compensated frame may be generated as described above with respect to block 1108 and as described above with respect to block 1110. Figure 8 The residual may be a prediction residual generated by a decoder network 809 of the residual prediction system 806. The prediction residual may be generated by the decoder network 809 based on the latent representation z of the residual received from the encoder network 807. r As input to generate, such as Figure 8 exemplified.

[0186] In some examples, the processes described herein (e.g., process 1100 and / or other processes described herein) can be performed by a computing device or apparatus (such as a computer having Fig.12 The computing device may include any suitable device, such as an autonomous vehicle computer, a robotic device, a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device, a server device, a video game device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a camera device, a set-top box device, and / or any other computing device having the resource capabilities to perform the processes described herein, including process 1100. In some examples, the computing device may include a mobile device, a wearable device, an XR device, a personal computer, a laptop computer, a video server, a television, a camera, a set-top box, a video game console, or other device. In some examples, process 1100 may be performed by a computer having a processor that implements Figure 8 The computing device architecture 1200 of the machine learning-based video decoding system is executed by the computing device.

[0187] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers, or combined transmitter-receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0188] Components of a computing device may be implemented in circuitry. For example, a component may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0189] The process 1100 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0190] In addition, the processes described herein (including process 1100) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination of the above. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0191] Fig.12 An example computing device architecture 1200 is illustrated of an example computing device that may implement the various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an XR device, a personal computer, a laptop, a video server, a video game console, a robotic device, a set-top box, a television, a camera, a server, or other device. For example, the computing device architecture 1200 may implement Figure 812. The components of the computing device architecture 1200 are shown as being in electrical communication with each other using a connection 1205, such as a bus. The example computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and a computing device connection 1205 that couples various computing device components including a computing device memory 1215, such as a read-only memory (ROM) 1220 and a random access memory (RAM) 1225, to the processor 1210.

[0192] The computing device architecture 1200 may include a cache of high-speed memory directly connected to the processor 1210, very close to the processor, or integrated as part of the processor. The computing device architecture 1200 may copy data from the memory 1215 and / or the storage device 1230 to the cache 1212 for fast access by the processor 1210. In this way, the cache can provide a performance improvement that avoids delays in the processor 1210 when waiting for data. These modules and other modules can control or be configured to control the processor 1210 to perform various actions. Other computing device memories 1215 can also be used. The memory 1215 may include a variety of different types of memory with different performance characteristics. The processor 1210 may include any general-purpose processor and hardware or software services (such as service 1 1232, service 2 1234, and service 3 1236 stored in the storage device 1230) configured to control the processor 1210, as well as a special-purpose processor in which software instructions are incorporated into the processor design. The processor 1210 may be a self-contained system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0193] To enable user interaction with the computing device architecture 1200, the input device 1245 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 1235 may also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-modal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 1200. The communication interface 1240 may generally govern and manage user input and computing device output. There is no restriction to operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0194] The storage device 1230 is a non-volatile memory and may be a hard disk or other type of computer readable medium that can store computer accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 1225, a read-only memory (ROM) 1220, and hybrids thereof. The storage device 1230 may include services 1232, 1234, 1236 for controlling the processor 1210. Other hardware or software modules are contemplated. The storage device 1230 may be connected to the computing device connector 1205. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer readable medium connected to the necessary hardware components (such as the processor 1210, the connector 1205, the output device 1235, etc.) to perform the function.

[0195] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device may include any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific aspects. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0196] Specific details are provided in the description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those components shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown as not having unnecessary details to avoid blurring examples.

[0197] Various aspects and / or examples may be described above as a process or method, which is depicted as a flow chart, a flowchart, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0198] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0199] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as flash memory), memory or memory devices, disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, etc. Computer-readable media may store thereon code and / or machine-executable instructions, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, messaging, token passing, network sending, etc.

[0200] In some aspects, computer readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0201] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0202] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0203] In the foregoing description, various aspects of the present application are described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or jointly. In addition, without departing from the scope of this specification, various aspects of the present disclosure can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0204] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0205] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0206] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0207] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0208] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0209] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0210] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0211] Illustrative aspects of the present disclosure include:

[0212] Aspect 1: A device for decoding video data, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: obtain an encoded video data frame associated with an input frame, the encoded video data frame comprising camera information and residual associated with generating the video data; generate a camera motion compensated frame based on a reference frame and the camera information; generate optical flow information associated with an object motion determined based at least on the input frame and the reference frame; generate a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; and generate a reconstructed input frame based on the motion compensated frame and the residual.

[0213] Aspect 2: The apparatus according to aspect 1, wherein the camera information comprises one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame.

[0214] Aspect 3: The apparatus according to aspect 2, wherein the depth information associated with the input frame is a depth map including depth information of a plurality of pixels included in the input frame.

[0215] Aspect 4: An apparatus according to any one of Aspects 1 to 3, wherein, in order to generate the camera motion compensated frame, the at least one processor is configured to: regularize the reference frame based on a reconstructed depth map associated with the input frame and the camera information; wherein the camera information includes camera pose information associated with the input frame and camera pose information associated with the reference frame.

[0216] Aspect 5: The apparatus according to Aspect 4, wherein the at least one processor is further configured to generate the reconstructed depth map based on a latent representation of a depth map associated with the input frame, wherein the latent representation of the depth map is generated using an autoencoder.

[0217] Aspect 6: An apparatus according to any one of Aspects 4 to 5, wherein: the optical flow information is generated based on a latent representation of the optical flow associated with the input frame; and the optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0218] Aspect 7: An apparatus according to any one of Aspects 1 to 6, wherein in order to generate the camera motion compensated frame, the at least one processor is configured to: regularize the reference frame based on a reconstructed optical flow map associated with the camera information; wherein the reconstructed optical flow map is generated based on camera motion information determined using camera pose information and a depth map associated with the input frame.

[0219] Aspect 8: The apparatus according to Aspect 7, wherein the camera pose information comprises camera pose information associated with the input frame and camera pose information associated with the reference frame.

[0220] Aspect 9: The apparatus according to aspect 8, wherein the at least one processor is further configured to generate the reconstructed optical flow map based on a latent representation of an optical map generated using the camera pose information and the depth map.

[0221] Aspect 10: An apparatus according to any one of Aspects 7 to 9, wherein: the optical flow information is generated based on a latent representation of the optical flow associated with the input frame; and the optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0222] Aspect 11: A method for decoding video data, the method comprising: obtaining an encoded video data frame associated with an input frame, the encoded video data frame comprising camera information and residual associated with generating the video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with an object motion determined at least based on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; and generating a reconstructed input frame based on the motion compensated frame and the residual.

[0223] Aspect 12: The method according to aspect 11, wherein the camera information includes one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame.

[0224] Aspect 13: The method according to aspect 12, wherein the depth information associated with the input frame is a depth map including depth information of a plurality of pixels included in the input frame.

[0225] Aspect 14: A method according to any one of Aspects 11 to 13, wherein generating the camera motion compensated frame includes regularizing the reference frame based on a reconstructed depth map associated with the input frame and the camera information, wherein the camera information includes camera pose information associated with the input frame and camera pose information associated with the reference frame.

[0226] Aspect 15: The method according to aspect 14, further comprising generating the reconstructed depth map based on a latent representation of a depth map associated with the input frame, wherein the latent representation of the depth map is generated using an autoencoder.

[0227] Aspect 16: A method according to any one of Aspects 14 to 15, wherein: the optical flow information is generated based on a latent representation of the optical flow associated with the input frame; and the optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0228] Aspect 17: A method according to any one of Aspects 11 to 16, wherein generating the camera motion compensated frame includes regularizing the reference frame based on a reconstructed optical flow map associated with the camera information, wherein the reconstructed optical flow map is generated based on camera motion information determined using camera pose information and a depth map associated with the input frame.

[0229] Aspect 18: The method according to Aspect 17, wherein the camera pose information comprises camera pose information associated with the input frame and camera pose information associated with the reference frame.

[0230] Aspect 19: The method according to Aspect 18, further comprising generating the reconstructed optical map based on a latent representation of an optical flow map generated using the camera pose information and the depth map.

[0231] Aspect 20: A method according to any one of Aspects 17 to 19, wherein: the optical flow information is generated based on a latent representation of the optical flow associated with the input frame; and the optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0232] Aspect 21: A device for encoding video data, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: obtain an input frame of video data and camera information associated with generating the input frame of the video data; generate a camera motion compensation frame based on a reference frame and the camera information; generate optical flow information associated with an object motion determined at least based on the input frame and the reference frame; generate a motion compensation frame by regularizing the camera motion compensation frame based on the optical flow information; determine a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensation frame; and generate an encoded video data frame associated with the input frame of the video data, the encoded video data frame including the camera information and the residual.

[0233] Aspect 22: The apparatus according to Aspect 21, wherein the camera information comprises one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame.

[0234] Aspect 23: The apparatus according to Aspect 22, wherein the depth information associated with the input frame is a depth map including depth information of a plurality of pixels included in the input frame.

[0235] Aspect 24: An apparatus according to any one of Aspects 21 to 23, wherein, in order to generate the camera motion compensated frame, the at least one processor is configured to: regularize the reference frame based on a reconstructed depth map associated with the input frame and the camera information; wherein the camera information includes camera pose information associated with the input frame and camera pose information associated with the reference frame.

[0236] Aspect 25: The apparatus of Aspect 24, wherein the at least one processor is further configured to generate the reconstructed depth map based on a latent representation of a depth map associated with the input frame, wherein the latent representation of the depth map is generated using an autoencoder.

[0237] Aspect 26: An apparatus according to any one of Aspects 24 to 25, wherein: the optical flow information is generated based on a latent representation of the optical flow associated with the input frame; and the optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0238] Aspect 27: An apparatus according to any one of Aspects 21 to 26, wherein in order to generate the camera motion compensated frame, the at least one processor is configured to: regularize the reference frame based on a reconstructed optical flow map associated with the camera information; wherein the reconstructed optical flow map is generated based on camera motion information determined using camera pose information and a depth map associated with the input frame.

[0239] Aspect 28: The apparatus according to Aspect 27, wherein the at least one processor is further configured to generate the reconstructed optical flow map based on a latent representation of an optical map generated using the camera pose information and the depth map.

[0240] Aspect 29: An apparatus according to any one of Aspects 27 to 28, wherein: the optical flow information is generated based on a latent representation of the optical flow associated with the input frame; and the optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

[0241] Aspect 30: A method for encoding video data, the method comprising: obtaining an input frame of video data and camera information associated with the input frame for generating the video data; generating a camera motion compensation frame based on a reference frame and the camera information; generating optical flow information associated with an object motion determined at least based on the input frame and the reference frame; generating a motion compensation frame by regularizing the camera motion compensation frame based on the optical flow information; determining a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensation frame; and generating an encoded video data frame associated with the input frame of the video data, the encoded video data frame including the camera information and the residual.

[0242] Aspect 31: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations described in aspects 1 to 20.

[0243] Aspect 32: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations described in aspects 21 to 30.

[0244] Aspect 33: An apparatus comprising means for performing any of the operations according to aspects 1 to 20.

[0245] Aspect 34: An apparatus comprising means for performing any of the operations according to aspects 21 to 30.

Claims

1. A device for decoding video data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: Obtaining an encoded video data frame associated with an input frame, the encoded video data frame including camera information and residuals associated with generating the video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with a motion of an object determined based at least on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; as well as A reconstructed input frame is generated based on the motion compensated frame and the residual. 2 . The apparatus of claim 1 , wherein the camera information comprises one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame. 3 . The apparatus according to claim 2 , wherein the depth information associated with the input frame is a depth map including depth information of a plurality of pixels included in the input frame.

4. The apparatus of claim 1 , wherein to generate the camera motion compensated frame, the at least one processor is configured to: regularizing the reference frame based on a reconstructed depth map associated with the input frame and the camera information; The camera information includes camera pose information associated with the input frame and camera pose information associated with the reference frame.

5. The apparatus of claim 4, wherein the at least one processor is further configured to generate the reconstructed depth map based on a latent representation of a depth map associated with the input frame, wherein the latent representation of the depth map is generated using an autoencoder.

6. The device according to claim 4, wherein: generating the optical flow information based on a latent representation of optical flow associated with the input frame; and The optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

7. The apparatus of claim 1 , wherein to generate the camera motion compensated frame, the at least one processor is configured to: regularizing the reference frame based on a reconstructed optical flow map associated with the camera information; Wherein the reconstructed optical flow map is generated based on camera motion information determined using camera pose information and a depth map associated with the input frame. 8 . The apparatus of claim 7 , wherein the camera pose information comprises camera pose information associated with the input frame and camera pose information associated with the reference frame.

9. The apparatus of claim 8, wherein the at least one processor is further configured to generate the reconstructed optical flow map based on a latent representation of an optical flow map generated using the camera pose information and the depth map.

10. The device according to claim 7, wherein: generating the optical flow information based on a latent representation of optical flow associated with the input frame; and The optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

11. A method for decoding video data, the method comprising: Obtaining an encoded video data frame associated with an input frame, the encoded video data frame including camera information and residuals associated with generating the video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with a motion of an object determined based at least on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; as well as A reconstructed input frame is generated based on the motion compensated frame and the residual. 12 . The method of claim 11 , wherein the camera information comprises one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame. 13 . The method of claim 12 , wherein the depth information associated with the input frame is a depth map including depth information of a plurality of pixels included in the input frame.

14. The method of claim 11, wherein generating the camera motion compensated frame comprises regularizing the reference frame based on a reconstructed depth map associated with the input frame and the camera information, wherein the camera information comprises camera pose information associated with the input frame and camera pose information associated with the reference frame.

15. The method of claim 14, further comprising generating the reconstructed depth map based on a latent representation of a depth map associated with the input frame, wherein the latent representation of the depth map is generated using an autoencoder.

16. The method of claim 14, wherein: generating the optical flow information based on a latent representation of optical flow associated with the input frame; and The optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

17. The method of claim 11, wherein generating the camera motion compensated frame comprises regularizing the reference frame based on a reconstructed optical flow map associated with the camera information, wherein the reconstructed optical flow map is generated based on camera motion information determined using camera pose information and a depth map associated with the input frame.

18. The method of claim 17, wherein the camera pose information comprises camera pose information associated with the input frame and camera pose information associated with the reference frame.

19. The method of claim 18, further comprising generating the reconstructed optical flow map based on a latent representation of an optical flow map generated using the camera pose information and the depth map.

20. The method of claim 17, wherein: generating the optical flow information based on a latent representation of optical flow associated with the input frame; and The optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

21. An apparatus for encoding video data, the apparatus comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: Obtaining an input frame of video data and camera information associated with generating the input frame of video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with a motion of an object determined based at least on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; determining a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensated frame; as well as An encoded video data frame associated with the input frame of video data is generated, the encoded video data frame including the camera information and the residual.

22. The device of claim 21, wherein the camera information comprises one or more of a camera projection matrix, camera pose information associated with the input frame, and depth information associated with the input frame. 23 . The apparatus of claim 22 , wherein the depth information associated with the input frame is a depth map including depth information of a plurality of pixels included in the input frame.

24. The apparatus of claim 21 , wherein to generate the camera motion compensated frame, the at least one processor is configured to: regularizing the reference frame based on a reconstructed depth map associated with the input frame and the camera information; The camera information includes camera pose information associated with the input frame and camera pose information associated with the reference frame.

25. The apparatus of claim 24, wherein the at least one processor is further configured to generate the reconstructed depth map based on a latent representation of a depth map associated with the input frame, wherein the latent representation of the depth map is generated using an autoencoder.

26. The apparatus of claim 24, wherein: generating the optical flow information based on a latent representation of optical flow associated with the input frame; and The optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

27. The apparatus of claim 21, wherein to generate the camera motion compensated frame, the at least one processor is configured to: regularizing the reference frame based on a reconstructed optical flow map associated with the camera information; Wherein the reconstructed optical flow map is generated based on camera motion information determined using camera pose information and a depth map associated with the input frame.

28. The apparatus of claim 27, wherein the at least one processor is further configured to generate the reconstructed optical flow map based on a latent representation of an optical flow map generated using the camera pose information and the depth map.

29. The apparatus of claim 27, wherein: generating the optical flow information based on a latent representation of optical flow associated with the input frame; and The optical flow information includes object motion information determined between the input frame and the reference frame using an optical flow autoencoder.

30. A method for encoding video data, the method comprising: Obtaining an input frame of video data and camera information associated with generating the input frame of video data; generating a camera motion compensated frame based on a reference frame and the camera information; generating optical flow information associated with a motion of an object determined based at least on the input frame and the reference frame; generating a motion compensated frame by regularizing the camera motion compensated frame based on the optical flow information; determining a residual based on a difference between the input frame and a reconstructed input frame generated using the motion compensated frame; as well as An encoded video data frame associated with the input frame of video data is generated, the encoded video data frame including the camera information and the residual.

Images