Video Compression Using a Recursion-Based Machine Learning System

Abstract

This specification describes a technique for coding video content using a recursion-based machine learning tool. The device may include a neural network system including an encoder portion and a decoder portion. The encoder portion may generate output data for a current time-step operation of the neural network system based on an input video frame for a current time-step operation of the neural network system, reconstructed motion estimation data from a previous time-step operation, reconstructed residual data from the previous time-step operation, and recurrent state data from at least one recursive layer of the decoder portion of the neural network system from the previous time-step operation. The decoder portion of the neural network system may generate a reconstructed video frame for the current time-step operation based on the output data and the recurrent state data from the previous time-step operation.

Description

[Technical field]

[0001]

[0001] The present disclosure relates generally to video coding, including encoding (or compressing) and decoding (decompressing) video. For example, aspects of the present disclosure relate to coding (e.g., encoding and / or decoding) video content using one or more recursion-based machine learning systems. [Background technology]

[0002]

[0002] Many devices and systems enable video data to be processed and output for consumption. Digital video data includes large amounts of data to meet the demands of consumers and video providers. For example, consumers of video data desire high quality video, including high fidelity, resolution, frame rate, etc. As a result, the large amounts of video data required to meet these demands strain communication networks and devices that process and store the video data.

[0003]

[0003] Video coding techniques can be used to compress video data. The goal of video coding is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation to the video quality. As ever-evolving video services become available, encoding techniques with better coding efficiency are needed. Summary of the Invention

[0004]

[0004] Systems and techniques are described for coding (e.g., encoding and / or decoding) video content using one or more recursion-based machine learning systems. In one illustrative example, a method for processing video data is provided. The method includes obtaining, by an encoder portion of the neural network system, an input video frame for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, output data for the current time step operation of the neural network system, the output data being generated at least in part based on the input video frame, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; obtaining, by a decoder portion of the neural network system, output data for the current time step operation and the recurrent state data from at least one recurrent layer of the decoder portion from the previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame.

[0005] In another example, an apparatus is provided for processing video data, the apparatus including a memory and a processor coupled to the memory (e.g., implemented in a circuit). In some examples, two or more processors may be coupled to the memory and used to perform one or more of the operations. The processor is configured to: obtain, using an encoder portion of the neural network system, an input video frame for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generate, using the encoder portion, output data for the current time step operation of the neural network system, the output data being generated based at least in part on the input video frame, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; obtain, using a decoder portion of the neural network system, output data for the current time step operation and the recurrent state data from at least one recurrent layer of the decoder portion from the previous time step operation; generate a reconstructed video frame for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame.

[0006]

[0006] In another example, a non-transitory computer readable medium that, when executed by one or more processors, causes the one or more processors to: obtain, using an encoder portion of a neural network system, input video frames for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; and generate, using the encoder portion of the neural network system, output data for the current time step operation of the neural network system. A non-transitory computer-readable medium is provided having instructions stored thereon to cause the neural network system to: obtain, using a decoder portion of the neural network system, output data for the current time step operation and recurrent state data from at least one recurrent layer of the decoder portion from the previous time step operation; the output data being generated based at least in part on the input video frame, reconstructed motion estimation data from a previous time step operation, reconstructed residual data from the previous time step operation, and recurrent state data from the previous time step operation; and generate, using the decoder portion, a reconstructed video frame for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame.

[0007] In another example, an apparatus for processing video data is provided, the apparatus including: means for obtaining an input video frame for a current time step operation of a neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer from the previous time step operation; means for generating output data for a current time step operation of the neural network system, the output data being generated at least in part based on the input video frame, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; means for obtaining output data for a current time step operation and the recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; means for generating a reconstructed video frame for a current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame.

[0008]

[0008] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: determining, by a decoder portion of the neural network system, motion estimation data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; obtaining a previously reconstructed video frame generated during the previous time step operation; and generating a warped reconstructed video frame for the current time step operation by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step.

[0009]

[0009] In some aspects, the methods, apparatus, and computer-readable media described above further include determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and the recurrent state data from a previous time step operation; and generating a reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further comprise determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and the recurrent state data from a previous time step operation, and processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step. In such aspects, a reconstructed video frame is generated for the current time step operation based on processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0011] In some aspects, the above-described methods, apparatus, and computer-readable media further comprise: determining, by a motion estimation network of the neural network system, motion estimation data for a current time step operation based at least in part on the input video frame and at least one of a previously reconstructed video frame from a previous time step operation or a previous input video frame from a previous time step operation; generating a warped reconstructed video frame for the current time step by modifying one or more pixels of at least one of the previously reconstructed video frame or the previous input video frame using the motion estimation data determined for the current time step; and sending the warped reconstructed video frame and the motion estimation data determined for the current time step operation to an encoder portion of the neural network system. In such aspects, the output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the warped reconstructed video frame and the motion estimation data determined for the current time step operation.

[0012]

[0012] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame.

[0013] In some aspects, the recurrent state data includes one or more values ​​of nodes of at least one recurrent layer.

[0014] In some aspects, the decoder portion of the neural network system includes a plurality of recurrent layers, each of which provides a respective recurrent state data.

[0015] In some aspects, the methods, apparatus, and computer-readable media described above further comprise generating warped recursive state data for the current time step operation by modifying the recursive state data from the previous time step operation using the motion estimation data determined for the current time step operation or the previous time step operation. In such aspects, the reconstructed video frame generated for the current time step operation is generated based at least in part on the warped recursive state data.

[0016]

[0016] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: generating, by the decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicating the presence of data in a previously reconstructed video frame generated during a previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame generated for the current time step operation, and the reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data.

[0017]

[0017] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: obtaining a previously reconstructed video frame generated during a previous time step operation; generating a warped reconstructed video frame for the current time step operation by modifying one or more pixels of the previously reconstructed video frame using motion estimation data determined for the current time step during a current time step operation; determining, by the decoder portion during the current time step operation, first reconstructed residual data for the current time step operation; generating a reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; determining, by the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation; modifying the previously reconstructed video frame using the second reconstructed residual data during the current time step operation; and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation.

[0018] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: obtaining, by an encoder portion of the neural network system, one or more distortion maps; and generating, by the encoder portion of the neural network system, output data based at least in part on the one or more distortion maps. In some aspects, the one or more distortion maps include a first distortion map indicating a distortion between an input video frame for a current time step operation and a previously reconstructed video frame from a previous time step operation. In some aspects, the one or more distortion maps include a second distortion map indicating a distortion between an input video frame for a current time step operation and a warped reconstructed video frame. In such aspects, the warped reconstructed video frame may be generated by modifying one or more pixels of a previously reconstructed video frame from a previous time step operation.

[0019]

[0019] In some aspects, the methods, apparatus, and computer-readable media described above further include processing input data by a first layer of an encoder portion of the neural network system; determining, by the first layer of the encoder portion, a plurality of weight values ​​for the first layer of the encoder portion; setting weights of a second layer of the encoder portion of the neural network system to the plurality of weight values; and processing the input data by the second layer of the encoder portion using weights set to the plurality of weight values.

[0020] In some aspects, the methods, apparatus, and computer-readable media described above further comprise storing the output data in a storage medium.

[0021] In some aspects, the methods, apparatus, and computer-readable media described above further comprise storing the reconstructed video frames in a storage medium.

[0022]

[0022] In some aspects, the methods, apparatus, and computer-readable media described above further comprise sending the reconstructed video frames via a transmission medium to at least one device.

[0023]

[0023] In another illustrative example, a method for processing video data is provided, the method including: obtaining, by a decoder portion of a neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; determining, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtaining a previously reconstructed video frame generated during the previous time step operation; generating a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; and generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame.

[0024]

[0024] In another example, an apparatus for processing video data includes a memory and a processor coupled to the memory (e.g., implemented in a circuit). In some examples, two or more processors may be coupled to the memory and used to perform one or more of the operations. The processor is configured to: obtain, using a decoder portion of the neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; determine, using the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtain a previously reconstructed video frame generated during the previous time step operation; generate a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; and generate a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame.

[0025]

[0025] In another example, a non-transitory computer-readable medium is provided having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: use a decoder portion of the neural network system to obtain output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; use the decoder portion of the neural network system to determine motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtain a previously reconstructed video frame generated during the previous time step operation; generate a warped reconstructed video frame for the current time step operation by at least in part modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; and generate a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame.

[0026] In another example, an apparatus for processing video data is provided, the apparatus including: means for obtaining output data for a current time step operation of a neural network system and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from a previous time step operation of the neural network system; means for determining motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; means for obtaining a previously reconstructed video frame generated during the previous time step operation; means for generating a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; and means for generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame.

[0027]

[0027] In some aspects, the methods, apparatus, and computer-readable media described above further include determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; and generating a reconstructed video frame for the current time step operation at least in part by adding a warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0028]

[0028] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: determining, by the decoder portion during the current time step operation, reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion of the neural network system during the current time step operation; modifying, during the current time step operation, the previously reconstructed video frame using the reconstructed residual data from the previous time step operation; and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation.

[0029]

[0029] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step; and generating a reconstructed video frame for the current time step operation based at least in part on the processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0030]

[0030] In some aspects, the methods, apparatus, and computer-readable media described above further comprise generating warped recursive state data for the current time step operation, at least in part, by modifying recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or motion estimation determined for the previous time step operation; and generating a reconstructed video frame for the current time step operation based at least in part on the warped recursive state data.

[0031]

[0031] In some aspects, the methods, apparatus, and computer-readable media described above further comprise: generating, by the decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicating the presence of data in a previously reconstructed video frame generated during a previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame for the current time step operation, and the reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data.

[0032]

[0032] In some aspects, the methods, apparatus, and computer-readable media described above further comprise obtaining, by an encoder portion of the neural network system, input video frames for a current time step operation, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, output data for the current time step operation, the output data generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation.

[0033] In some aspects, the apparatus comprises a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, or other device. In some aspects, the apparatus comprises a mobile device (e.g., a mobile phone or a so-called "smartphone"). In some aspects, the apparatus further includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more videos including one or more images and / or video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to transmit the reconstructed video frames via a transmission medium to at least one device. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0034]

[0034] This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used separately to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0035]

[0035] The foregoing, together with other features and embodiments, will become more apparent from the following specification, claims, and accompanying drawings.

[0036]

[0036] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0037] [Figure 1]

[0037] FIG. 1 illustrates an example implementation of a system on a chip (SOC). [Figure 2A]

[0038] FIG. 1 is a diagram showing an example of a fully connected neural network. [Figure 2B]

[0039] FIG. 1 is a diagram showing an example of a locally connected neural network. [Figure 2C]

[0040] FIG. 1 illustrates an example of a convolutional neural network. [Figure 2D]

[0041] A detailed example of a deep convolutional network (DCN) designed to recognize visual features from images. [Diagram 3]

[0042] A block diagram showing a deep convolutional network (DCN). [Figure 4]

[0043] FIG. 1 illustrates an example of a system including a device operable to perform video coding using a recursion-based neural network, in accordance with some examples. [Figure 5A]

[0044] FIG. 1 illustrates an example of a neural network system used for video coding, in accordance with some examples. [Figure 5B]

[0045] FIG. 1 illustrates a detailed example of a recurrent-based neural network used for video coding, according to some examples. [Figure 6]

[0046] 1 illustrates an example of a motion estimation and motion compensation technique, according to some examples. [Figure 7]

[0047] FIG. 2 illustrates another example of a recursion-based neural network system used for video coding, in accordance with some examples. [Figure 8]

[0048] FIG. 2 illustrates another example of a recursion-based neural network system used for video coding, in accordance with some examples. [Figure 9]

[0049] FIG. 2 illustrates another example of a recursion-based neural network system used for video coding, in accordance with some examples. [Figure 10]

[0050] FIG. 1 illustrates an example of a single-level recursion and a multi-level recursion configuration, according to some examples. [Figure 11]

[0051] FIG. 2 illustrates another example of a recursion-based neural network system used for video coding, in accordance with some examples. [Figure 12]

[0052] FIG. 2 illustrates another example of a recursion-based neural network system used for video coding, in accordance with some examples. [Figure 13]

[0053] FIG. 2 illustrates another example of a recursion-based neural network system used for video coding, in accordance with some examples. [Figure 14]

[0054] FIG. 13 illustrates an example of using a distortion map as an input to a recursion-based neural network system, in accordance with some examples. [Figure 15]

[0055] FIG. 1 illustrates an example of using additional neural network layers in a recurrent-based neural network system, in accordance with some examples. [Figure 16]

[0056] 5 is a graph illustrating an example of results obtained using techniques described herein versus results obtained using other techniques, in accordance with some examples. [Figure 17]

[0057] 5 is a graph illustrating an example of results obtained using techniques described herein versus results obtained using other techniques, in accordance with some examples. [Figure 18]

[0058] 1 is a flowchart illustrating an example of a process for processing video data, according to some examples. [Figure 19]

[0059] 6 is a flowchart illustrating another example of a process for processing video data, according to some examples. [Figure 20]

[0060] FIG. 1 illustrates an example computing device architecture for an example computing device capable of implementing various techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038]

[0061] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be implemented without these specific details. The figures and descriptions are not limiting.

[0039]

[0062] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0040]

[0063] Digital video data can contain large amounts of data, especially as the demand for high quality video data continues to grow. For example, consumers of video data generally desire increasingly higher quality video with high fidelity, resolution, frame rates, etc. However, the large amounts of video data required to meet such demands can place a significant strain on communication networks as well as devices that process and store the video data.

[0041]

[0064] Various techniques may be used to code video data. Video coding may be performed according to a particular video coding standard. Exemplary video coding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG) coding, and Universal Video Coding (VVC). Video coding often uses prediction methods such as inter-prediction or intra-prediction that exploit redundancy present in a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality. As the demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0042]

[0065] Described herein are techniques for performing video coding using one or more machine learning (ML) systems. ML is a subset of pseudo-intelligence (AI). ML systems include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without explicit instructions. One example of an ML system is a neural network (also called a pseudo-neural network), which may be composed of an interconnected group of pseudo-neurons (e.g., nerve cell models). Neural networks may be used for various applications and / or devices, such as image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, among others.

[0043]

[0066] Individual nodes in a neural network may emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, the input data of each node may be multiplied by a corresponding weight value, and the products may be summed. The sums of the products may be adjusted by an optional bias, and an activation function may be applied to the result, resulting in the node's output signal or "output activation" (sometimes called an activation map or feature map). The weight values ​​may be determined initially by an iterative flow of training data through the network (e.g., the weight values ​​are established during a training phase in which the network learns how to discriminate between certain classes by their typical input data characteristics).

[0044]

[0067] There are various types of neural networks, including recurrent neural networks (RNNs), multi-layer perceptron (MLP) neural networks, and convolutional neural networks (CNNs), among others. RNNs operate on the principle of saving the output of a layer and feeding this output back to the input to aid in predicting the outcome of the layer. In MLP neural networks, data may be fed into an input layer, and one or more hidden layers provide a level of abstraction for the data. Predictions may then be made on the output layer based on the abstracted data. MLPs may be particularly well suited for classification prediction problems, where inputs are assigned classes or labels. Convolutional neural networks (CNNs) are a type of feed-forward artificial neural network. Convolutional neural networks may include a collection of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) that collectively tile the input space. Convolutional neural networks have numerous applications. In particular, CNNs are widely used within the areas of pattern recognition and classification.

[0045]

[0068] In a layered neural network architecture (called a deep neural network when there are multiple hidden layers), the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input of a third layer of artificial neurons, and so on. Convolutional neural networks can be trained to recognize a hierarchy of features. Computations in convolutional neural network architectures can be distributed across a population of processing nodes that can be configured in one or more computational chains. These multi-layered architectures can be trained one layer at a time and fine-tuned using backpropagation.

[0046]

[0069] 1 illustrates an exemplary implementation of a system-on-chip (SOC) 100 that may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., neural networks with weights), delays, frequency bin information, task information, among other information, may be stored in memory blocks associated with a neural processing unit (NPU) 108, in memory blocks associated with the CPU 102, in memory blocks associated with a graphics processing unit (GPU) 104, in memory blocks associated with a digital signal processor (DSP) 106, in memory blocks 118, and / or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from the memory blocks 118.

[0047]

[0070] The SOC 100 may also include additional processing blocks adapted for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 that may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and a multimedia processor 112 that may detect and recognize gestures, for example. In one implementation, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120 that may include a global positioning system.

[0048]

[0071] The SOC 100 may be based on an ARM instruction set. In one aspect of the disclosure, the instructions loaded into the CPU 102 may comprise code for searching a look-up table (LUT) for a stored multiplication result corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also comprise code for disabling a multiplier during a multiplication operation of the multiplication product when a look-up table hit of the multiplication product is detected. In addition, the instructions loaded into the CPU 102 may comprise code for storing a calculated multiplication product of an input value and a filter weight when a look-up table miss of the multiplication product is detected.

[0049]

[0072] SOC 100 and / or components thereof may be configured to perform video compression and / or decompression (also referred to as video encoding and / or decoding, collectively referred to as video coding) using machine learning techniques in accordance with aspects of the disclosure described herein. By using deep learning architectures to perform video compression and / or decompression, aspects of the disclosure can increase the efficiency of video compression and / or decompression on a device. For example, a device using the described video coding techniques can compress video more efficiently using machine learning based techniques and transmit the compressed video to another device, which can decompress the video more efficiently using the machine learning based techniques described herein.

[0050]

[0073] As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes of the input layer, processing is performed by hidden nodes of one or more hidden layers, and output is generated through output nodes of the output layer. A deep learning network generally includes multiple hidden layers. Each layer of a neural network may include a feature map or activation map that may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. The nodes may include one or more weights that are used to indicate the importance of one or more nodes of the layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers being used to determine simple, low-level characteristics of the input, and later layers building a hierarchy of more complex and abstract characteristics.

[0051]

[0074] Deep learning architectures may learn a hierarchy of features. For example, when presented with visual data, a first layer may learn to recognize relatively simple features in the input stream, such as edges. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data, or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0052]

[0075] Deep learning architectures can work particularly well when applied to problems that have a natural hierarchical structure. For example, motor vehicle classification can benefit from first learning to recognize wheels, windshields, and other features. These features can then be combined in different ways in higher layers to recognize cars, trucks, and planes.

[0053]

[0076] Neural networks can be designed with a variety of connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, with each neuron in a given layer communicating to a neuron in a higher layer. As explained above, in successive layers of a feedforward network, a hierarchical representation can be accumulated. Neural networks can also have recurrent or (also called top-down) feedback connections. In recurrent connections, the output from a neuron in a given layer can be communicated to another neuron in the same layer. Recurrent architectures can be useful for recognizing patterns across two or more of the input data chunks that are sent sequentially to the neural network. Connections from neurons in a given layer to neurons in a lower layer are called feedback (or top-down) connections. Networks with many feedback connections can be useful when recognition of high-level concepts can help discriminate certain low-level features of the input.

[0054]

[0077] The connections between layers of a neural network can be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, a neuron in a first layer can communicate its output to every neuron in a second layer such that each neuron in the second layer receives input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, a neuron in a first layer can be connected to a limited number of neurons in the second layer. More generally, the locally connected layer of the locally connected neural network 204 can be configured such that each neuron in the layer has the same or similar connectivity pattern, but with connection strengths that can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of the locally connected can result in spatially separate receptive fields in the upper layer, since upper layer neurons in a given region can receive inputs that are conditioned through training to the properties of a limited portion of the total inputs to the network.

[0055]

[0078] An example of a locally connected neural network is a convolutional neural network. FIG. 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g., 208). The convolutional neural network may be suitable for problems in which the spatial location of the inputs is meaningful. The convolutional neural network 206 may be used to implement one or more aspects of video compression and / or restoration according to aspects of the present disclosure.

[0056]

[0079] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN 200 designed to recognize visual features from images 226 input from an image capture device 230, such as an on-board camera. The DCN 200 of this example may be trained to identify traffic signs and numbers given on traffic signs. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or identifying traffic signals.

[0057]

[0080] The DCN 200 may be trained using supervised learning. During training, the DCN 200 may be presented with an image, such as an image 226 of a speed limit sign, and then a forward pass may be computed to generate the output 222. The DCN 200 may include a feature extraction section and a classification section. Upon receiving the image 226, the convolution layer 232 may apply a convolution kernel (not shown) to the image 226 to generate the first set of feature maps 218. As an example, the convolution kernel for the convolution layer 232 may be a 5×5 kernel that generates a 28×28 feature map. In this example, four different feature maps are generated in the first set of feature maps 218, so four different convolution kernels were applied to the image 226 in the convolution layer 232. A convolution kernel may also be referred to as a filter or a convolution filter.

[0058]

[0081] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0059]

[0082] In the example of FIG. 2D , the second set of feature maps 220 are convolved to generate a first feature vector 224. Moreover, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as “sign”, “60”, and “100”. A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 contains one or more features.

[0060]

[0083] In this example, the probability in output 222 for "sign" and "60" is higher than the probability of others of output 222, such as "30", "40", "50", "70", "80", "90", and "100". Before training, the output 222 generated by DCN 200 may be inaccurate. Thus, an error may be calculated between output 222 and a target output. The target output is the ground truth of image 226 (e.g., "sign" and "60"). The weights of DCN 200 may then be adjusted so that the output 222 of DCN 200 is more closely aligned with the target output.

[0061]

[0084] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error increases or decreases if the weights are adjusted. In the top layer, the gradient may correspond directly to the values ​​of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the values ​​of the weights and the calculated error gradients of the upper layers. The weights may then be adjusted to reduce the error. This manner of adjusting weights is sometimes called "back propagation" because it involves a "backward pass" through the neural network.

[0062]

[0085] In practice, the error gradient of the weights may be calculated over a small number of examples such that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the overall system no longer decreases or until the error rate reaches a target level. After training, the DCN may be presented with new images and a forward pass through the network may result in an output 222, which may be considered the inference or prediction of the DCN.

[0063]

[0086] A deep belief network (DBN) is a probabilistic model that comprises multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of a training dataset. DBNs can be obtained by stacking layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. RBMs are often used in unsupervised learning because they can learn probability distributions in the absence of information about the class into which each input should be categorized. Using a hybrid unsupervised and supervised paradigm, the lower RBM of the DBN can be trained in an unsupervised manner and act as a feature extractor, and the upper RBM can be trained in a supervised manner (on the joint distribution of inputs and target classes from the previous layer) and act as a classifier.

[0064]

[0087] Deep Convolutional Networks (DCNs) are networks of convolutional networks, composed of additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the network weights by using gradient descent methods.

[0065]

[0088] The DCN may be a feedforward network. In addition, as described above, the connections from neurons in the first layer of the DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of the DCN may be exploited for high speed processing. The computational burden of the DCN may be much less than that of a similarly sized neural network with, for example, recurrent or feedback connections.

[0066]

[0089] The processing of each layer of a convolutional network may be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input may be considered to be three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The output of the convolutional combinations may be considered to form a feature map in a subsequent layer, where each element of the feature map (e.g., 220) may receive inputs from various neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map may be further processed using nonlinearities, such as rectification, max(0,x), etc. Values ​​from neighboring neurons may be further pooled, which corresponds to downsampling, providing further local invariance and dimensionality reduction.

[0067]

[0090] 3 is a block diagram illustrating an example of a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 3, the deep convolutional network 350 includes convolutional blocks 354A, 354B. Each of the convolutional blocks 354A, 354B may be composed of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0068]

[0091] The convolution layer 356 may include one or more convolution filters, which may be applied to the input data 352 to generate feature maps. Although only two convolution blocks 354A, 354B are shown, the disclosure is not so limited and instead any number of convolution blocks (e.g., blocks 354A, 354B) may be included in the deep convolution network 350 according to design preferences. The normalization layer 358 may normalize the output of the convolution filters. For example, the normalization layer 358 may perform whitening or lateral suppression. The max pooling layer 360 may perform downsampling aggregation across space for local invariance and dimensionality reduction.

[0069]

[0092] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. Additionally, the deep convolutional network 350 may access other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and the navigation module 120, which are dedicated to sensors and navigation, respectively.

[0070]

[0093] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350, there are weights (not shown) that must be updated. The output of each of the layers (e.g., 356, 358, 360, 362, 364) may serve as an input of a subsequent one of the layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) provided at the first one of the convolutional blocks 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data includes a feature from the set of features.

[0071]

[0094] As mentioned above, digital video data can contain large amounts of data, which can place a significant burden on communication networks as well as devices that process and store the video data. For example, recording uncompressed video content generally results in large file sizes that increase significantly as the resolution of the recorded video content increases. In one illustrative example, uncompressed 16-bit per channel video recorded at 1080p / 24 (e.g., a resolution of 1920 pixels wide by 1080 pixels high captured at 24 frames per second) may occupy 12.4 megabytes per frame or 297.6 megabytes per second. Uncompressed 16-bit per channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame or 1195.2 megabytes per second.

[0072]

[0095] Network bandwidth is another constraint that can become problematic for large video files. For example, video content is often delivered over wireless networks (e.g., over LTE, LTE Advanced, New Radio (NR), WiFi, Bluetooth, or other wireless networks) and can constitute a large portion of consumer Internet traffic. Despite advances in the amount of available bandwidth in wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content in these networks.

[0073]

[0096] Because uncompressed video content can result in large files that can involve large memory for physical storage and significant bandwidth for transmission, video coding techniques can be utilized to compress and then decompress such video content.

[0074]

[0097] In order to reduce the size of video content, and therefore the amount of storage involved in storing the video content and the amount of bandwidth involved in sending out the video content, various video coding techniques may be implemented according to specific video coding standards, such as HEVC, AVC, MPEG, VVC, among others. Video coding often uses prediction methods, such as inter-prediction or intra-prediction, that exploit the redundancy present in a video image or sequence. A common goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality. As the demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0075]

[0098] Generally, an encoding device encodes video data according to a video coding standard to generate an encoded video bitstream. In some examples, an encoded video bitstream (or "video bitstream" or "bitstream") is a sequence of one or more coded video sequences. An encoding device may generate a coded representation of a picture by partitioning each picture into multiple slices. A slice is independent of other slices such that information in a slice is coded without dependency on data from other slices in the same picture. A slice includes one or more slice segments, including independent slice segments, and one or more dependent slice segments, if any, that depend on previous slice segments. In HEVC, a slice is partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, together with syntax for the samples, are referred to as a coding tree unit (CTU). A CTU may also be referred to as a "treeblock" or a "largest coding unit" (LCU). A CTU is the basic processing unit for HEVC encoding. A CTU may be divided into multiple coding units (CUs) of various sizes. The CU contains luma and chroma sample arrays called coding blocks (CBs).

[0076]

[0099] The luma and chroma CBs may be further divided into prediction blocks (PBs). A PB is a block of luma or chroma component samples that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). The luma PB and one or more chroma PBs, together with associated syntax, form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) is signaled in the bitstream for each PU and is used for inter prediction of the luma PB and one or more chroma PBs. The motion parameters are sometimes referred to as motion information. The CBs may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples of a color component to which a residual transform (e.g., possibly the same two-dimensional transform) is applied to code the prediction residual signal. A transform unit (TU) represents the TBs of luma and chroma samples and the corresponding syntax elements. Transform coding is described in more detail below.

[0077]

[0100] According to the HEVC standard, the transform may be performed using TUs. The TUs may be sized based on the size of the PUs in a given CU. The TUs may be the same size as the PUs or smaller than the PUs. In some examples, the residual samples corresponding to the CU may be subdivided into smaller units using a quadtree structure known as a residual quadtree (RQT). Leaf nodes of the RQT may correspond to the TUs. Pixel difference values ​​associated with the TUs may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoding device.

[0078]

[0101] When a picture of video data is partitioned into CUs, the encoding device predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, a prediction mode may be signaled in the bitstream using syntax data. The prediction mode may include intra prediction (or intra-picture prediction) or inter prediction (or inter-picture prediction). Intra prediction exploits the correlation between spatially adjacent samples in a picture. For example, using intra prediction, each PU is predicted from neighboring image data in the same picture using, for example, DC prediction to find an average value for the PU, planar prediction to fit a flat surface to the PU, directional prediction to extrapolate from neighboring data, or any other suitable type of prediction. Inter prediction uses temporal correlation between pictures to derive a motion compensated prediction for a block of image samples. For example, using inter prediction, each PU is predicted using a motion compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision of whether to code a picture area using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0079]

[0102] In some examples, one or more slices of a picture are assigned a slice type. The slice types include I slice, P slice, and B slice. An I slice (intraframe, independently decodable) is a slice of a picture that is coded only by intra prediction, and therefore is independently decodable because an I slice only requires data within a frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded using intra prediction and using unidirectional inter prediction. Each prediction unit or prediction block in a P slice is coded using either intra prediction or inter prediction. When inter prediction is applied, a prediction unit or prediction block is predicted only by one reference picture, and therefore the reference samples are only from one reference area of ​​one frame. A B slice (bidirectionally predicted frame) is a slice of a picture that can be coded using intra prediction and using inter prediction (e.g., either bi-predictive or uni-predictive). A prediction unit or prediction block of a B slice may be bidirectionally predicted from two reference pictures, where each picture derives one reference region, and the sample sets of the two reference regions are weighted (for example, with equal weights or with different weights) to generate a prediction signal of the bidirectionally predicted block. As described above, the slices of a picture are coded independently. In some cases, a picture may be coded as just one slice.

[0080]

[0103] After performing prediction using intra prediction and / or inter prediction, the encoding device may perform transformation and quantization. For example, after prediction, the encoding device may calculate a residual value corresponding to the PU. The residual value may comprise pixel difference values ​​between a current block of pixels being coded (PU) and a predictive block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a predictive block (e.g., issuing an inter prediction or intra prediction), the encoding device may generate a residual block by subtracting the predictive block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantify differences between pixel values ​​of the current block and pixel values ​​of the predictive block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0081]

[0104] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., size 32x32, 16x16, 8x8, 4x4, or other suitable size) may be applied to the residual data in each CU. In some embodiments, TUs may be used for the transform and quantization process implemented by the encoding device. A given CU having one or more PUs may also include one or more TUs. As described in more detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TUs to generate serialized transform coefficients for entropy coding.

[0082]

[0105] The encoding device may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0083]

[0106] Once quantization is performed, the coded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partition information, and any other suitable data, such as other syntax data. Different elements of the coded video bitstream may then be entropy coded by the coding device. In some examples, the coding device may utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector that may be entropy coded. In some examples, the coding device may perform an adaptive scan. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the coding device may entropy code the vector. For example, the coding device may use context-adaptive variable length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partition entropy coding, or another suitable entropy coding technique.

[0084]

[0107] The encoding device may send the encoded video bitstream data over a communication link to a receiving device, which may store the encoded video bitstream and / or may include a decoding device. The encoding device may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) to extract elements of one or more coded video sequences that make up the encoded video data. The encoding device may then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to a prediction stage of the encoding device. The decoding device then predicts blocks of pixels (e.g., PUs) using intra prediction, inter prediction, IBC, and / or other types of prediction. In some examples, the prediction is added to the output of the inverse transform (the residual data). The decoding device may output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a consumer of the content.

[0085]

[0108] Video coding systems and techniques defined by various video coding standards (e.g., the HEVC video coding techniques described above) may be capable of preserving most of the information in the raw video content and may be defined a priori based on signal processing and information theory concepts. However, while predefined compression algorithms may be generally applicable (e.g., to any type of video content), these predefined video coding techniques may lack some properties that are beneficial in network-based or other types of environments. For example, a video coding system must be capable of implementing low-latency operation. However, video coding systems and related techniques that operate according to existing video coding standards encode video and transmit it in batches of several frames (sometimes called groups of pictures or GoPs), and thus must wait for the accumulation of frames of the batches (e.g., GoPs) to be transmitted. Such batch-based video coding results in higher latency than would be achieved using frame-by-frame coding and transmission. Furthermore, such video coding systems and techniques lack the ability to adjust the GoP size (i.e., how often to transmit full frames rather than just residual information) at run-time depending on the characteristics of the video, leading to less-than-optimal operating points in rate-distortion space.

[0086]

[0109] Described herein are methods, systems, apparatus, and computer-readable media (collectively "systems and techniques") for performing video coding (video compression / encoding and / or video restoration / decoding) using a recurrent neural network architecture that provides feedback to an encoder stage and / or a decoder stage. In some examples, the systems and techniques may utilize multiple neural network systems to compress and decompress video frames or images. For example, a first neural network system may act as an image encoder (e.g., an image autoencoder) that compresses a full frame without using motion compensation based on one or more previous frames, similar to an intra-predicted frame (also called an I-frame). A second neural network system (e.g., a recurrent autoencoder) may be used to encode and / or decode a frame using motion compensation based on one or more previous frames and using residual information (similar to an inter-predicted frame, such as a uni-predicted frame using a single reference frame (also called a P-frame) or a bi-predicted frame using two reference frames (called a B-frame)).

[0087]

[0110] The motion compensation performed by the second neural network system may be used to determine data for the current frame by describing how pixels from a previously reconstructed frame along with residual information have moved to new positions in the current frame. The second neural network system may include one or more recurrent layers (e.g., in the decoder portion of the second neural network) that store state information that may be fed back to some layers of the neural network in future time steps. For example, the second neural network system may determine the motion compensation for frame x at current time step t. t and a second neural network system processes the subsequent frame x at the next time step t+1. t+1To process frame x from one or more recurrent layers t The one or more recurrent layers may be part of one or more recurrent neural networks (RNNs), one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, any combination thereof, and / or other types of neural network architectures.

[0088]

[0111] Various aspects related to such neural network systems are described herein. For example, in some implementations, the second neural network can have a fully autoregressive conditional structure (referring only to previous frames) that can operate frame-by-frame without any dependency on reverse-temporal frames (such as the GoP structure used in conventional standards-based video codecs). In some cases, the output of the second neural network can include previous reconstructed frames warped by motion compensation, residuals, reconstructed frames, motion estimation (e.g., optical flow, block motion vectors, or adaptive convolution kernels), any combination thereof, and / or other outputs. Feedback information from a first time step (e.g., time step t) can be provided from the decoder to the encoder and / or to the decoder at a later time step (e.g., at time step t+1). The feedback can be used to represent errors introduced in the decoding stage, which can be used by the encoder to compensate for those errors. The information provided to the encoder at the next time step from the decoder of the previous time step may include one or more previously reconstructed frames, one or more previously reconstructed motion estimates, one or more previously reconstructed residuals, one or more previous recurrent states of the decoder (e.g., from one or more recurrent layers of the decoder), any combination thereof, and / or other outputs. The recurrent states of a recurrent layer are sometimes referred to herein as hidden states.

[0089]

[0112] By coding (e.g., coding / compressing and decoding / decompressing) video content using one or more recursion-based neural network systems and techniques described herein, aspects of the present disclosure can provide video encoding and decoding mechanisms that generate compressed video content with video bitrates and / or quality that approach or exceed those of traditional, a priori-defined video compression algorithms (e.g., HEVC, VVC, MPEG-4, among others). In some cases, video encoding and decoding mechanisms using the recursion-based neural network systems and techniques described herein may be adaptable to various use cases by retraining the neural network. Additionally, aspects of the present disclosure may provide for autonomous generation of video encoders and decoders without requiring large amounts of syntax to be transmitted along with the bitstream.

[0090]

[0113] FIG. 4 illustrates a system 400 including a device 402 configured to perform video encoding and decoding. The device 402 is coupled to a camera 407 and a storage medium 414 (e.g., a data storage device). In some implementations, the camera 407 is configured to provide image data 408 (e.g., a video data stream) to a processor 404 for encoding by a first neural network system 410 and / or a second neural network system 411. In some implementations, the device 402 may be coupled to and / or include multiple cameras (e.g., a dual camera system, three cameras, or other number of cameras). In some cases, the device 402 may be coupled to a microphone and / or other input devices (e.g., a keyboard, a mouse, a touch input device such as a touch screen and / or a touch pad, and / or other input devices). In some examples, the camera 407, the storage medium 414, the microphone, and / or other input devices may be part of the device 402.

[0091]

[0114] The device 402 is also coupled to the second device 490 via a transmission medium 418, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 418 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The transmission medium 418 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 418 may include routers, switches, base stations, or any other equipment that may be useful for enabling communication from a source device to a receiving device. The wireless network may include any wireless interface or combination of wireless interfaces, including any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi, radio frequency (RF), UWB, WiFi-Direct, cellular, long term evolution (LTE), WiMax, etc.). The wired network may include any wired interface (e.g., fiber, Ethernet, power line Ethernet, Ethernet over coaxial cable, digital signal line (DSL), etc.). Wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0092]

[0115] The device 402 includes one or more processors 404 (referred to herein as “processors”) coupled to a memory 406, a first interface (“I / F 1”) 412, and a second interface (“I / F 2”) 416. The processor 404 is configured to receive image data 408 from a camera 407, from the memory 406, and / or from a storage medium 414. The processor 404 is coupled to the storage medium 414 via the first interface 412 (e.g., via a memory bus) and to a transmission medium 418 via a second interface 416 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0093]

[0116] The processor 404 includes a first neural network system 410 including an encoder portion 462 and a decoder portion 466, and a second neural network system 411 including an encoder portion 432 and a decoder portion 436. In some implementations, the neural network system 410 and / or the neural network system 411 may include one or more autoencoders. The encoder portion 462 is configured to receive input data 470 and process the input data 470 to generate output data 474 based at least in part on the input data 470. Similarly, the encoder portion 432 is configured to receive input data 440 and process the input data 440 to generate output data 444 based at least in part on the input data 440. In some cases, the encoder portion 432 of the neural network system 411 is configured to generate output data 444 based on the input data 440 and further based on the first state data 450, as described further below. In some cases, the encoder portion 432 is configured to generate the output data 444 at the current time step t based on the input data 440 and the first state data 450, and further based on a representation 476 generated by the decoder portion 466 of the neural network system 410 at the previous time step t-1 (e.g., when processing the input data 470 at the previous time step t-1), as described further below. In some cases, the encoder portion 432 is configured to generate the output data 444 at the current time step t based on the input data 440 and the first state data 450, and further based on a representation 446 generated by the decoder portion 436 of the neural network system 411 at the previous time step t-1 (e.g., when processing the input data 440 at the previous time step t-1), as described further below.

[0094]

[0117] In some implementations, the encoder portion 462 of the neural network system 410 is configured to perform a lossy compression of the input data 470 to generate the output data 474, such that the output data 474 has fewer bits than the input data 470. The encoder portion 462 may be trained to compress the input data 470 (e.g., a video frame) without using motion compensation based on any prior representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 462 may compress a video frame using only video data from that video frame and without using any data from previously reconstructed frames. Herein, the video frames processed by the encoder portion 462 may be referred to as intra-predicted frames (I-frames). In some examples, the I-frames may be generated using conventional video coding techniques (e.g., in accordance with HEVC, VVC, MPEG-4, or other video coding standards). In such examples, the processor 404 may include or be coupled to a video coding device (e.g., an encoding device) configured to perform block-based intra-prediction, such as those described above with respect to the HEVC standard. In such an example, the neural network system 410 may be excluded from the processor 404.

[0095]

[0118] In some implementations, the encoder portion 432 of the neural network system 411 is configured to perform a lossy compression of the input data 440 to generate the output data 444, resulting in the output data 444 having fewer bits than the input data 440. The encoder portion 432 may be trained to compress the input data 440 (e.g., a video frame) using motion compensation based on a previous representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 432 may compress a video frame using video data from that video frame and using data from a previously reconstructed frame. In this specification, the video frames processed by the encoder portion 462 may be referred to as intra-predicted frames (P-frames). Motion compensation may be used to determine data for a current frame by describing how pixels from a previously reconstructed frame along with residual information have moved to new positions in the current frame. An example of a motion compensation technique is described below with respect to FIG. 6.

[0096]

[0119] In a particular implementation, the encoder portion 462 of the neural network system 410 includes a neural network 463 and a quantizer 464. The neural network 463 may include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), any combination thereof, and / or other types of neural network architectures that generate intermediate data 472. The intermediate data 472 is input to the quantizer 464. The encoder portion 432 of the neural network system 411 includes a neural network 433 and a quantizer 434. The neural network 433 may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, any combination thereof, and / or other types of neural network architectures that generate intermediate data 442 that is then input to the quantizer 434. Examples of components that may be included in the encoder portion 462 and the encoder portion 432 are shown in FIGS. 5A-15.

[0097]

[0120] The quantizer 464 is configured to perform quantization, possibly entropy coding, of the intermediate data 472 to generate the output data 474. The output data 474 may include the quantized (possibly entropy coded) data. Similarly, the quantizer 434 is configured to perform quantization, possibly entropy coding, of the intermediate data 442 to generate the output data 444. The output data 444 may include the quantized (possibly entropy coded) data. The quantization operations performed by the quantizer 464 and the quantizer 434 may be the same operation, which may result in the generation of quantized codes (or data representative of the quantized codes generated by the neural network system 410 and / or the neural network system 411) from the intermediate data 472 and the intermediate data 442. The quantized codes (or data representative of the quantized codes) may also be referred to as latent codes or latents. In this specification, the entropy model applied to the latents may also be referred to as a "prior". In some examples, the quantization and entropy coding operations may be performed using existing quantization and entropy coding operations performed when encoding and / or decoding video data according to existing video coding standards, or may be performed by the neural network system 410 and / or the neural network system 411. For example, the neural network system 410 and / or the neural network system 411 may be trained using supervised training, during which residual data is used as input and quantization codes and entropy codes are used as known outputs (labels).

[0098]

[0121] The decoder portion 466 of the neural network system 410 is configured to receive the output data 474 (e.g., from the direct quantizer 464 and / or from the storage medium 414) and process the output data 474 to generate a representation 476 of the input data 470 based at least in part on the output data 474. Similarly, the decoder portion 436 of the neural network system 411 is configured to receive the output data 444 (e.g., from the direct quantizer 434 and / or from the storage medium 414) and process the output data 444 to generate a representation 446 of the input data 440 based at least in part on the output data 444. In some cases, the decoder portion 436 is configured to generate the representation 446 based on the output data 444 and the second state data 452, as described further below. In some cases, the decoder portion 436 is configured to generate the representation 446 at the current time step t based on the output data 444 and the second state data 452, as described further below, and further based on a representation 476 generated by the decoder portion 466 of the neural network system 410 at the previous time step t-1 (e.g., when processing the input data 470 at the previous time step t-1). In some cases, the decoder portion 436 is configured to generate the output data 446 at the current time step t based on the output data 444 and the second state data 452, as described further below, and further based on a representation 446 generated by the decoder portion 436 at the previous time step t-1 (e.g., when processing the input data 440 at the previous time step t-1).

[0099]

[0122] In certain implementations, the decoder portion 466 of the neural network system 410 includes a neural network 468 that may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, any combination thereof, and / or other types of neural network architectures. The decoder portion 436 of the neural network system 411 includes a neural network 438 that may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, any combination thereof, and / or other types of neural network architectures. Examples of components that may be included in the decoder portion 436 are shown in FIGS. 5A-15.

[0100]

[0123] The neural network 438 of the decoder portion 436 has a recurrent component that may include one or more RNNs, GRUs, LSTMs, ConvRNNs, ConvGRUs, ConvLSTMs, any combination thereof, and / or other neural network architectures with one or more recurrent layers. The recurrent component (e.g., one or more recurrent layers) may provide first state data 450 and second state data 452. The first state data 450 and second state data 452 correspond to a state of the neural network 438 resulting from generation of a representation 446 for one set of input data 440. The first state data 450 (e.g., associated with a time step t, e.g., during processing of a current video frame) and the second state data 452 are provided as inputs to the encoder portion 432 and the decoder portion 436, respectively, for a next successive set of input data 440 (e.g., at a next time step t+1, e.g., during processing of a next video frame processed after the current video frame). In an illustrative example, the state of the decoder portion 436 represented in the state data 450, 452 includes one or more values ​​of nodes in the neural network 438. Thus, the neural network system 411 operates as a recursion-based neural network system that can be trained to reduce (e.g., minimize) the difference between the input data 440 and the representation 446 of the input data 440 via a training set (e.g., a training set of input images and output images). In some implementations, the first state data 450 matches the second state data 452 (e.g., the first state data 450 is the same as the second state data 452). In other implementations, the first state data 450 can be different from the second state data 452.

[0101]

[0124] As shown in FIG. 4, feedback information is provided from the decoder portion 436 of the neural network system 411 to the encoder portion 432 of the neural network system 411. The feedback may be used to represent errors introduced in the decoding stage, which may be used by the encoder portion 432 to compensate for those errors. Information from the previous time step t-1 may be provided to the encoder from the decoder portion 436 at the next time step t by (

[0102]

number

[0103] The feedback information may include one or more of a previously reconstructed frame (denoted as ), a previously reconstructed motion estimate, a previously reconstructed residual, and / or a previous recursive state. For example, in the case of a sequence of video frames that may exhibit temporal correlation, a previous recursive state of the decoder portion 436 associated with a frame of the video may be provided as an input to both the encoder portion 432 and the decoder portion 436 when processing a next frame of the video. Using such recursive states in addition to other feedback information may increase the compression ratio (e.g., the number of bits in the input data 440 compared to the number of bits in the output data 444) and thus increase coding efficiency by allowing the encoder portion 432 to take into account errors encountered by the decoder portion 436. As an illustrative example, the state data 450, 452 can serve as inputs that enable the encoder portion 432 and the decoder portion 436 to generate a prediction for a next frame of a sequence of frames, and the encoder portion 432 can encode the residual (e.g., the difference between the next frame and the prediction of the next frame) as output data 444 at a reduced bit rate compared to encoding the output data 444 without taking into account the history of the decoder portion 436. Thus, the encoded data can be stored using a reduced amount of storage space, transmitted using a reduced amount of network bandwidth or at a higher transmission rate, or any combination thereof.

[0104]

[0125] The processor 404 is configured to send the output data 444 to at least one of the transmission medium 418 or the storage medium 414. For example, the output data 444 may be stored in the storage medium 414 for later retrieval and restoration by the decoder portion 436 to generate a representation 446 of the input data 440 as reconstructed data, such as for playback of video data that has been encoded / compressed to generate the output data 444. In some implementations, the output data 444 may be decoded in another decoder device that matches the decoder portion 436 (e.g., in the device 402 or in another device) to generate the representation 446 of the input data 440 as reconstructed data. As another example, the second device 490 may include a decoder that matches (or substantially matches) the decoder portion 436, and the output data 444 may be transmitted via the transmission medium 418 to generate the representation 446 of the input data 440 as reconstructed data in the second device 490.

[0105]

[0126] The components of system 400 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuitry (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing various operations described herein.

[0106]

[0127] Although system 400 is shown to include several components, one skilled in the art will appreciate that system 400 can include more or fewer components than those shown in Figure 4. For example, system 400 can also include or be part of a computing device, including input and output devices (not shown). In some implementations, system 400 may also include, or be part of, a computing device that includes one or more memory devices (e.g., one or more random access memory (RAM) components, read only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) in communication with and / or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and a baseband processor for each wireless interface) for implementing wireless communications, one or more wired interfaces (e.g., serial interfaces such as universal serial bus (USB) inputs, Lightning connectors, and / or other wired interfaces) for implementing communications via one or more wired connections, and / or other components not shown in FIG. 4 .

[0107]

[0128] In some implementations, system 300 may be implemented locally by and / or included within a computing device. For example, the computing device may include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses, or other AR device), a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), a television, and / or any other computing device with resource capabilities to perform the techniques described herein.

[0108]

[0129] In one example, the neural network system 411 (and, in some cases, the neural network system 410) may be incorporated into a portable electronic device that includes a memory 406 coupled to the processor 404 and configured to store instructions executable by the processor 404, and a wireless transceiver coupled to an antenna and the processor 404 and operable to transmit output data 444 to a remote device.

[0109]

[0130] 5A illustrates a system including neural network system 510 (as one example of neural network system 410 in FIG. 4) and neural network system 511 (as one example of neural network system 411). I-frame encoder 563 of neural network system 510 may include a neural network (e.g., including one or more CNNs) that is trained to encode or compress a video frame using video data from only that video frame without performing motion compensation based on data from a previously reconstructed frame. For example, as shown in FIG. 5A, I-frame encoder 563 may receive as input video frame x I-frame Receive video frame x I-frameVideo frame x using only the data contained within I-frame The I-frame encoder 563 encodes or compresses video frames x to generate intermediate data (e.g., intermediate data 472). I-frame may be encoded or compressed. In some cases, the intermediate data may include data representing the residual, which is I-frame Prediction of x and the actual input video frame x I-frame It represents the difference between

[0110]

[0131] The quantizer 564 of the neural network system 510 is an example of the quantizer 464 shown in Figure 4. The quantizer 564 generates a latent from the intermediate data output from the I-frame encoder 563. In some cases, the latent may be a quantized value of the compressed video frame x I-frame The quantizer 564 also includes an I-frame prior. The I-frame prior may include or represent an entropy model that is applied to the latent to generate an entropy code or neural network data (e.g., a nodal activation map or feature map) that represents the entropy code. The quantizer 564 generates output data (e.g., output data 474) that is output to the I-frame decoder 568 and / or storage device (e.g., storage medium 414) for later retrieval. For example, the output data from the quantizer 564 may be transmitted to the device 490 via the transmission medium 418.

[0111]

[0132] The I-frame decoder 568 of the neural network system 510 generates the reconstructed frame

[0112]

number

[0113] The I-frame encoder 563 processes the output data from the quantizer 564 to generate a loss function (L(x,y)). In some implementations, the I-frame encoder 563, the quantizer 564, and the I-frame decoder 568 may be implemented using a single neural network or multiple neural networks that are trained together exhaustively.

[0114]

number

[0115] may be used to train a single neural network using supervised learning techniques with a training set including the input images and the reconstructed output images. Further details regarding training of neural networks are described below with respect to the neural network system 511. In some implementations, the neural networks of the I-frame encoder 563, the I-frame decoder 568, and the quantizer 564 may be trained separately.

[0116]

[0133] As noted above, in some implementations, the I-frame encoder 563, the I-frame decoder 568, and the quantizer 564 may perform conventional video coding techniques (e.g., according to HEVC, VVC, MPEG-4, or other video coding standards) to generate I-frames. For example, a video coding device (capable of performing video encoding and decoding) may be configured to perform block-based intra prediction to generate I-frames according to the HEVC standard or other video coding standards.

[0117]

[0134] The neural network system 511 may sometimes be referred to as a P-frame autoencoder because the neural network system 511 uses motion compensation techniques to perform video coding (e.g., encoding and / or decoding). The encoder 533 of the neural network system 511 may include a neural network (e.g., including one or more CNNs) that is trained to encode or compress video frames using motion compensation based on data of one or more previously reconstructed frames. The neural network system 511 is shown in FIG. 5A at two time steps, t and t+1. The time step t shown in FIG. 5A is the time when an I-frame was generated (e.g.,

[0118]

number

[0119] ) of the operation of the neural network system 511 at the first time step. At time step t, the encoder 533 receives as input a video frame x t and the previously reconstructed frame x I-frame and receive video frame x t and video frame x I-frame by performing motion compensation to determine the amount of motion between video frame x t Encode the following.

[0120]

[0135] Various types of motion compensation may be performed by the encoder 533. For example, the encoder 533 may perform block-based motion compensation (similar to that done by video coding standards such as HEVC, VVC, MPEG-4, etc.), optical flow-based motion compensation, dynamic (or adaptive) convolution-based motion compensation, any combination thereof, and / or other types of motion compensation. Various motion compensation options are described below with respect to FIG. 6.

[0121]

[0136] Video Frame x t As a result of encoding, the encoder 533 may generate intermediate data (e.g., intermediate data 442). In some cases, the intermediate data may include data representing the residual and data representing the video frame x t and video frame x I-frame and motion data representing the amount of motion between video frame x t Prediction of x and the actual input video frame x t The difference between video frame x and t and video frame x I-frame The motion data, which represents the amount of motion between, may be conceptualized as being similar to a motion vector in conventional video coding techniques (e.g., HEVC, VVC, among others).

[0122]

[0137] The quantizer 534 of the neural network system 511 is one example of the quantizer 434 shown in Figure 4. The quantizer 534 generates a latent from the intermediate data output from the encoder 533. In some cases, the latent may be a quantized value of the encoded video frame x t (e.g., a quantized version of the residual and a quantized version of the motion data). In some cases, the latent may include neural network data (e.g., a nodal activation map or feature map) that represents the quantization code. The quantizer 534 also includes a prior that may include or represent an entropy model that is applied to the latent to generate an entropy code or neural network data (e.g., a nodal activation map or feature map) that represents the entropy code. The quantizer 534 generates output data (e.g., output data 444) that is output to the decoder 538 and / or a storage device (e.g., storage medium 414) for later retrieval. For example, the output data from the quantizer 534 may be transmitted to the device 490 via the transmission medium 418.

[0123]

[0138] The decoder 538 of the neural network system 511 outputs the reconstructed frame

[0124]

number

[0125] 5A , the decoder 538 processes the output data to generate quantizer 534. The decoder 538 can receive the output data from the quantizer 534, from a storage device (e.g., storage medium 414), or from another device via a transmission medium (e.g., transmission medium 418). As shown in FIG. 5A , the decoder 538 includes recurrent layers or components including one or more pre-recurrent decoder layers 570, one or more recurrent layers 571, and one or more post-recurrent layers 572. The one or more recurrent layers 571 can store state information and can feed back state information to some layers of the decoder 538 and / or encoder 533 at a subsequent time step (e.g., at time step t+1 as shown in FIG. 5A ). The feedback provided to previous layers in the neural network system 511 at a subsequent time step provides the neural network system 511 with a type of memory. The one or more pre-recursive decoder layers 570 and the one or more post-recursive layers 572 may include forward layers that unidirectionally pass information forward through other layers of the network, while the one or more recursive layers may pass information forward and / or backward through feedback, as described above.

[0126]

[0139] The recurrent state of the decoder 538 is shown in FIG. 5A as h. The recurrent state h shown in FIG. 5A corresponds to time step t-1, when the neural network system 510 generated the I-frame. t-1 may start with one or more zero values ​​(e.g., because the neural network system 510 does not include any recurrent layers). t represents the recurrent state of the decoder 538 at timestamp t.t may include neural network information from one or more recurrent layers 571. For example, the neural network information may include activation maps or feature maps (e.g., generated based on multiplication of weights of one or more recurrent layers 571 multiplied by data passed through the decoder 538). As shown in FIG. 5A, the recurrent state h t is passed as feedback to one or more recurrent layers 571 and to the encoder 533 at the next time step t+1.

[0127]

[0140] Based on the processing of the output data, one or more post-recursive layers 572 of the decoder 538 generate reconstructed motion estimation data

[0128]

number

[0129] and the reconstructed residual data

[0130]

number

[0131] Optionally, the reconstructed motion estimation data

[0132]

number

[0133] may include optical flow data (e.g., displacement vectors), dynamic convolution data (e.g., matrices or kernels for convolving the data), or block-based motion data (e.g., motion vectors), as described below with respect to FIG.

[0134]

number

[0135] is the previously reconstructed frame

[0136]

number

[0137] , which results in the generation of a warped frame.

[0138]

number

[0139] The pixels of are either moved to a new location or reconstructed using the motion estimation data

[0140]

number

[0141] The warped frame output by warping engine 567 is output to adder engine 569, which outputs the reconstructed frame for the current time step t.

[0142]

number

[0143] The residual data reconstructed to produce

[0144]

number

[0145] Add (or combine) the warped frames to the

[0146]

[0141] Recursive state information (e.g., recursive state h t ) plus the reconstructed frame

[0147]

number

[0148] can also be fed back to the neural network system 511 for operation at the next time step t+1. For example, as shown in FIG. 5A, the reconstructed frame

[0149]

number

[0150] and the recurrent state h t and are provided as input to the encoder 533. In some cases, as shown in FIG. 5A, previously reconstructed motion estimation data

[0151]

number

[0152] and the previously reconstructed residual data

[0153]

number

[0154] may also be fed back as input to the encoder 533 at the next operation time step t+1. At time step t+1, the encoder 533 calculates the current input video frame

[0155]

number

[0156] To generate a coded version of the current input video frame

[0157]

number

[0158] and the previously reconstructed frame

[0159]

number

[0160] and the recurrent state h t and the previously reconstructed motion estimation data

[0161]

number

[0162] and the previously reconstructed residual data

[0163]

number

[0164] The quantizer 534 may quantize the coded video data to generate latents and may perform entropy coding on the latents (using the prior) using the prior to generate output data.

[0142] One or more pre-recursive decoder layers 570, one or more recursive layers 571, and one or more post-recursive layers 572 of the decoder 538 can process the output data for the current time step t+1 to generate reconstructed motion estimation data

[0165]

number

[0166] and the reconstructed residual data

[0167]

number

[0168] The warping engine 567 of the decoder 538 may output a warped frame for the current time step t+1 by warping the previously reconstructed frame

[0169]

number

[0170] The adder engine 569 may warp (e.g., by performing motion compensation) the reconstructed frame for the current time step t+1.

[0171]

number

[0172] The reconstructed residual data

[0173]

number

[0174] We can add the warped frame to the

[0175]

[0143] Reconstructed frame

[0176]

number

[0177] and the recurrent state h t+1and the reconstructed motion estimation data

[0178]

number

[0179] and the reconstructed residual data

[0180]

number

[0181] may be fed back as an input to encoder 533 at the next time step t+2 of operation. Neural network system 511 may continue the motion estimation based coding process for the video sequence frame by frame until the next I-frame is needed (e.g., for random access to a point in time in the video to reset any motion error that may have accumulated, etc.). When an I-frame is needed, the next frame of video in the video sequence may be encoded by neural network system 510.

[0182]

[0144] In some implementations, the encoder 533, the quantizer 534, and the decoder 538 may be implemented using a single neural network or multiple neural networks that are trained together exhaustively. Figure 5B illustrates an example of a neural network architecture that may be used for the neural network system 511. In some implementations, a neural network architecture similar to that shown in Figure 5B may be used for the neural network system 510, but without a recurrent layer (e.g., a ConvGRU layer).

[0183] As shown in FIG. 5B, the input to the encoder 533 is the current frame to be encoded by the encoder 533.

[0184]

number

[0185] and a previously reconstructed frame (e.g., from neural network system 510 or neural network system 511).

[0186]

number

[0187] and the previous recursive state h t+1 Other inputs that may be provided to the encoder include previously reconstructed motion estimation data

[0188]

number

[0189] and the previously reconstructed residual data

[0190]

number

[0191] Includes:

[0192] The input is fed to a first convolutional layer (denoted as Conv c:64 k:5 s:2, where c is the number of channels (e.g., kernels or filters), k is the kernel size, and s is the stride associated with a given convolutional layer). The convolutional layer is fed to a node called the receptive field (e.g., the current frame

[0193]

number

[0194] A convolutional layer processes input data in a convolutional manner, such as by processing a region of pixels (of pixels in the current frame). A convolutional layer may be composed of one or more kernels (also called filters), where each kernel may correspond to a different activation or feature map, and each convolutional iteration of a kernel is a node or neuron of the convolutional layer. For example, the region of the current frame that the kernel covers in each convolutional iteration is

[0195]

number

[0196] The region of would be the receptive field for the kernel. In one illustrative example, the frame

[0197]

number

[0198] If k contains a 28×28 array and each kernel (and corresponding receptive field) is a 5×5 array (where k=5), then there will be 24×24 nodes in the convolutional layer. Note that video frames typically have a resolution greater than 28×28, such as 1920×1080 for 1080p video. Each connection between a node and a receptive field for that node learns a weight and possibly a global bias, such that each node learns to analyze its particular local receptive field in the input. Each node in a convolutional layer can have the same weights, and in some cases, biases (called shared weights and shared biases). For example, a kernel has the same depth as the array of weights (numbers) as input. In one example, a kernel learns a frame

[0199]

number

[0200] The current frame according to its three color components (for example, the red (R), green (G), and blue (B) color components).

[0201]

number

[0202] For example, you can have a depth of 3.

[0203] The convolutional nature of a convolutional layer results from the fact that each node in the convolutional layer is applied to its corresponding receptive field. For example, the kernel of a convolutional layer is

[0204]

number

[0205] You can start at the top left corner of the array, frame

[0206]

number

[0207] As mentioned above, each convolution iteration of the kernel can be considered as a node or neuron of the convolutional layer. In each convolution iteration, the value of the kernel is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5 × 5 kernel array is used for the image

[0208]

number

[0209] The multiplications from each convolution iteration can be added together to get the sum for that iteration or node. The process then multiplies the frame according to the receptive field of the next node in the convolution layer.

[0210]

number

[0211] The kernel continues at the next location in the convolutional layer. For example, the kernel may be moved by a stride amount (denoted by s in FIG. 5B) toward the next receptive field. The stride amount may be set to 1, 2, or other suitable amount. For example, if the step amount is set to 1, the kernel will be moved one pixel to the right every convolutional iteration. Processing the kernel at each unique location of the input quantity produces a number representing the kernel result for that location, resulting in a summation value being determined for each node of the convolutional layer.

[0212] The mapping from the input layer to the convolutional hidden layer 622a is called the activation map (or feature map). The activation map contains a value for each node that represents the kernel result at each location of the input quantity. The activation map is an array (e.g., a vector of the activation map for the current frame) that contains the various sum values ​​resulting from each iteration of the kernel on the input quantity.

[0213]

number

[0214] ), for example, if a 5x5 kernel is applied to each pixel (with a step amount of 1) of a 28x28 input frame, then the activation map will contain a 24x24 array. A convolutional layer can contain several activation maps to identify multiple features of the input. For example, using three activation maps, a convolutional layer can detect three different types of features, with each feature being detectable across the entire input.

[0215]

[0149] Normalization layers and nonlinear layers may be applied after the first convolutional layer. A normalization layer may be used (e.g., during training) to nonlinearly normalize the inputs in a hidden layer (e.g., a convolutional layer). One illustrative example of a normalization layer includes BatchNorm, which is used in the architecture of FIG. 5B. In one illustrative example, BatchNorm normalizes the inputs of hidden layers z as follows: k can be normalized.

[0216]

number

[0217]

[0150] where E[x] is the first moment of x (eg, the mean or expectation) and V[x] is the second moment of x (eg, the variance).

[0218]

[0151] Nonlinear layers can be used to introduce nonlinearity into a system that was previously computing linear operations. One illustrative example of a nonlinear layer is the rectified linear unit (ReLU) as used in the architecture of FIG. 5B. The ReLU layer can apply a function f(x)=max(0,x) to all of the values ​​in the input quantities, which changes all negative activations to 0. Thus, the ReLU can increase the nonlinear properties of the neural network without affecting the receptive field of the convolutional layers.

[0219]

[0152] The output of the first ReLU is fed to a second set of convolutional, BatchNorm and ReLU layers. The output of the second ReLU is fed to a first residual layer (ResBlock B), which feeds its output to a second residual layer (ResBlock A). Details of ResBlock A and ResBlock B are also shown in Fig. 5B. The output of ResBlock A is combined (or added) with the output of the second ReLU, and the combined output is fed to a third convolutional layer. The output of the third convolutional layer is fed to a third BatchNorm layer, which outputs data to a quantization layer (denoted as soft / hard quantization). The soft / hard quantization layer outputs one or more quantized (possibly entropy coded) coefficients z t Outputs one or more coefficients z t , may be optionally stored and optionally provided to the decoder 538.

[0220]

[0153] The input to the decoder 538 is one or more coefficients z t One or more coefficients z tis processed by a first convolutional layer (denoted as Tconv c:128 k:3 s:2), followed by a first BatchNorm layer and a first ReLu layer. The output of the first ReLU is fed to a first residual layer (ResBlock B), which feeds its output to a second residual layer (ResBlock A). The output of ResBlock A is combined (or added) with the output of the first ReLU layer, and the combined output is fed to a second convolutional layer (denoted as Tconv c:64 k:5 s:2). The second convolutional layer is followed by a second BatchNorm layer and a second ReLu layer, which outputs data to a third convolutional layer (denoted as Tconv c:C2 k:5 s:2), followed by a third BatchNorm layer. The output of the third BatchNorm layer is fed to a recurrent layer (denoted as ConvGRU c:C3 k:3 s:1). As shown, another input to the recurrent layer is the previous recurrent state h for the previous time step t-1. t-1 where the output of the recurrent layer contains the recurrent state for the current time step t. The output of the recurrent layer is fed to the last convolutional layer (denoted as Conv c:5 k:5 s:1), which produces the reconstructed motion estimation data

[0221]

number

[0222] and the reconstructed residual data

[0223]

number

[0224] and is output.

[0225] As described above, the warping engine 567 of the decoder 538 uses the previously reconstructed frame t to generate a warped frame for the current time step t.

[0226]

number

[0227] (e.g., by performing motion compensation), and the adder engine 569 generates a reconstructed frame for the current time step t

[0228]

number

[0229] The reconstructed residual data

[0230]

number

[0231] The warped frame can be added to the warped frame. In some implementations, the warping engine 567 and / or the adder engine 569 can be part of the neural network system 511 (e.g., implemented using one or more convolutional layers, one or more normalization layers, one or more nonlinear layers, one or more residual blocks, or other neural network layers), such as in the encoder configuration shown in FIG. 7. In some implementations, the warping engine 567 and / or the adder engine 569 can be components external to the neural network system 511. For example, in such implementations, the warping and addition are unlearned operators, in which case these operations are not performed using a neural network or other machine learning system.

[0232]

[0155] The neural network system 511 may be trained using any suitable machine learning training technique. In one illustrative example, a supervised learning technique may be used to train the neural network system 511. For example, a backpropagation training process may be used to adjust the weights (and possibly other parameters such as biases) of the nodes of the neural network. Backpropagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. The process may be repeated for a certain number of iterations for each set of training data until the parameter weights of the neural network system 511 are precisely tuned.

[0233] In one illustrative example, the training data used to train the neural network system 511 includes as input (unencoded / uncompressed) video frames x t and the reconstructed video frame as the known output

[0234]

number

[0235] (Input video frame x t The forward pass can include passing the input video frame through the neural network. The weights can be initially randomized before the neural network is trained. For a first training iteration for a neural network system, the weights have not yet been calibrated, so the output can include values ​​that do not give a preference to any particular output. For example, the output can include a reconstructed version of the reconstructed video frame.

[0236]

number

[0237] After the first training iteration using the initial weights, the reconstructed video frame

[0238]

number

[0239] presumably corresponds to the corresponding input frame x t It will be something that does not resemble.

[0240] A loss function can be used to analyze the errors in the output:

[0241]

number

[0242] In an example using the loss function shown in FIG. 5A to train the neural network system 511,

[0243]

number

[0244] When other training data is used, other loss functions may be used. An example of another loss function is

[0245]

number

[0246] This includes the mean squared error (MSE), which is defined as: MSE is calculated as half the sum of the squared differences between the actual and predicted (output) responses.

[0247]

[0158] The loss (or error) may be high for the first training video frames because the actual output values ​​(the reconstructed frames output by the network) may differ significantly from the predicted outputs (the known reconstructed versions of the input frames). The objective of training is to minimize the amount of loss for the predicted outputs. The neural network can perform a backward pass by determining which inputs (weights) contributed most to the neural network's loss, and the weights can be adjusted so that the loss is reduced and eventually minimized. To determine the weights that contributed most to the neural network's loss, the derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight in a particular layer) can be calculated. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be done by:

[0248]

number

[0249] where w denotes the weight and w i where η denotes the initial weights and η denotes the learning rate. The learning rate may be set to any suitable value, with a high learning rate involving larger weight updates and a lower value indicating smaller weight updates. The neural network system 511 may continue to be trained in such a manner until a desired output is achieved. In some cases, the neural network system 510 may be trained in a similar manner.

[0250]

[0159] Figure 6 is a diagram illustrating different types of motion estimation that may be implemented by a motion compensation based neural network system (e.g., neural network system 411 and / or neural network system 511). In Figure 6, the term x denotes a frame for which motion may be estimated (corresponding to a previous frame when a current frame is being coded), the term f denotes motion estimation, and the term y denotes a motion compensated frame, which may be calculated as y = f(x).

[0251]

[0160] One type of motion estimation that may be implemented is a block-based motion estimation technique 602. Block-based motion estimation may be implemented on a block-by-block basis. For example, for each block in a current frame y (to be motion compensated), a motion estimate f defines the location of a corresponding block in a previous frame x. In one illustrative example, the motion estimate f may include a motion vector indicating the displacement (e.g., horizontal and vertical displacement) of a block in the current frame y relative to a corresponding block in the previous frame x. A block from a previous frame x may be determined to correspond to a block in the current frame y by determining the similarity (e.g., similarity of pixel values) between the blocks.

[0252]

[0161] Another type of motion estimation that may be implemented is the optical flow motion estimation technique 604. Optical flow motion estimation may be implemented on a pixel-by-pixel basis. For example, for each pixel in a current frame y, a motion estimate f defines the location of the corresponding pixel in a previous frame x. The motion estimate f for each pixel may include a vector indicating the movement of the pixel between frames. In some cases, an optical flow map (also called a motion vector map) may be generated based on the calculation of optical flow vectors between frames. The optical flow map may include an optical flow vector for each pixel in a frame, where each vector indicates the movement of the pixel between frames. In one illustrative example, the optical flow vector for a pixel may be a displacement vector indicating the movement of the pixel from a first frame to a second frame (e.g., indicating horizontal and vertical displacements, such as x and y displacements). The encoder (e.g., encoder 533) of the motion compensation based neural network system may encode or compress an optical flow vector for each pixel in the current frame y, or may encode or compress an optical flow map when an optical flow map is used. The motion estimation information reconstructed by the decoder (e.g., decoder 538) of the motion compensation based neural network system

[0253]

number

[0254] may contain a group of reconstructed optical flow vectors (eg, one for each pixel in the frame) or a reconstructed optical flow map.

[0255]

[0162] In some cases, the optical flow map may include vectors for some pixels in a frame. For example, dense optical flow may be calculated between adjacent frames to generate optical flow vectors for each pixel in the frame that may be included in the dense optical flow map. In another example, Lucas-Kanade optical flow may be calculated between adjacent frames to generate optical flow vectors for some or all pixels in the frame that may be included in the optical flow map. Any other suitable type of optical flow technique or algorithm may be used to determine optical flow between frames. Each optical flow map may include a 2D vector field, where each vector is a displacement vector that indicates the movement of a point from a first frame to a second frame.

[0256] As described above, optical flow vectors or maps are calculated between adjacent frames in a sequence of frames (e.g., adjacent frames x t and x t-1 The two adjacent frames may include two directly adjacent frames that are consecutively captured frames or two frames that are a certain distance apart in a series of frames (e.g., within 2 frames of each other, within 3 frames of each other, or any other suitable distance). t-1 From frame x t The optical flow to Ox t-1, ,x t =dof(x t-1, ,x t ), where dof is the dense optical flow. Any suitable optical flow process may be used to generate the optical flow map. In one illustrative example, t-1 Pixel I(x,y,t) in the next frame x tThe pixel is the same and the intensity is the same in frame x. t-1 and the next frame x t , the following equation can be assumed:

[0257]

number

[0258]

[0164] By taking a Taylor series approximation of the right hand side of equation (2) above, then eliminating common terms and dividing by Δt, the optical flow equation can be derived as follows:

[0259]

number

[0260]

[0165] Here,

[0261]

number

[0262]

[0166] Using the optical flow equation (3), the image gradient f x and f y is (f tThe terms u and v are the x and y components of the velocity or optical flow of I(x,y,t) and are unknown. Estimation techniques may be needed in some cases when the optical flow equation may not be solved with two unknown variables. Any suitable estimation technique may be used to estimate the optical flow. Examples of such estimation techniques include differential methods (e.g., Lucas-Kanade estimation, Horn-Schunck estimation, Buxton-Buxton estimation, or other suitable differential methods), phase correlation, block-based methods, or other suitable estimation techniques. For example, Lucas-Kanade assumes that the optical flow (displacement of image pixels) is small and approximately constant in a local neighborhood of pixel I, and solves a basic optical flow equation for all pixels in that neighborhood using a least squares method.

[0263]

[0167] Another type of motion estimation that can be implemented is the dynamic convolution motion estimation technique 606 (also called adaptive convolution). Dynamic convolution motion estimation can be implemented on a pixel-by-pixel basis. For example, for each pixel in the current frame y, the motion estimation f defines a linear combination of a neighborhood of pixels in the previous frame x. As shown in FIG. 6, for a pixel at a pixel location in the current frame y, a kernel or matrix is ​​defined around a neighborhood of pixels surrounding the pixel at the same pixel location in the previous frame x. The exemplary dynamic convolution motion estimation of FIG. 6 is implemented for a pixel at location (2,7) (corresponding to (row number, column number)) in the current frame y, where the top-left pixel location in the current frame y is at location (0,0). A kernel is defined around the pixel at location (2,7) in the previous frame x. For example, as shown, a kernel of size 5×5 is applied to a neighborhood of pixels surrounding the pixel at location (2,7) in the previous frame x.

[0264]

[0168] Each entry in the kernel or matrix may contain a number (a set of 5x5 numbers in the example of Figure 6). The set of numbers in the kernel may describe how to combine pixels from a neighborhood of pixels surrounding a pixel at a pixel location in a previous frame x that corresponds to the same pixel location of the pixel being reconstructed in the newly reconstructed frame (current frame y). An encoder (e.g., encoder 533) of the motion compensation based neural network system may compress a representation of the numbers from the kernel (e.g., 25 numbers from the 5x5 kernel from Figure 6) for each pixel in the current frame y. Motion estimation information reconstructed by a decoder (e.g., decoder 538) of the motion compensation based neural network system

[0265]

number

[0266] may include a pixel-by-pixel reconstructed kernel that may be used to perform data convolution to warp a previously reconstructed frame.

[0267]

[0169] Figure 7 illustrates an example of a neural network system 511 with an additional post-warping network 574 for performing post-warping extensions. In some cases, after the warping engine 567 performs the warping and the adder engine 569 adds the residual, there may still be room to improve the reconstructed frame. For example, the reconstructed frame

[0268]

number

[0269] is the original frame after warping (by warping engine 567) and adding the residual (by adder engine 569).

[0270]

number

[0271] The reconstructed frames may have visual artifacts that were not present in the original image. The postwarping network 574 may include neural network components (e.g., one or more convolutional layers) that are used to further enhance or refine the reconstructed frames (e.g., by removing artifacts).

[0272]

[0170] The post-warping network 574 may be applied after the warping engine 567 warps a previously reconstructed frame. For example, as shown in Figure 7, at time step t, the warped frame (after warping has been applied) and the reconstructed residual

[0273]

number

[0274] and are input to a post-warping network 574. The post-warping network generates a reconstructed frame,

[0275]

number

[0276] In some examples, the post-warping network 574 is trained independently of the neural networks of the remaining neural network system 511. In some examples, the post-warping network 574 is not trained independently of the remaining neural network system 511, in which case the entire neural network system 511 is trained end-to-end. For example, a clean or uncompressed frame may be input to the neural network of the neural network system 511, and the network outputs a reconstructed frame. Backpropagation may be used to assess the error or loss associated with the reconstructed frame (compared to the input frame), and parameters of the neural network system 511 may be tuned as described above.

[0277] 8 and 9 show an example of a neural network system 511 with an additional motion estimation network 575. The motion estimation network 575 can provide extended learning of motion estimation. As shown in FIG. 8 and 9, the previous input frame x t and the motion estimation determined at the encoder side by the motion estimation network 575

[0278]

number

[0279] and (the warped reconstructed frame

[0280]

number

[0281] (denoted as

[0282]

number

[0283] Additional input variables including the previous reconstructed frame warped by

[0284]

[0172] As mentioned above, in some implementations, the neural network system 511 is trained in an end-to-end manner. In some cases, the neural network system 511 may have problems during training when attempting to calculate some parameters. For example, problems may occur when the neural network tries to determine motion estimation parameters (e.g., optical flow) and warp a previously reconstructed frame using those motion estimation parameters. In another example, problems may occur when estimating a residual and adding the warped frame to the residual. Such problems may occur because the neural network system 511 tries to obtain the best possible reconstruction regardless of how the reconstruction is obtained. For example, in attempting to generate a reconstructed frame that is as close as possible to the original frame, the neural network system 511 may not properly learn the motion estimation parameters (e.g., optical flow) and may rely only on the residual to generate the reconstructed frame. In some cases, the output of the reconstructed optical flow may even be a blank tensor or other data structure with no data. However, a large amount of redundancy in a video may come from such motion data (eg, from optical flow between frames), in which case learning motion estimation parameters may be important.

[0285]

[0173] The motion estimation network 575 can help the neural network system 511 learn the motion estimation parameters by providing a motion estimation neural network on the encoder side of the neural network system 511. For example, the motion estimation network 575 can include a separate neural network that estimates motion estimation parameters (e.g., optical flow vectors, optical flow maps, block-based motion information, per-pixel kernels used to perform data convolution, etc.) and provides those motion estimation parameters to the encoder 533. By determining the encoder-side motion estimation parameters of the neural network system 511, an actual set of motion estimation parameters is provided to the encoder 533, causing the encoder 533 to consider motion estimation (e.g., optical flow).

[0286] In some examples, the motion estimation network 575 may be trained independently of the rest of the neural network system 511, or the entire neural network system 511 (including the motion estimation network 575) may be trained together in an end-to-end manner. Various loss terms that take the motion estimation network 575 into account may include:

[0287]

number

[0288] In some implementations, the motion estimation network 575 may be trained using unsupervised learning (without using ground truth output). The Loss1 and Loss2 terms may be used on the encoder side, and the Loss3 and Loss4 terms may be used on the decoder side. In one illustrative example using optical flow as an example of motion estimation, the loss of two consecutive frames x t and x t+1 Then, the objective of the motion estimation network 575 is to estimate the motion of two frames x tand x t+1 The motion estimation network 575 outputs estimated optical flow parameters, which are t A reconstructed version of

[0289]

number

[0290] The loss (e.g., Loss2 above) is used to warp the warped

[0291]

number

[0292] and the original frame x t The loss may then be used to tune the weights (and possibly other parameters) using the backpropagation technique described above. In some cases, the loss term Loss1 may be used on the encoder side, rather than Loss2.

[0293] The same concept can be used at the decoder side using Loss3 or Loss4 to train the decoder 538 to recover motion estimation parameters (e.g., optical flow) estimated at the encoder side. For example, the decoder 538 may use Loss3 or Loss4 to train the decoder 538 to recover motion estimation parameters (e.g., optical flow) estimated at the encoder side. t and the previous frame x t-1 (when Loss3 is used) or the previously reconstructed frame

[0294]

number

[0295] The motion (e.g., optical flow) between two consecutive frames, including either x (when Loss4 is used) or x (when Loss5 is used), can be calculated or estimated. In this way, uncompressed frames can be used to estimate the motion estimation parameters. The obtained motion estimation parameters are t In an attempt to match the previous frame (x t-1 or

[0296]

number

[0297] ), the output of the warping is the input frame x t It is desirable for the

[0298]

[0177] Warped frame and input frame x t The loss between x and the actual previous frame x can then be calculated using either Loss3 or Loss4. As mentioned above, Loss3 is the loss between the actual previous frame x and the t-1 Loss4 includes the use of the previously reconstructed frame

[0299]

number

[0300] (Previous frame x t-1 The actual previous frame x from time step t-1 is used. t-1 Loss4 on the decoder side (Loss2 on the encoder side) may be used in situations where Loss is not available to the neural network system 511 at the next time step t. Backpropagation may be used to assess the loss and adjust or tune the parameters of the neural network system 511. Such training helps the decoder 538 learn the motion estimation parameters.

[0301] In some cases, (the actual previous frame x t-1 Loss terms Loss1 and Loss3 or loss terms Loss2 and Loss4 are activated at different stages of training (depending on whether Loss1 and Loss3 are available at time step t). For example, Loss2 at the encoder side and Loss4 at the decoder side may be activated and used initially when training of the neural network system 511 begins and is used until a certain point, after which the neural network system 511 is trained using Loss0 as the only loss term (as shown in FIG. 5A and above) (loss terms Loss1-Loss4 are deactivated). By initially using loss terms Loss1 and Loss3 or Loss2 and Loss4 during training, the neural network system 511 is forced to learn motion estimation parameters (e.g., optical flow vectors or maps). Once the neural network system 511 has learned the motion estimation parameters sufficiently, it can retain that information and will maintain how to estimate the motion estimation parameters in further training iterations and during inference (when the trained network is deployed for operation on real video data). Therefore, the additional loss terms Loss1 through Loss4 may be deactivated once the motion estimation parameters are learned by the neural network system 511, and Loss0 may continue to be used to further train the neural network system 511.

[0302]

[0179] Figure 10 illustrates an example of different recurrent layer configurations of the neural network system 511, including single-level recurrent configurations and multi-level recurrent configurations. The encoder 533 and the decoder 538 are shown as different layers separated by a bottleneck 1034. The bottleneck 1034 corresponds to the latent representation quantized using the quantizer 534. In the single-level recurrent configuration, the encoder 533 includes a forward layer, and the decoder includes four forward layers and a single recurrent layer 1071. The forward layer of the encoder 533 may include one or more of the neural network layers shown in Figure 5B, such as one or more convolutional layers, normalization layers, nonlinear layers, residual blocks, or other neural network layers. The forward layer 1070a, the forward layer 1070b, and the forward layer 1070c correspond to one or more pre-recurrent decoder layers 570 shown in Figure 5A. In one illustrative example, the forward layers 1070a, 1070b, and 1070c are (ConvGRU c:C 3 10 may include all layers of the decoder 538 of FIG. 5B (shown in FIG. 5B as layer 579) that occur before the recurrent layer (shown in FIG. 5B as k:3 s:1). The recurrent layer 1071 of FIG. 10 corresponds to one or more of the recurrent layers 571 shown in FIG. 5A. An example of a recurrent layer 1071 is the (setting c:C 3 5A. The forward layer 1073 may be a convolutional layer (shown as Conv c:5k:5 s:1) of the decoder 538 of FIG. 5B. As shown, the recurrent state of the recurrent layer 1071 from time step t is provided to the recurrent layer 1071 and to the encoder 533 at the next time step t+1. The forward layer 1073 corresponds to one or more post-recurrent layers 572 shown in FIG. 5A. In one illustrative example, the forward layer 1073 may be a convolutional layer (shown as Conv c:5k:5 s:1) of the decoder 538 of FIG. 5B.

[0303]

[0180] A multi-level recurrent configuration includes the same forward layer as the single recurrent configuration, but includes multiple levels of recurrent layers in the decoder 538. Each of the different recurrent layers maintains a separate recurrent state (also called a hidden state), with each recurrent state capturing features of a different level up to that point in the neural network system 511. As shown in Figure 10, in addition to being fed back to the recurrent layer of the decoder 538, the recurrent states are fed back from the recurrent layer of the decoder 538 to a mirror location in the encoder 533. Multiple recurrent states may enable the encoder 533 and the decoder 538 to utilize additional information to perform more efficient video coding.

[0304]

[0181] Figure 11 illustrates an example of a neural network system 511 with an additional warping engine 576. Various components of the neural network system 511 are omitted from Figure 11 for illustrative purposes (e.g., quantizer 534, one or more recurrent pre-decoder layers 570, etc.), but those skilled in the art will understand that those components are present in the neural network system 511. The warping engine 576 may be used to warp previous recurrent states (or hidden states) from a previous time step (e.g., time step t) of one or more recurrent layers 571 before using the recurrent states in a current time step (e.g., time step t+1). The warping engine 576 may use motion estimation parameters available to the encoder 533 and the decoder 538 to warp the recurrent states to match the locations of pixels in the current frame.

[0305]

[0182] Warping of previous recurrent states before using them in the current time step can be useful for video. For example, an object may move in time across video frames, and such motion may be modeled by the motion estimation techniques described above (e.g., using optical flow, block-based motion estimation, etc.). When the recurrent states of one or more recurrent layers 571 of the decoder 538 are directly fed back to the decoder 538 and the encoder 533 at a later time step, the motion of the object is not modeled in the hidden state (e.g., because motion compensation is performed after the recurrent layers, as shown in FIG. 5A). The recurrent state represents the input frame (e.g., it contains a feature representation of the input image), in which case a movement of a pixel in the input image must result in a similar movement in the feature space of the recurrent state. Thus, by warping the recurrent state based on motion estimation, each movement estimated from the input frame is transferred to a corresponding movement in the recurrent state, making the recurrent state match the location of the pixel in the current frame.

[0306]

[0183] Referring to Figure 11, the warping engine 576 at the current time step is used to warp the previous recurrent or hidden state h of the recurrent layer of the decoder 538 from the previous time step. The previous hidden state is warped using motion estimation parameters (e.g., optical flow) estimated at the current time step before being reused in the recurrent layer at the current time step. For example, the warping engine 576 warps the motion estimation parameters estimated at time step t+1,

[0307]

number

[0308] Using the previous recurrent state h from time step t tThe warped recurrent state can then be reused in the recurrent layer at the current time step t.

[0309] In some implementations, an additional recurrent layer may be provided before the warping engine 576. The additional recurrent layer may be used to estimate the motion estimation parameters for the motion of the current time step t+1.

[0310]

number

[0311] (e.g., optical flow), which can be used to reconstruct the previous recurrent state h from time step t. t The warped previous recursive state

[0312]

number

[0313] may then be input to a recursive layer at the current time step t. In some implementations, an iterative operation may first be performed to determine motion estimation parameters (e.g., optical flow) at the current time step t+1, which is based on the previous recursive state h from time step t. t The warped previous recursive state

[0314]

number

[0315] may then be input to a recursive layer at the current time step t. For example, a forward pass may be performed to restore or reconstruct the motion estimation parameters, and then the operation may return to the recursive layer to warp the previous recursive state. In some implementations, the warped previously reconstructed frame (from time step t-1) as feedback to the recursive layer at the current time step (t).

[0316]

[0185] Figure 12 illustrates an example of a neural network system 511 with additional masking functionality. As shown in Figure 12, one or more post-recursion layers 573 apply the reconstructed motion estimation data

[0317]

number

[0318] and the reconstructed residual data

[0319]

number

[0320] plus an element-wise mask

[0321]

number

[0322] The warping engine 567 may output a previously reconstructed frame, similar to that described above with respect to FIG.

[0323]

number

[0324] The previous reconstructed frame warped by warping

[0325]

number

[0326] It is possible to generate a mask.

[0327]

number

[0328] can be a pixel-wise mask having a value for each pixel in the video frame.

[0329]

number

[0330] is the reconstructed frame

[0331]

number

[0332] When reconstructing, we use the warped previous reconstructed frame

[0333]

number

[0334] and residual

[0335]

number

[0336] For example, in some implementations, a mask

[0337]

number

[0338] may be applied according to the following formula:

[0339]

number

[0340] According to equation (4), the mask

[0341]

number

[0342] As the value of increases, the reconstructed frame

[0343]

number

[0344] Above is the reconstructed frame before warping.

[0345]

number

[0346] Increased effect of restructured frames

[0347]

number

[0348] Residual data above

[0349]

number

[0350] The effect of the mask is reduced.

[0351]

number

[0352] As the value of decreases, the warped reconstructed frame

[0353]

number

[0354] The effect of is reduced, and the residual data

[0355]

number

[0356] In some implementations, the effect of

[0357]

number

[0358] may be limited to a range of values ​​between 0 and 1 ([0,1] inclusive). In one illustrative example, if the mask has a value of 0 for a pixel, the value in the next frame for that pixel will be taken entirely from the residual.

[0359] In some examples, the mask

[0360]

number

[0361] may be an occlusion mask that masks out portions of the current video frame that were not present in the previous video frame. Such scenarios may occur, for example, when portions of a scene are temporarily occluded, when an object enters the scene, etc. Masking of such unavailable portions of the frame may be performed because motion information (and thus warping) for those portions is unavailable. For example, if pixels are unavailable in the previous frame, motion estimation between the current frame and the previous frame may not be performed for pixels of the current frame. In such an example, the portions of the video frame that contain those pixels may not be warped. In such a case, reconstruction of the current frame may require additional processing, depending on the amount of occlusion, using residual data.

[0362]

number

[0363] The determination may be made by relying heavily (or in some cases entirely) on the

[0364]

[0188] Figure 13 illustrates an example of a neural network system 511 with additional reconstruction refinement capabilities. Using the configuration illustrated in Figure 13, the neural network system 511 can reconstruct each video frame in two time steps of operation of the neural network system 511. For example, for each reconstructed video frame (e.g.,

[0365]

number

[0366] ) may be further refined at a subsequent time step (e.g., time step t). In such a case, the previously reconstructed video frame

[0367]

number

[0368] can serve two roles: for example, at the current time step t, the reconstruction of the current time step (as is also done in the configuration shown in FIG. 5A).

[0369]

number

[0370] The previously reconstructed video frames

[0371]

number

[0372] is warped and the residual

[0373]

number

[0374] is added. Furthermore, the residual

[0375]

number

[0376] Also output is the previously reconstructed video frame

[0377]

number

[0378] is added to the previously reconstructed video frame

[0379]

number

[0380] An improved reconstructed frame having improved quality compared to

[0381]

number

[0382] This results in

[0383] As shown in FIG. 13, at each time step, the previous ground truth (uncompressed) frame x is multiplied by the previous reconstructed frame

[0384]

number

[0385] For example, at time step t, the previous ground truth frame x t-1 but the previous reconstructed frame

[0386]

number

[0387] As shown, at time step t, one or more post-recursive layers 573 calculate the residual

[0388]

number

[0389] In addition to the output residual

[0390]

number

[0391] Gives the residual

[0392]

number

[0393] is the previous ground truth frame x t-1 The improved reconstructed frame is generated based on

[0394]

number

[0395] To generate

[0396]

number

[0397] For example, an improved reconstruction may be generated as follows:

[0398]

number

[0399]

[0190] Improved Reconstructed Frame

[0400]

number

[0401] may be used as the final reconstruction for the previous frame at time step t-1. In such a case, the system 500 delays the output video sequence by one time step operation of the neural network system 511. It may be beneficial in terms of coding efficiency (e.g., rate distortion) for the neural network system 511 to be able to improve the reconstructed frame at the next time step by knowing the errors made in the first initial unimproved reconstruction. Such a configuration allows any errors made in the previous time step to be compensated at the next time step, which may benefit rate distortion optimization.

[0402]

[0191] Figure 14 illustrates an example of using one or more distortion maps as input to the encoder 533 of the neural network system 511. Two exemplary distortion maps are shown in Figure 14, including distortion map 577 and distortion map 578. Distortion map 577 is a representation of the distortion of a current source frame x from a current time step t. t and the previous reconstruction from the previous time step t-1

[0403]

number

[0404] The distortion map 578 contains the distortion between the current source frame x t The reconstructed video frame before warping with

[0405]

number

[0406] As mentioned above, the previously reconstructed video frame

[0407]

number

[0408] may be warped according to a motion estimate such as optical flow. In such cases, the optical flow may be estimated at the encoder side, such as by using a flow net, which is a neural network architecture that learns optical flow using a convolutional network.

[0409]

[0192] The distortions included in the distortion map may be distortions (or errors or losses) used as loss functions (e.g., Loss0, Loss1, Loss2, Loss3, and / or Loss4 described above) for training the neural network system 511. The distortions may be based on any type of distortion, such as multi-scale structural similarity index (MS-SSIM), pixel-wise mean square error (MSE), among others. The distortion map may also be referred to as a perceptual distortion map.

[0410]

[0193] By explicitly computing the distortion map (outside the neural network system 511) and providing the distortion map as an input to the encoder 533, the encoder 533 can be guided as to which parts of the image are contributing to perceptual distortion and therefore where it needs to allocate more information. For example, the encoder 533 can focus the bit rate on areas of the image that contribute most to distortion. Such benefits can be an extension for some distortion maps, such as perceptual distortion maps (e.g., MS-SSIM), that are important to learn by neural networks.

[0411]

[0194] Figure 15 illustrates an example of a neural network system 511 that uses an additional linear neural network layer that allows for multiplicative interactions in the encoder 533. The convolutional neural network architecture applied in a default manner does not allow for multiplicative interactions. Multiplicative interactions are when the calculations being performed on the inputs depend on the inputs themselves. For example, traditional convolutional neural networks learn weights during training. The weights are the same and are applied to every input regardless of the input values ​​that are input to the neural network.

[0412]

[0195] Referring to FIG. 15, the multiplicative interaction is made available in the encoder 533 by applying the linear layer 581 of the encoder 533 (shown in the linear layer (1) of FIG. 15) independently at each spatial location of the input. The weights of the linear layer 581 are determined by the weights 583 output by the linear layer 582 (shown in the linear layer (2) of FIG. 15). For example, the weights 583 can be used as the weights for the linear layer 581. The same inputs 580 are applied to the linear layer 581 and the linear layer 582. Using the linear layer 582 to define the weights of the linear layer 581 allows the calculations performed by the linear layer 581 to depend on the value of the input 580 itself (and therefore the calculations applied to the input 580 depend on the input itself). For example, the neural network system 511 can learn to perform different calculations in regions where the distortion between the previous reconstructed frame and the current frame is negligible compared to regions where the distortion is significant.

[0413]

[0196] In some implementations, the configuration of Fig. 15 may be used in combination with the configuration shown in Fig. 16. For example, one or more distortion maps may be input to the linear layer 582 in addition to the input 580, which may help improve the calculation of the weights 583. In one example, the weights at different spatial locations may be set differently according to the values ​​of the distortion at the different spatial locations, as indicated by the distortion maps.

[0414]

[0197] The recurrent-based neural network system (e.g., system 400 and / or system 500) and related techniques described herein for video compression provide a video coding system that can perform at state-of-the-art levels in terms of high quality bits per pixel (bpp) values. For example, as shown in the graphs provided in Figures 16 and 17, such techniques outperform current industry standard codecs including HEVC and AVC in the range of >0.14 bpp for 1080p resolution (which is the bpp region of practical interest) (e.g., Netflix streams 1080p video using approximately 0.09-0.12 bpp). "RecurrentAE" in Figures 16 and 17 refers to results using the recurrent-based neural network system described herein, and "RDAE" in Figures 16 and 17 refers to another machine learning based video compression technique.

[0415]

[0198] The recursion-based neural network system described herein (e.g., system 400 and / or system 500) provides low latency operation. For example, the system allows for generation and transmission of video frame-by-frame rather than in batches of several frames (e.g., in Groups of Pictures (GoPs)), thus eliminating the need to wait for accumulation of frames to be transmitted. Furthermore, the system provides the ability to adjust GoP size (i.e., how often to transmit full frames rather than just residual information) at run-time depending on the characteristics of the video, thus leading to a better operating point in the rate-distortion space. Various other benefits are also provided by the recursion-based neural network system described herein.

[0416]

[0199] Figure 18 is a flow chart illustrating an example of a process 1800 for processing video using one or more of the recursion-based machine learning techniques described herein. In block 1802, the process 1800 includes obtaining, by an encoder portion of the neural network system, an input video frame for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from a previous time step operation of the neural network system, and at least one recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from a previous time step operation. In some cases, the recurrent state data includes one or more values ​​of nodes of at least one recurrent layer. In some examples, other input data, such as a previously reconstructed video frame generated by a previous time step operation of the neural network system, may be obtained by the encoder portion, as shown in Figure 5A.

[0417] At block 1804, the process 1800 includes generating, by an encoder portion of the neural network system, output data for a current time step operation of the neural network system. The output data is generated based at least in part on the input video frames, reconstructed motion estimation data from a previous time step operation, reconstructed residual data from a previous time step operation, and at least one recurrent state data from a previous time step operation. The output data may be generated based on other input data in some cases, such as previously reconstructed video frames generated by a previous time step operation of the neural network system, as shown in FIG.

[0418] At 1806, the process 1800 includes obtaining, by a decoder portion of the neural network system, output data for the operation of the current time step and at least one recurrent state data from at least one recurrent layer of the decoder portion from the operation of the previous time step. For example, as shown in FIG. 5A, for the operation of the current time step t+1, the decoder 538 may obtain output data from the quantizer 534, and the recurrent layer of the decoder 538 may obtain recurrent state data h for itself from the operation of the previous time step t. t In some cases, the decoder portion of the neural network system includes multiple recurrent layers, each of which provides respective recurrent state data.

[0419]

[0202] At block 1808, the process 1800 includes generating a reconstructed video frame for the operation of the current time step based at least in part on the output data and the at least one recursive state data from the operation of the previous time step. The reconstructed video frame represents the input video frame. For example, the reconstructed video frame is a reconstructed version of the input video frame. The reconstructed video frame may be output for display, storage, and / or transmission. For example, in some examples, the process 1800 includes storing the output data in a storage medium. In some examples, the process 1800 includes storing the reconstructed video frame in a storage medium. In some examples, the process 1800 includes sending the reconstructed video frame via a transmission medium to at least one device. In some cases, the reconstructed video frame may be further processed (e.g., to remove visual artifacts due to the reconstruction process, or for other processing) before being output for display, storage, and / or transmission.

[0420]

[0203] In some examples, process 1800 includes determining, by a decoder portion of the neural network system, motion estimation data for a current time step operation based at least in part on the output data and at least one recurrent state data from a previous time step operation. In such examples, process 1800 may include obtaining a previously reconstructed video frame generated during the previous time step operation, and generating a warped reconstructed video frame for the current time step operation by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step.

[0421]

[0204] In some examples, process 1800 includes determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and at least one recurrent state data from a previous time step operation. In such examples, process 1800 may include generating a reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0422]

[0205] In some examples, the process 1800 includes determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and at least one recurrent state data from a previous time step operation. In such examples, the process 1800 may include processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step. In some cases, a reconstructed video frame is generated for the current time step operation based on processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0423]

[0206] In some examples, the process 1800 includes determining, by a motion estimation network of the neural network system, motion estimation data for a current time step operation based at least in part on an input video frame and at least one of a previously reconstructed video frame from a previous time step operation or a previous input video frame from a previous time step operation. In such examples, the process 1800 can include generating a warped reconstructed video frame for the current time step by modifying one or more pixels of at least one of the previously reconstructed video frame or the previous input video frame using the motion estimation data determined for the current time step. The process 1800 can include sending the warped reconstructed video frame and the motion estimation data determined for the current time step operation to an encoder portion of the neural network system. In some cases, the output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the warped reconstructed video frame and the motion estimation data determined for the current time step operation.

[0424]

[0207] In some examples, the process 1800 includes training the neural network system using a first loss function for one or more training iterations. The first loss function determines a loss between one or more input video frames and one or more warped reconstructed video frames. For example, the first loss function may include any of the loss functions Loss1, Loss2, Loss3, and / or Loss4 described above. In one illustrative example, Loss1 may be used to train the encoder 533, and Loss3 may be used to train the decoder 538. In another illustrative example, Loss2 may be used to train the encoder 533, and Loss4 may be used to train the decoder 538. In such examples, the process 1800 may include deactivating the first loss function (e.g., deactivating Loss1, Loss2, Loss3, and / or Loss4). For example, the first loss function may be deactivated in response to determining that the neural network system is trained to learn the motion estimation parameters, as described above with respect to Figures 8 and 9. In such an example, the process 1800 may include training the neural network system using the second loss function for one or more subsequent training iterations performed after the one or more training iterations. The second loss function determines a loss between the at least one input video frame and the at least one reconstructed video frame. For example, the second loss function may include Loss0 described above.

[0425] In some examples, the process 1800 includes generating warped recursive state data for the current time step operation by modifying at least one recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or the previous time step operation. In such examples, the reconstructed video frame generated for the current time step operation is generated based at least in part on the warped recursive state data.

[0426]

[0209] In some examples, the process 1800 includes generating, by the decoder portion, a reconstructed mask. The reconstructed mask includes one or more values ​​indicative of the presence of data in a previously reconstructed video frame generated during a previous time step operation. In such examples, the process 1800 includes generating a reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame generated for the current time step operation, and the reconstructed residual data determined for the current time step operation. The reconstructed mask weights the contributions of the warped reconstructed video frame and the reconstructed residual data. In one illustrative example, as described above with respect to FIG. 12, the mask is calculated based on Equation (4):

[0427]

number

[0428] It can be implemented using

[0429]

[0210] In some examples, the process 1800 includes obtaining a previously reconstructed video frame generated during a previous time step operation. In such examples, the process 1800 includes generating, during a current time step operation, a warped reconstructed video frame for the current time step operation by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step. In such examples, the process 1800 may include determining, by the decoder portion during the current time step operation, first reconstructed residual data for the current time step operation. In such examples, the process 1800 may include generating a reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step. The process 1800 may include determining, by the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation. In such an example, the process 1800 may include modifying the previously reconstructed video frame during the current time step operation using the second reconstructed residual data and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation. For example, as described above with respect to FIG.

[0430]

number

[0431] is the reconstructed video frame at time step t+1

[0432]

number

[0433] In such an example, the reconstructed video frame may be output (e.g., for display, storage, transmission, and / or other operation) at time step t+1 instead of outputting the reconstructed video frame

[0434]

number

[0435] can be output for modification at the next time step t+2.

[0436]

[0211] In some examples, the process 1800 includes obtaining, by an encoder portion of the neural network system, one or more distortion maps, and generating, by the encoder portion of the neural network system, output data based at least in part on the one or more distortion maps. In some cases, as described above with respect to FIG. 14, the one or more distortion maps may include a first distortion map indicating a distortion between an input video frame for a current time step operation and a previously reconstructed video frame from a previous time step operation. In some cases, as described above with respect to FIG. 14, the one or more distortion maps may include a second distortion map indicating a distortion between an input video frame for a current time step operation and a warped reconstructed video frame (wherein the warped reconstructed video frame may be generated by modifying one or more pixels of a previously reconstructed video frame from a previous time step operation, as described above).

[0437]

[0212] In some examples, the process 1800 includes processing input data by a first layer of an encoder portion of the neural network system. In such examples, the process 1800 includes determining, by the first layer of the encoder portion, a plurality of weight values ​​for the first layer of the encoder portion. In such examples, the process 1800 includes setting weights of a second layer of the encoder portion of the neural network system to the plurality of weight values, and processing the input data by the second layer of the encoder portion using the weights set to the plurality of weight values. For example, as described above with respect to FIG. 15, the weights of the linear layer 581 may be determined by the weights 583 output by the linear layer 582, which may allow the calculations performed by the linear layer 581 to depend on the value of the input 580 itself.

[0438]

[0213] Figure 19 is a flow chart illustrating another example of a process 1900 for processing video using one or more of the recursion-based machine learning techniques described herein. At block 1902, the process 1900 includes obtaining, by a decoder portion of the neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system. Using the example of Figure 5A for illustration, for an operation at a current time step t+1, the decoder 538 may obtain output data from the quantizer 534, and the recurrent layer of the decoder 538 may obtain recurrent state data h for itself from an operation at a previous time step t. t In some cases, the decoder portion of the neural network system includes multiple recurrent layers, each of which provides respective recurrent state data.

[0439]

[0214] At block 1904, the process 1900 includes determining, by a decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation. At block 1906, the process 1900 includes obtaining a previously reconstructed video frame generated during the previous time step operation. At block 1908, the process 1900 includes generating a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step.

[0440]

[0215] At block 1910, the process 1900 includes generating a reconstructed video frame for the operation of the current time step of the operation based at least in part on the warped reconstructed video frame. The reconstructed video frame represents an input video frame processed by an encoder portion of the neural network system or an encoder of another neural network system. For example, the reconstructed video frame is a reconstructed version of the input video frame. The reconstructed video frame may be output for display, storage, and / or transmission. For example, in some examples, the process 1800 includes storing the output data in a storage medium. In some examples, the process 1800 includes storing the reconstructed video frame in a storage medium. In some examples, the process 1800 includes sending the reconstructed video frame via a transmission medium to at least one device. In some cases, the reconstructed video frame may be further processed (e.g., to remove visual artifacts due to the reconstruction process or for other processing) before being output for display, storage, and / or transmission.

[0441]

[0216] In some examples, the process 1900 includes determining, by a decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation. In such examples, the process 1900 may include generating a reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0442]

[0217] In some examples, the process 1900 includes determining, by the decoder portion during the current time step operation, reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion of the neural network system during the current time step operation. In such examples, the process 1900 may include modifying the previously reconstructed video frame during the current time step operation using the reconstructed residual data from the previous time step operation. The process 1900 may include outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation.

[0443]

[0218] In some examples, the process 1900 includes determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation. In such examples, the process 1900 may include processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step. The process 1900 may include generating a reconstructed video frame for the current time step operation based at least in part on the processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0444]

[0219] In some examples, process 1900 includes generating warped recursive state data for the current time step operation by at least in part modifying recursive state data from a previous time step operation using the motion estimation data determined for the current time step operation or a motion estimation determined for a previous time step operation. In such examples, process 1900 may include generating a reconstructed video frame for the current time step operation based at least in part on the warped recursive state data.

[0445]

[0220] In some examples, the process 1900 includes generating, by the decoder portion, a reconstructed mask. The reconstructed mask includes one or more values ​​indicative of the presence of data in a previously reconstructed video frame generated during a previous time step operation. In such examples, the process 1900 may include generating a reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame for the current time step operation, and the reconstructed residual data determined for the current time step operation. The reconstructed mask weights the contributions of the warped reconstructed video frame and the reconstructed residual data. In one illustrative example, as described above with respect to FIG. 12, the mask is calculated based on Equation (4):

[0446]

number

[0447] It can be implemented using

[0448]

[0221] In some examples, the process 1900 includes obtaining, by the encoder portion of the neural network system, an input video frame for a current time step operation, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from a previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from a previous time step operation. In some cases, the recurrent state data includes one or more values ​​of nodes of the at least one recurrent layer. In some examples, other input data, such as a previously reconstructed video frame generated by a previous time step operation of the neural network system, may be obtained by the encoder portion, as shown in FIG. 5A. The process 1900 may include generating, by the encoder portion of the neural network system, output data for a current time step operation, the output data being generated based at least in part on the input video frame, the reconstructed motion estimation data from a previous time step operation, the reconstructed residual data from a previous time step operation, and the recurrent state data from a previous time step operation. The output data may be generated based on other input data in some cases, such as previously reconstructed video frames generated by a previous time step operation of the neural network system, as shown in FIG. 5A.

[0449]

[0222] In some examples, the process 1900 includes training the neural network system using a first loss function for one or more training iterations. The first loss function determines a loss between one or more input video frames and one or more warped reconstructed video frames. For example, the first loss function may include any of the loss functions Loss1, Loss2, Loss3, and / or Loss4 described above. In one illustrative example, Loss1 may be used to train the encoder 533, and Loss3 may be used to train the decoder 538. In another illustrative example, Loss2 may be used to train the encoder 533, and Loss4 may be used to train the decoder 538. In such examples, the process 1900 may include deactivating the first loss function (e.g., deactivating Loss1, Loss2, Loss3, and / or Loss4). For example, the first loss function may be deactivated in response to determining that the neural network system is trained to learn the motion estimation parameters, as described above with respect to Figures 8 and 9. In such an example, the process 1900 may include training the neural network system using the second loss function for one or more subsequent training iterations performed after the one or more training iterations. The second loss function determines a loss between the at least one input video frame and the at least one reconstructed video frame. For example, the second loss function may include Loss0 described above.

[0450]

[0223] In some examples, the process 1900 includes obtaining a previously reconstructed video frame generated during a previous time step operation. In such examples, the process 1900 includes generating, during a current time step operation, a warped reconstructed video frame for the current time step operation by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step. In such examples, the process 1900 may include determining, by the decoder portion during the current time step operation, first reconstructed residual data for the current time step operation. In such examples, the process 1900 may include generating a reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step. The process 1900 may include determining, by the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation. In such an example, the process 1900 may include modifying the previously reconstructed video frame during the current time step operation using the second reconstructed residual data and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation. For example, as described above with respect to FIG.

[0451]

number

[0452] is the reconstructed video frame at time step t+1

[0453]

number

[0454] In such an example, instead of outputting the reconstructed video frame t, the reconstructed video frame t may be output (e.g., for display, storage, transmission, and / or other operation) at time step t+1.

[0455]

number

[0456] can be output for modification at the next time step t+2.

[0457] In some examples, the processes described herein (e.g., process 1800, process 1900, and / or other processes described herein) may be performed by a computing device or apparatus, such as a computing device having a computing device architecture 2000 shown in FIG. 20. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, or other device. In one example, process 1800 and / or process 1900 may be performed by a computing device having a computing device architecture 2000 that implements system 400 or system 500 (using any of the architectures shown in FIGS. 5A-15 or any combination thereof). The computing device may include any suitable device, such as an autonomous vehicle, a robotic device, a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device, and / or any other computing device with resource capabilities to perform the processes described herein, including process 1800. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers or combined transmitting and receiving units (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein.In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0458]

[0225] Components of a computing device may be implemented with circuits. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0459]

[0226] Process 1800 and process 1900 are illustrated as logic flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0460]

[0227] Furthermore, the processes described herein (including process 1800, process 1900, and / or other processes described herein) may be performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0461] FIG. 20 illustrates an exemplary computing device architecture 2000 of an exemplary computing device that can implement various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, or other devices. For example, the computing device architecture 2000 can implement the system 400 and / or the system 500 (including any one of the architectures shown in FIGS. 5A-15 or any combination thereof). The components of the computing device architecture 2000 are illustrated in electrical communication with each other using a connection 2005, such as a bus. The exemplary computing device architecture 2000 includes a processing unit (CPU or processor) 2010 and a computing device connection 2005 that couples various computing device components, including a computing device memory 2015, such as a read-only memory (ROM) 2020 and a random access memory (RAM) 2025, to the processor 2010.

[0462]

[0229] The computing device architecture 2000 may include a cache of high speed memory directly connected to the processor 2010, in close proximity to the processor 2010, or integrated as part of the processor 2010. The computing device architecture 2000 may copy data from the memory 2015 and / or storage device 2030 to the cache 2012 for quick access by the processor 2010. In this manner, the cache may provide performance improvements that avoid processor 2010 delays while waiting for data. These and other modules may control or be configured to control the processor 2010 to perform various actions. Other computing device memories 2015 may also be available for use. The memory 2015 may include multiple different types of memory with different performance characteristics. The processor 2010 may include any general purpose processor, as well as hardware or software services, such as service 1 2032, service 2 2034, and service 3 2036 stored in a storage device 2030, configured to control the processor 2010, and special purpose processors where software instructions are built into the processor design. The processor 2010 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0463] To enable user interaction with the computing device architecture 2000, the input device 2045 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 2035 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 2000. The communication interface 2040 can generally govern and manage user input and computing device output. There is no restriction to operating on any particular hardware configuration, and thus the basic features herein can be easily substituted with improved hardware or firmware configurations as they are developed.

[0464]

[0231] The storage device 2030 is a non-volatile memory and may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 2025, a read-only memory (ROM) 2020, and hybrids thereof. The storage device 2030 may include services 2032, 2034, 2036 for controlling the processor 2010. Other hardware or software modules are contemplated. The storage device 2030 may be connected to a computing device connection 2005. In one aspect, a hardware module that performs a particular function may include software components stored in a computer readable medium in relation to the necessary hardware components, such as the processor 2010, the connection 2005, the output device 2035, etc., to perform that function.

[0465] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, a smartphone, a tablet, a laptop computer, a vehicle, a drone, or other device) that includes or is coupled to one or more active depth-sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0466]

[0233] The term "device" is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system, etc.). A device, as used herein, may be any electronic device with one or more parts and may implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Furthermore, the term "system" is not limited to multiple components or a specific embodiment. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0467]

[0234] Specific details are given in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method implemented in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0468]

[0235] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0469]

[0236] The processes and methods according to the above-mentioned examples may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions, or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0470]

[0237] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as flash memory, memories or memory devices, magnetic or optical disks, flash memories, USB devices with non-volatile memory, network-connected storage devices, compact discs (CDs) or digital versatile discs (DVDs), among others, and any suitable combination thereof. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0471] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0472]

[0239] A device implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.

[0473]

[0240] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0474]

[0241] In the above description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in various ways, and the appended claims are to be construed to include such variations, except as limited by the prior art. The various features and aspects of the above-described application examples may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be considered illustrative rather than restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0475]

[0242] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used in this specification may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this specification.

[0476]

[0243] When a component is described as being "configured to" perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor, or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0477]

[0244] The phrase "coupled to" refers to any component that is physically connected, directly or indirectly, to another component and / or any component that is in direct or indirect communication with another component (e.g., connected to the other component over a wired or wireless connection and / or other suitable communication interface).

[0478]

[0245] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or more than one member of the set (in any combination) satisfies the claim. For example, a claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0479]

[0246] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled engineers may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0480]

[0247] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. When implemented in software, the techniques may be realized, at least in part, by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM®), FLASH® memory, magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0481]

[0248] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0482]

[0249] Illustrative examples of the present disclosure include the following:

[0483]

[0250] Aspect 1: A method of processing video data, comprising: obtaining, by an encoder portion of a neural network system, input video frames for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, output data for the current time step operation of the neural network system; the force data being generated based at least in part on an input video frame, reconstructed motion estimation data from a previous time step operation, reconstructed residual data from the previous time step operation, and recurrent state data from the previous time step operation; obtaining, by a decoder portion of the neural network system, output data for a current time step operation and recurrent state data from at least one recurrent layer of the decoder portion from the previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame.

[0484]

[0251] Aspect 2: The method of aspect 1, further comprising: determining, by a decoder portion of the neural network system, motion estimation data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; obtaining a previously reconstructed video frame generated during the previous time step operation; and generating a warped reconstructed video frame for the current time step operation by at least in part modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step.

[0485]

[0252] Aspect 3: The method of aspect 2, further comprising: determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; and generating a reconstructed video frame for the current time step operation at least in part by adding a warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0486]

[0253] Aspect 4: A method as described in any of aspects 2 and 3, further comprising: determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; and processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step, wherein a reconstructed video frame is generated for the current time step operation based on the processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0487]

[0254] Aspect 5: A method according to any of aspects 1 to 4, further comprising: determining, by a motion estimation network of the neural network system, motion estimation data for a current time step operation based at least in part on an input video frame and at least one of a previously reconstructed video frame from a previous time step operation or a previous input video frame from a previous time step operation; generating a warped reconstructed video frame for the current time step at least in part by modifying one or more pixels of at least one of the previously reconstructed video frame or the previous input video frame using the motion estimation data determined for the current time step; sending the warped reconstructed video frame and the motion estimation data determined for the current time step operation to an encoder portion of the neural network system, wherein output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the warped reconstructed video frame and the motion estimation data determined for the current time step operation.

[0488]

[0255] Aspect 6: The method of aspect 5, further comprising: training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame.

[0489]

[0256] Aspect 7: A method according to any one of aspects 1 to 6, wherein the recurrent state data includes one or more values ​​of nodes of at least one recurrent layer.

[0490]

[0257] Aspect 8: A method according to any of aspects 1 to 7, wherein the decoder portion of the neural network system includes a plurality of recurrent layers, each of which provides respective recurrent state data.

[0491]

[0258] Aspect 9: A method as described in any of aspects 1 to 8, further comprising generating warped recursive state data for the current time step operation at least in part by modifying recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or the previous time step operation, wherein the reconstructed video frame generated for the current time step operation is generated at least in part based on the warped recursive state data.

[0492]

[0259] Aspect 10: A method as described in any of aspects 1 to 9, further comprising: generating, by a decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicating the presence of data in a previously reconstructed video frame generated during a previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame generated for the current time step operation, and the reconstructed residual data determined for the current time step operation, the reconstructed mask weighting the contributions of the warped reconstructed video frame and the reconstructed residual data.

[0493]

[0260] Aspect 11: A method according to any of aspects 1 to 10, further comprising: obtaining a previously reconstructed video frame generated during a previous time step operation; generating a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using motion estimation data determined for the current time step during a current time step operation; determining, by a decoder portion during the current time step operation, first reconstructed residual data for the current time step operation; generating a reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; determining, by the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation; modifying the previously reconstructed video frame using the second reconstructed residual data during the current time step operation; and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation.

[0494]

[0261] Aspect 12: A method as described in any of aspects 1 to 11, further comprising obtaining one or more distortion maps by an encoder portion of the neural network system, and generating output data by the encoder portion of the neural network system based at least in part on the one or more distortion maps.

[0495]

[0262] Aspect 13: The method described in aspect 12, wherein the one or more distortion maps include a first distortion map indicating distortion between an input video frame for operation of a current time step and a previously reconstructed video frame from operation of a previous time step.

[0496]

[0263] Aspect 14: A method described in any of aspects 12 and 13, wherein the one or more distortion maps include a second distortion map indicating distortion between the input video frame and the warped reconstructed video frame for the current time step operation, the warped reconstructed video frame being generated at least in part by modifying one or more pixels of a previously reconstructed video frame from a previous time step operation.

[0497]

[0264] Aspect 15: A method described in any of aspects 1 to 14, further comprising: processing input data by a first layer of an encoder portion of the neural network system; determining by the first layer of the encoder portion a plurality of weight values ​​for the first layer of the encoder portion; setting weights of a second layer of the encoder portion of the neural network system to the plurality of weight values; and processing the input data by the second layer of the encoder portion using weights set to the plurality of weight values.

[0498]

[0265] Aspect 16: The method of any one of aspects 1 to 15, further comprising storing the output data in a storage medium.

[0499]

[0266] Aspect 17: The method of any one of aspects 1 to 16, further comprising storing the reconstructed video frames in a storage medium.

[0500]

[0267] Aspect 18: The method of any one of aspects 1 to 17, further comprising sending the reconstructed video frames to at least one device via a transmission medium.

[0501]

[0268] Aspect 19: An apparatus for processing video data. The apparatus includes a memory (eg, implemented in a circuit) and a processor (or multiple processors) coupled to the memory. The processor is configured to: obtain, by an encoder portion of the neural network system, an input video frame for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generate, by the encoder portion of the neural network system, output data for the current time step operation of the neural network system, the output data being generated at least in part based on the input video frame, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; obtain, by a decoder portion of the neural network system, output data for the current time step operation and the recurrent state data from at least one recurrent layer of the decoder portion from the previous time step operation; generate a reconstructed video frame for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame.

[0502]

[0269] Aspect 20: The apparatus described in aspect 19, wherein the processor is configured to: determine, by a decoder portion of the neural network system, motion estimation data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; obtain a previously reconstructed video frame generated during the previous time step operation; and generate a warped reconstructed video frame for the current time step operation by at least in part modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step.

[0503]

[0270] Aspect 21: The apparatus described in aspect 20, wherein the processor is configured to: determine, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; and generate a reconstructed video frame for the current time step operation at least in part by adding a warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0504]

[0271] Aspect 22: An apparatus as described in any of aspects 20 and 21, wherein the processor is configured to: determine, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; process, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step, and a reconstructed video frame is generated for the current time step operation based on the processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0505]

[0272] Aspect 23: An apparatus as described in any of aspects 19 to 22, wherein the processor is configured to: determine, by a motion estimation network of the neural network system, motion estimation data for a current time step operation based at least in part on an input video frame and at least one of a previously reconstructed video frame from a previous time step operation or a previous input video frame from the previous time step operation; generate a warped reconstructed video frame for the current time step at least in part by modifying one or more pixels of at least one of the previously reconstructed video frame or the previous input video frame using the motion estimation data determined for the current time step; send the warped reconstructed video frame and the motion estimation data determined for the current time step operation to an encoder portion of the neural network system, and output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the warped reconstructed video frame and the motion estimation data determined for the current time step operation.

[0506]

[0273] Aspect 24: The apparatus described in aspect 23, wherein the processor is configured to: train the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; train the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame.

[0507]

[0274] Aspect 25: An apparatus described in any of aspects 19 to 24, wherein the recurrent state data includes one or more values ​​of nodes of at least one recurrent layer.

[0508]

[0275] Aspect 26: An apparatus described in any of aspects 19 to 25, wherein the decoder portion of the neural network system includes multiple recurrent layers, each of the multiple recurrent layers providing respective recurrent state data.

[0509]

[0276] Aspect 27: An apparatus described in any of aspects 19 to 26, wherein the processor is configured to generate warped recursive state data for the current time step operation at least in part by modifying recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or the previous time step operation, and the reconstructed video frame generated for the current time step operation is generated at least in part based on the warped recursive state data.

[0510]

[0277] Aspect 28: An apparatus as described in any of aspects 19 to 27, wherein the processor is configured to: generate, by a decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicating the presence of data in a previously reconstructed video frame generated during a previous time step operation; generate a reconstructed video frame for a current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame generated for the current time step operation, and the reconstructed residual data determined for the current time step operation, and the reconstructed mask weights contributions of the warped reconstructed video frame and the reconstructed residual data.

[0511]

[0278] Aspect 29: An apparatus as described in any of aspects 19 to 28, wherein the processor is configured to: obtain a previously reconstructed video frame generated during a previous time step operation; generate a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using motion estimation data determined for the current time step during a current time step operation; determine, by a decoder portion during the current time step operation, first reconstructed residual data for the current time step operation; generate a reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; determine, by the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation; modify, during the current time step operation, the previously reconstructed video frame using the second reconstructed residual data; and output the modified previously reconstructed video frame as a final output video frame for the current time step operation.

[0512]

[0279] Aspect 30: An apparatus described in any of aspects 19 to 29, wherein the processor is configured to: obtain, by an encoder portion of the neural network system, one or more distortion maps; and generate, by the encoder portion of the neural network system, output data based at least in part on the one or more distortion maps.

[0513]

[0280] Aspect 31: The apparatus described in aspect 30, wherein the one or more distortion maps include a first distortion map indicating distortion between an input video frame for a current time step operation and a previously reconstructed video frame from a previous time step operation.

[0514]

[0281] Aspect 32: An apparatus described in any of aspects 30 and 31, wherein the one or more distortion maps include a second distortion map indicating a distortion between the input video frame and the warped reconstructed video frame for a current time step operation, the warped reconstructed video frame being generated at least in part by modifying one or more pixels of a previously reconstructed video frame from a previous time step operation.

[0515]

[0282] Aspect 33: An apparatus described in any of aspects 19 to 32, wherein the processor is configured to: process input data by a first layer of an encoder portion of the neural network system; determine by the first layer of the encoder portion a plurality of weight values ​​for the first layer of the encoder portion; set weights of a second layer of the encoder portion of the neural network system to the plurality of weight values; and process the input data by the second layer of the encoder portion using weights set to the plurality of weight values.

[0516]

[0283] Aspect 34: The apparatus of any of aspects 19 to 33, wherein the processor is configured to store the output data in the memory.

[0517]

[0284] Aspect 35: The apparatus of any of aspects 19 to 34, wherein the processor is configured to store the reconstructed video frame in the memory.

[0518]

[0285] Aspect 36: The apparatus of any of aspects 19 to 35, further comprising a transmitter configured to transmit the reconstructed video frames via a transmission medium to at least one device.

[0519]

[0286] Aspect 37: An apparatus described in any of aspects 19 to 36, wherein the processor includes a neural processing unit (NPU).

[0520]

[0287] Aspect 38: An apparatus described in any of aspects 19 to 37, wherein the apparatus comprises a mobile device.

[0521]

[0288] Aspect 39: An apparatus described in aspects 19 to 37, wherein the apparatus is equipped with an extended reality device.

[0522]

[0289] Aspect 40: The apparatus described in aspects 19 to 37, wherein the apparatus comprises a television.

[0523]

[0290] Aspect 41: A device described in any of aspects 19 to 39, further comprising a display.

[0524]

[0291] Aspect 42: An apparatus described in any of aspects 19 to 41, wherein the apparatus includes a camera configured to capture one or more video frames.

[0525]

[0292] Aspect 43: A method for processing video data comprising: obtaining, by a decoder portion of a neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; determining, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtaining a previously reconstructed video frame generated during the previous time step operation; generating a warped reconstructed video frame for the current time step operation by at least in part modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; and generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame.

[0526]

[0293] Aspect 44: The method described in aspect 43, further comprising: determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; and generating a reconstructed video frame for the current time step operation at least in part by adding a warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0527]

[0294] Aspect 45: A method described in any of aspects 43 and 44, further comprising: determining, by the decoder portion during the current time step operation, reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion of the neural network system during the current time step operation; modifying, during the current time step operation, the previously reconstructed video frame using the reconstructed residual data from the previous time step operation; and outputting the modified previously reconstructed video frame as the final output video frame for the current time step operation.

[0528]

[0295] Aspect 46: A method according to any of aspects 43 to 45, further comprising: determining, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step; and generating a reconstructed video frame for the current time step operation based at least in part on the processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0529]

[0296] Aspect 47: A method described in any of aspects 43 to 46, further comprising: generating warped recursive state data for the current time step operation, at least in part, by modifying recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or motion estimation determined for the previous time step operation; and generating a reconstructed video frame for the current time step operation based at least in part on the warped recursive state data.

[0530]

[0297] Aspect 48: A method as described in any of aspects 43 to 47, further comprising: generating, by a decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicating the presence of data in a previously reconstructed video frame generated during a previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame for the current time step operation, and the reconstructed residual data determined for the current time step operation, the reconstructed mask weighting the contributions of the warped reconstructed video frame and the reconstructed residual data.

[0531]

[0298] Aspect 49: A method according to any of aspects 43 to 48, further comprising: obtaining, by an encoder portion of the neural network system, input video frames for a current time step operation, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, output data for the current time step operation, the output data being generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation.

[0532]

[0299] Aspect 50: An apparatus for processing video data. The apparatus includes a memory (e.g., implemented in a circuit) and a processor (or multiple processors) coupled to the memory. The processor is configured to: obtain, by a decoder portion of the neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; determine, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtain a previously reconstructed video frame generated during the previous time step operation; generate a warped reconstructed video frame for the current time step operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; and generate a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame.

[0533]

[0300] Aspect 51: The apparatus described in aspect 50, wherein the processor is configured to: determine, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; and generate a reconstructed video frame for the current time step operation at least in part by adding a warped reconstructed video frame to the reconstructed residual data determined for the current time step.

[0534]

[0301] Aspect 52: An apparatus as described in any of aspects 50 and 51, wherein the processor is configured to determine, by the decoder portion during the current time step operation, reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion of the neural network system during the current time step operation; modifying, during the current time step operation, the previously reconstructed video frame using the reconstructed residual data from the previous time step operation; and outputting the modified previously reconstructed video frame as the final output video frame for the current time step operation.

[0535]

[0302] Aspect 53: An apparatus described in any of aspects 50 to 52, wherein the processor is configured to: determine, by a decoder portion of the neural network system, reconstructed residual data for a current time step operation based at least in part on the output data and recurrent state data from a previous time step operation; process, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step; and generate a reconstructed video frame for the current time step operation based at least in part on the processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network.

[0536]

[0303] Aspect 54: An apparatus described in any of aspects 50 to 53, wherein the processor is configured to: generate warped recursive state data for the current time step operation, at least in part by modifying recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or motion estimation determined for the previous time step operation; and generate a reconstructed video frame for the current time step operation based at least in part on the warped recursive state data.

[0537]

[0304] Aspect 55: An apparatus described in any of aspects 50 to 54, wherein the processor is configured to: generate, by a decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicating the presence of data in a previously reconstructed video frame generated during a previous time step operation; generate a reconstructed video frame for a current time step operation based at least in part on the reconstructed mask, a warped reconstructed video frame for the current time step operation, and reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data.

[0538]

[0305] Aspect 56: An apparatus as described in any of aspects 50 to 55, wherein the processor is configured to: obtain, by an encoder portion of the neural network system, input video frames for a current time step operation, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generate, by the encoder portion of the neural network system, output data for the current time step operation, the output data generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation.

[0539]

[0306] Aspect 57: An apparatus described in any of aspects 50 to 56, wherein the processor is configured to store the reconstructed video frame in the memory.

[0540]

[0307] Aspect 58: An apparatus described in any of aspects 50 to 57, wherein the processor is configured to cause the reconstructed video frame to be displayed.

[0541]

[0308] Aspect 59: An apparatus described in any of aspects 50 to 58, further comprising a display configured to display the reconstructed video frames.

[0542]

[0309] Aspect 60: An apparatus described in any of aspects 50 to 59, wherein the processor includes a neural processing unit (NPU).

[0543]

[0310] Aspect 61: An apparatus described in any of aspects 50 to 60, wherein the apparatus comprises a mobile device.

[0544]

[0311] Aspect 62: An apparatus described in aspects 50 to 60, wherein the apparatus comprises an extended reality device.

[0545]

[0312] Aspect 63: The apparatus described in aspects 50 to 60, wherein the apparatus comprises a television.

[0546]

[0313] Aspect 64: An apparatus described in any of aspects 50 to 63, wherein the apparatus comprises a camera configured to capture one or more video frames.

[0547]

[0314] Aspect 65: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations of claims 1 to 18.

[0548]

[0315] Aspect 66: An apparatus comprising means for performing any of the operations of claims 1 to 18.

[0549]

[0316] Aspect 67: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations of claims 43 to 49.

[0550]

[0317] Aspect 68: An apparatus comprising means for performing any of the operations of claims 43 to 49.

[0551]

[0318] Aspect 69: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations of claims 1 to 18 and any of the operations of claims 43 to 49.

[0552]

[0319] Aspect 70: An apparatus comprising means for performing any of the operations of claims 1 to 18 and any of the operations of claims 8 to 49. The invention as described in the claims of the original application is set forth below. [C1] A method for processing video data, comprising the steps of: obtaining, by an encoder portion of the neural network system, an input video frame for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, output data for the current time step operation of the neural network system, the output data being generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; obtaining, by the decoder portion of the neural network system, the output data for the current time step operation and the recurrent state data from the at least one recurrent layer of the decoder portion from the previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the output data and the recursive state data from the previous time step operation, the reconstructed video frame representing the input video frame; A method for providing the above. [C2] determining, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtaining a previously reconstructed video frame generated during operation of the previous time step; generating a warped reconstructed video frame for the current time step motion, at least in part, by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; The method of claim C1, further comprising: [C3] The method of C2, further comprising: determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; and generating the reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step. [C4] determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; and processing, by a post-warping network of the decoder portion, the warped reconstructed video frame and the reconstructed residual data determined for the current time step. Further equipped with the reconstructed video frame is generated for the current time step operation based on processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network. The method according to C2. [C5] determining, by a motion estimation network of the neural network system, motion estimation data for the current time step motion based at least in part on the input video frame and at least one of a previously reconstructed video frame from the previous time step motion or a previous input video frame from the previous time step motion; generating a warped reconstructed video frame for the current time step, at least in part, by modifying one or more pixels of at least one of the previously reconstructed video frame or the previously input video frame using the motion estimation data determined for the current time step; sending the warped reconstructed video frame and the motion estimation data determined for the current time step of motion to the encoder portion of the neural network system; Further equipped with the output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the warped reconstructed video frame and the motion estimation data determined for the current time step operation. The method according to C1. [C6] training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; Deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame; The method of C5, further comprising: [C7] The method of C1, wherein the recurrent state data includes one or more values ​​of nodes of the at least one recurrent layer. [C8] The method of C1, wherein the decoder portion of the neural network system includes a plurality of recurrent layers, each recurrent layer of the plurality of recurrent layers providing respective recurrent state data. [C9] generating warped recursive state data for the current time step operation at least in part by modifying the recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or the previous time step operation; Further equipped with the reconstructed video frame generated for the current time step of operation is generated based at least in part on the warped recurrent state data. The method according to C1. [C10] generating, by said decoder portion, a reconstructed mask, said reconstructed mask including one or more values ​​indicative of the presence of data in a previously reconstructed video frame generated during operation of said previous time step; generating the reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, a warped reconstructed video frame generated for the current time step operation, and reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data; The method of claim C1, further comprising: [C11] obtaining a previously reconstructed video frame generated during operation of said previous time step; generating a warped reconstructed video frame for the current time step operation, at least in part, by modifying one or more pixels of the previously reconstructed video frame using motion estimation data determined for the current time step during the current time step operation; determining, by the decoder portion during the current time step operation, first reconstructed residual data for the current time step operation; generating the reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; determining, by the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation; during operation of the current time step, modifying the previously reconstructed video frame using the second reconstructed residual data; outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation; The method of C1, further comprising: [C12] acquiring one or more distortion maps by the encoder portion of the neural network system; generating, by the encoder portion of the neural network system, the output data based at least in part on the one or more distortion maps. [C13] The method of C12, wherein the one or more distortion maps include a first distortion map indicating distortion between the input video frame for the current time step motion and a previously reconstructed video frame from the previous time step motion. [C14] the one or more distortion maps include a second distortion map indicating a distortion between the input video frame and a warped reconstructed video frame for the current time step operation; The method of claim 12, wherein the warped reconstructed video frame is generated at least in part by modifying one or more pixels of a previously reconstructed video frame from operations of the previous time step. [C15] Processing input data by a first layer of the encoder portion of the neural network system; determining, by the first layer of the encoder portion, a plurality of weight values ​​for the first layer of the encoder portion; setting weights of a second layer of the encoder portion of the neural network system to the plurality of weight values; processing the input data by the second layer of the encoder portion using the weights set to the plurality of weight values; The method of claim C1, further comprising: [C16] storing the output data in a storage medium; The method of claim C1, further comprising: [C17] storing the reconstructed video frames in a storage medium; The method of claim C1, further comprising: [C18] sending the reconstructed video frames via a transmission medium to at least one device; The method of claim C1, further comprising: [C19] An apparatus for processing video data, comprising: Memory, a processor coupled to the memory; wherein the processor: obtaining, using an encoder portion of a neural network system, an input video frame for a current time step operation of the neural network system, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step operation; generating output data for the current time step operation of the neural network system using the encoder portion, the output data being generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; using the decoder portion of the neural network system to obtain the output data for the current time step operation and the recurrent state data from the at least one recurrent layer of the decoder portion from the previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation, the reconstructed video frame representing the input video frame; An apparatus configured to: [C20] The processor, determining motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; obtaining a previously reconstructed video frame generated during operation of the previous time step; generating a warped reconstructed video frame for the current time step motion by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; 20. The apparatus of claim 19, configured to: [C21] The processor, determining reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; generating the reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; The apparatus of C20, configured to perform the steps of: [C22] The processor, determining reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; processing the warped reconstructed video frame and the reconstructed residual data determined for the current time step using a post-warping network of the decoder portion; configured to: the reconstructed video frame is generated for the current time step operation based on processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network. The apparatus described in C21. [C23] The processor, determining motion estimation data for the current time step motion based at least in part on the input video frame and at least one of a previously reconstructed video frame from the previous time step motion or a previously input video frame from the previous time step motion using a motion estimation network of the neural network system; generating a warped reconstructed video frame for the current time step by modifying one or more pixels of at least one of the previously reconstructed video frame or the previously input video frame using the motion estimation data determined for the current time step; sending the warped reconstructed video frame and the motion estimation data determined for the current time step of motion to the encoder portion of the neural network system; configured to: the output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the warped reconstructed video frame and the motion estimation data determined for the current time step operation. The apparatus described in C19. [C24] The processor, training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; Deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame; The apparatus of C23, configured to: [C25] The apparatus of C19, wherein the recursive state data includes one or more values ​​of nodes of the at least one recursive layer. [C26] The apparatus of C19, wherein the decoder portion of the neural network system includes a plurality of recurrent layers, each recurrent layer of the plurality of recurrent layers providing respective recurrent state data. [C27] The processor, generating warped recursive state data for the current time step operation by modifying the recursive state data from the previous time step operation using motion estimation data determined for the current time step operation or the previous time step operation; configured to: the reconstructed video frame generated for the current time step of operation is generated based at least in part on the warped recurrent state data. The apparatus described in C19. [C28] The processor, generating, using the decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicative of the presence of data in a previously reconstructed video frame generated during operation of the previous time step; generating, using the decoder portion, the reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, a warped reconstructed video frame generated for the current time step operation, and reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data; 20. The apparatus of claim 19, configured to: [C29] The processor, obtaining a previously reconstructed video frame generated during operation of the previous time step; generating, using the decoder portion, a warped reconstructed video frame for the current time step operation by modifying one or more pixels of the previously reconstructed video frame using motion estimation data determined for the current time step during the current time step operation; determining, using the decoder portion during the current time step operation, first reconstructed residual data for the current time step operation; generating the reconstructed video frame for the current time step operation by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; determining, using the decoder portion during the current time step operation, second reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by the encoder portion during the current time step operation; using the decoder portion during the current time step operation to modify the previously reconstructed video frame using the second reconstructed residual data; and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation. 20. The apparatus of claim 19, configured to: [C30] The processor, obtaining one or more strain maps using the encoder portion of the neural network system; and generating the output data based at least in part on the one or more distortion maps using the encoder portion of the neural network system. [C31] The apparatus of C30, wherein the one or more distortion maps include a first distortion map indicating distortion between the input video frame for the current time step operation and a previously reconstructed video frame from the previous time step operation. [C32] the one or more distortion maps include a second distortion map indicating a distortion between the input video frame and a warped reconstructed video frame for the current time step operation; The apparatus of C30, wherein the warped reconstructed video frame is generated by modifying one or more pixels of a previously reconstructed video frame from operation of the previous time step. [C33] The processor, processing input data using a first layer of the encoder portion of the neural network system; determining a plurality of weight values ​​for the first layer of the encoder portion using the first layer of the encoder portion; setting weights of a second layer of the encoder portion of the neural network system to the plurality of weight values; processing the input data using the weights set to the plurality of weight values ​​using the second layer of the encoder portion; 20. The apparatus of claim 19, configured to: [C34] The processor, storing said output data in said memory; 20. The apparatus of claim 19, configured to: [C35] The processor, storing the reconstructed video frames in the memory; 20. The apparatus of claim 19, configured to: [C36] a transmitter configured to transmit the reconstructed video frames over a transmission medium to at least one device; The apparatus of C19, further comprising: [C37] The apparatus of C19, wherein the processor includes a neural processing unit (NPU). [C38] The apparatus of C19, wherein the apparatus comprises a mobile device. [C39] The apparatus of C19, wherein the apparatus comprises an extended reality device. [C40] The apparatus of C19, further comprising a display. [C41] The apparatus of C19, wherein the apparatus comprises a television. [C42] The apparatus of C19, wherein the apparatus comprises a camera configured to capture one or more video frames. [C43] A method for processing video data, comprising the steps of: obtaining, by a decoder portion of the neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; determining, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtaining a previously reconstructed video frame generated during operation of the previous time step; generating a warped reconstructed video frame for the current time step motion, at least in part, by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame; and A method for providing the above. [C44] The method of C43, further comprising: determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; and generating the reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step. [C45] determining, by the decoder portion during the current time step operation, reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by an encoder portion of the neural network system during the current time step operation; during the current time step operation, modifying the previously reconstructed video frame using the reconstructed residual data from the previous time step operation; and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation. The method of C44, further comprising: [C46] determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; processing, by a post-warping network of the decoder portion, the reconstructed residual data determined for the current time step and the warped reconstructed video frame; generating the reconstructed video frame for the current time step operation based at least in part on processing the warped reconstructed video frame and the reconstructed residual data by the postwarping network; The method of C43, further comprising: [C47] generating warped recursive state data for the current time step operation at least in part by modifying the recursive state data from the previous time step operation using the motion estimation data determined for the current time step operation or a motion estimation determined for the previous time step operation; generating the reconstructed video frame for the current time step operation based at least in part on the warped recurrent state data; The method of C43, further comprising: [C48] generating, by said decoder portion, a reconstructed mask, said reconstructed mask including one or more values ​​indicative of the presence of data in said previously reconstructed video frame generated during operation of said previous time step; generating the reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame for the current time step operation, and reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data; The method of C43, further comprising: [C49] obtaining, by an encoder portion of a neural network system, an input video frame for the current time step operation, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, the output data for the current time step operation, the output data being generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; The method of C43, further comprising: [C50] An apparatus for processing video data, comprising: Memory, a processor coupled to the memory; wherein the processor: obtaining, using a decoder portion of the neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system; determining motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; obtaining a previously reconstructed video frame generated during operation of the previous time step; generating a warped reconstructed video frame for the current time step motion, at least in part, by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame; and An apparatus configured to: [C51] The processor, 5. The apparatus of claim 4, further comprising: a decoder portion of the neural network system configured to: determine reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; and generate the reconstructed video frame for the current time step operation by at least in part adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step. [C52] The processor, determining, using the decoder portion during the current time step operation, reconstructed residual data from the previous time step operation based at least in part on a previous input video frame processed by an encoder portion of the neural network system during the current time step operation; during the current time step operation, modifying the previously reconstructed video frame using the reconstructed residual data from the previous time step operation; and outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation. The apparatus of C51, configured to perform the following: [C53] The processor, determining reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; processing the reconstructed residual data determined for the current time step and the warped reconstructed video frame using a post-warping network of the decoder portion; generating the reconstructed video frame for the current time step operation based at least in part on processing the warped reconstructed video frame and the reconstructed residual data by the postwarping network; The apparatus of C50, configured to perform the following: [C54] The processor, generating warped recursive state data for the current time step operation at least in part by modifying the recursive state data from the previous time step operation using the motion estimation data determined for the current time step operation or a motion estimation determined for the previous time step operation; generating the reconstructed video frame for the current time step operation based at least in part on the warped recurrent state data; The apparatus of C50, configured to perform the following: [C55] The processor, generating, using the decoder portion, a reconstructed mask, the reconstructed mask including one or more values ​​indicative of the presence of data in the previously reconstructed video frame generated during operation of the previous time step; generating the reconstructed video frame for the current time step operation based at least in part on the reconstructed mask, the warped reconstructed video frame for the current time step operation, and reconstructed residual data determined for the current time step operation, the reconstructed mask weighting contributions of the warped reconstructed video frame and the reconstructed residual data; The apparatus of C50, configured to perform the following: [C56] The processor, obtaining, using an encoder portion of a neural network system, an input video frame for the current time step operation, reconstructed motion estimation data from a previous time step operation of the neural network system, reconstructed residual data from the previous time step operation of the neural network system, and recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from the previous time step operation; The apparatus of C50, configured to: generate, using the encoder portion of the neural network system, the output data for the current time step operation, the output data being generated based at least in part on the input video frames, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation. [C57] The processor, storing the reconstructed video frames in the memory; The apparatus of C50, configured to perform the following: [C58] The processor, displaying the reconstructed video frames; The apparatus of C50, configured to perform the following: [C59] A display configured to display the reconstructed video frames. The apparatus of C50, further comprising: [C60] The apparatus of C50, wherein the processor includes a neural processing unit (NPU). [C61] The apparatus of C50, wherein the apparatus comprises a mobile device. [C62] The apparatus of C50, wherein the apparatus comprises an extended reality device. [C63] The apparatus of C50, wherein the apparatus comprises a television. [C64] The apparatus of C50, wherein the apparatus comprises a camera configured to capture one or more video frames.

Claims

1. 1. A method for processing video data, comprising the steps of: obtaining, by an encoder portion of a neural network system, an input video frame for a current time step operation of the neural network system, a previously reconstructed video frame generated by a decoder portion of the neural network system during a previous time step operation of the neural network system, reconstructed motion estimation data of the previously reconstructed video frame from the previous time step operation for inter prediction, reconstructed residual data of the previously reconstructed video frame generated by the decoder portion of the neural network system during the previous time step operation for inter prediction, and recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from the previous time step operation; generating, by the encoder portion of the neural network system, output data for the current time step operation of the neural network system, the output data being generated based at least in part on the input video frames, the previously reconstructed video frames from the previous time step operation, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; obtaining, by the decoder portion of the neural network system, the output data for the current time step operation and the recurrent state data from the at least one recurrent layer of the decoder portion from the previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the output data and the recursive state data from the previous time step operation, the reconstructed video frame representing the input video frame; determining, by a motion estimation network of the neural network system, motion estimation data for the current time step motion based at least in part on the input video frame and at least one of a previously reconstructed video frame from the previous time step motion or a previously input video frame from the previous time step motion; generating a warped reconstructed video frame for the current time step, at least in part, by modifying one or more pixels of at least one of the previously reconstructed video frame or the previously input video frame using the motion estimation data determined for the current time step; sending the warped reconstructed video frame and the motion estimation data determined for the current time step of motion to the encoder portion of the neural network system; training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; Deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame; Equipped with the output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the motion estimation data determined for the current time step operation and the warped reconstructed video frame. method.

2. determining, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; generating a warped reconstructed video frame for the current time step motion, at least in part, by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; The method of claim 1 further comprising:

3. determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; generating the reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; The method of claim 2 , further comprising:

4. determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; processing the warped reconstructed video frame and the reconstructed residual data determined for the current time step by a post-warping network of the decoder portion; Further equipped with the reconstructed video frame is generated for the current time step operation based on processing of the warped reconstructed video frame and the reconstructed residual data by the post-warping network. The method of claim 2.

5. The method of claim 1 , wherein the recurrent state data includes one or more values ​​of nodes of the at least one recurrent layer.

6. 2. The method of claim 1, wherein the decoder portion of the neural network system includes a plurality of recurrent layers, each recurrent layer of the plurality of recurrent layers providing a respective recurrent state data.

7. 1. An apparatus for processing video data, comprising: Memory, a processor coupled to the memory; wherein the processor: obtaining, using an encoder portion of a neural network system, an input video frame for a current time step operation of the neural network system, a previously reconstructed video frame generated by a decoder portion of the neural network system during a previous time step operation of the neural network system, reconstructed motion estimation data of the previously reconstructed video frame from the previous time step for inter prediction, reconstructed residual data of the previously reconstructed video frame generated by the decoder portion of the neural network system during the previous time step operation for inter prediction, and recurrent state data from at least one recurrent layer of the decoder portion of the neural network system from the previous time step operation; generating output data for the current time step operation of the neural network system using the encoder portion, the output data being generated based at least in part on the input video frames, the previously reconstructed video frames from the previous time step operation, the reconstructed motion estimation data from the previous time step operation, the reconstructed residual data from the previous time step operation, and the recurrent state data from the previous time step operation; obtaining, using the decoder portion of a neural network system, the output data for the current time step operation and the recurrent state data from the at least one recurrent layer of the decoder portion from the previous time step operation; generating a reconstructed video frame for the current time step operation based at least in part on the output data and the recursive state data from the previous time step operation, the reconstructed video frame representing the input video frame; determining motion estimation data for the current time step motion based at least in part on the input video frame and at least one of a previously reconstructed video frame from the previous time step motion or a previous input video frame from the previous time step motion using a motion estimation network of the neural network system; generating a warped reconstructed video frame for the current time step by modifying one or more pixels of at least one of the previously reconstructed video frame or the previously input video frame using the motion estimation data determined for the current time step; sending the warped reconstructed video frame and the motion estimation data determined for the current time step of motion to the encoder portion of the neural network system; training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; Deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame; configured to: the output data generated for the current time step operation is generated by the encoder portion of the neural network system based at least in part on the motion estimation data determined for the current time step operation and the warped reconstructed video frame. Device.

8. The apparatus of claim 7, further comprising means for performing a method according to any one of claims 2 to 6.

9. 1. A method for processing video data, comprising the steps of: obtaining, by a decoder portion of a neural network system, output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system, the output data being generated at least in part based on an input video frame for the current time step operation, a previously reconstructed video frame generated during the previous time step operation, reconstructed motion estimation data of the previously reconstructed video frame from the previous time step operation for inter prediction, reconstructed residual data of the previously reconstructed video frame generated during the previous time step operation for inter prediction, and the recurrent state data; determining, by the decoder portion of the neural network system, motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; obtaining a previously reconstructed video frame generated during operation of the previous time step; generating a warped reconstructed video frame for the current time step motion, at least in part, by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame; and training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; Deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame; A method for providing the above.

10. determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; generating the reconstructed video frame for the current time step operation at least in part by adding the warped reconstructed video frame to the reconstructed residual data determined for the current time step; The method of claim 9 further comprising:

11. during said current time step operation, modifying said previously reconstructed video frame using said reconstructed residual data from said previous time step operation; outputting the modified previously reconstructed video frame as a final output video frame for the current time step operation; The method of claim 10 further comprising:

12. determining, by the decoder portion of the neural network system, reconstructed residual data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation; processing the reconstructed residual data and the warped reconstructed video frame determined for the current time step by a post-warping network of the decoder portion; generating the reconstructed video frame for the current time step operation based at least in part on processing the warped reconstructed video frame and the reconstructed residual data by the postwarping network; The method of claim 9 further comprising:

13. generating warped recursive state data for the current time step operation at least in part by modifying the recursive state data from the previous time step operation using the motion estimation data determined for the current time step operation or a motion estimation determined for the previous time step operation; generating the reconstructed video frame for the current time step operation based at least in part on the warped recurrent state data; The method of claim 9 further comprising:

14. 1. An apparatus for processing video data, comprising: Memory, a processor coupled to the memory; wherein the processor: using a decoder portion of a neural network system to obtain output data for a current time step operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step operation of the neural network system, the output data being generated at least in part based on input video frames for the current time step operation, previously reconstructed video frames generated during the previous time step operation, reconstructed motion estimation data of the previously reconstructed video frames from the previous time step operation for inter prediction, reconstructed residual data generated during the previous time step operation, and the recurrent state data; determining motion estimation data for the current time step operation based at least in part on the output data and the recurrent state data from the previous time step operation using the decoder portion of the neural network system; obtaining a previously reconstructed video frame generated during operation of the previous time step; generating a warped reconstructed video frame for the current time step motion, at least in part, by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step; generating a reconstructed video frame for the current time step operation based at least in part on the warped reconstructed video frame; and training the neural network system using a first loss function for one or more training iterations, the first loss function determining a loss between one or more input video frames and one or more warped reconstructed video frames; Deactivating the first loss function; and training the neural network system using a second loss function for one or more subsequent training iterations performed after the one or more training iterations, the second loss function determining a loss between at least one input video frame and at least one reconstructed video frame; An apparatus configured to:

15. The apparatus of claim 14, further comprising means for performing a method according to any one of claims 10 to 13.

Images