Video Compression Using Optical Flow

Abstract

In one aspect, the method includes receiving a video sequence of frames, generating an optical flow between two sequential frames using a flow prediction network, the two sequential frames including a first frame and a second frame following the first frame, generating a predicted optical flow between the first frame and the second frame from the optical flow using a first autoencoder neural network, warping a reconstruction of the first frame according to the predicted optical flow, and subsequently applying a blurring operation to obtain an initial predicted reconstruction of the second frame.

Description

[Technical Field]

[0001] This specification relates to processing data using machine learning models. [Background technology]

[0002] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, which generate an output based on the received input and the values of the parameters of the model.

[0003] Some machine learning models are deep models that use multiple layers of the model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received input to generate an output. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Rico Jonschkowski et al., “What matters in unsupervised optical flow,” arXiv:2006.04902, 1(2):3, 2020 [Non-patent document 2] E. Agustsson et al., “Scale-space flow for end-to-end optimized video compression,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8503–8512, 2020. [Non-patent document 3] D. Minnen et al., “Joint autoregressive and hierarchical priors for learned image compression,” Advances in Neural Information Processing Systems, pp. 10771–10780, 2018. Summary of the Invention [Means for solving the problem]

[0005] This specification describes a compression system, implemented as a computer program on one or more computers at one or more locations, that is capable of compressing video data.

[0006] Throughout this specification, a "latency" can refer to an ordered collection of one or more numerical values, for example, a vector, matrix, or other tensor of numerical values.

[0007] Throughout this specification, "quantizing" an input number refers to mapping the input number to an output number drawn from a discrete set of possible numbers. For example, the input number may be mapped to the closest number from the discrete set of possible numbers. The discrete set of possible numbers may be, for example, integer values in the range [0,255] or another suitable discrete set of numbers.

[0008] Throughout this specification, the optical flow between a first video frame and a second video frame defines, for each pixel in the first video frame, a flow vector that represents the displacement (motion) of the pixel between the first and second video frames. Each flow vector may be, for example, a two-dimensional (2D) vector in the frame of reference of the video frame.

[0009] Each neural network described herein may have any suitable architecture that enables the neural network to perform its described functions. For example, each neural network may include any suitable type of neural network layers (e.g., convolutional layers, fully connected layers, attention layers, etc.), in any suitable number (e.g., 5 layers, 10 layers, or 20 layers) and combined in any suitable configuration (e.g., as a linear sequence of layers).

[0010] Throughout this specification, the first video frame in a sequence of video frames may be referred to as an "I-frame," and the video frames after the first video frame may be referred to as "P-frames." (In some cases, a single video may be divided into multiple sequences of video frames, and such a single video may have multiple frames designated as being I-frames.)

[0011] In one aspect, a method of compressing video implemented by a data processing apparatus is described, the method including receiving a video sequence of frames. The method may involve processing the video sequence of frames to predict a subsequent (P) frame from a previous frame, particularly by successively processing two sequential frames of the video sequence. The two sequential frames include a first frame of the video sequence and a second frame of the video sequence that follows the first frame.

[0012] Thus, the method may involve using a flow prediction network to generate an optical flow (more precisely, optical flow data representing the optical flow) between two sequential frames, where the two sequential frames include a first frame and a second frame following the first frame. The method may also involve using a first autoencoder neural network acting as a flow encoding engine to generate, from the optical flow, a predicted optical flow between the first frame and the second frame and a confidence mask. In an implementation, the confidence mask defines a set of confidence values having the same spatial dimensions as the predicted optical flow; for example, the confidence mask may have a confidence value (σ) for each pixel of the predicted optical flow within the range [0, σ_maz].

[0013] In an implementation, the first autoencoder neural network includes an encoder network coupled to (followed by) a generator network. In an implementation, the encoder neural network processes the optical flow to generate a flow latent representing the optical flow. The generator network processes the flow latent to generate a predicted optical flow (a reconstruction of the optical flow). In an implementation, the flow latent may be quantized and entropy coded.

[0014] The method may involve, for example, warping a reconstruction of a first frame according to a predicted optical flow from processing two previous sequential frames, followed by applying a blurring operation according to a confidence mask to obtain an initial predicted reconstruction of a second frame.

[0015] Warping the reconstruction of the first frame may include applying the predicted optical flow to the reconstruction of the first frame. Applying a blurring operation according to the confidence mask may include applying a blurring operation to the warped reconstruction of the first frame, where a confidence value defined by the confidence mask at a spatial location defines a scale (size) of the blurring. For example, a larger confidence value may define more blurring of pixels at a spatial location.

[0016] The method may also involve using a second autoencoder neural network acting as a residual coding engine to generate a prediction of a residual (current residual frame), which is the difference between the second frame and an earlier predicted reconstruction of the second frame.

[0017] In an implementation, the second autoencoder neural network includes an encoder network coupled to (followed by) a generator network.

[0018] In an implementation, an encoder neural network processes a current residual frame to generate a residual latent that represents the current residual frame. A generator network processes the residual latent to generate a reconstruction of the residual, i.e., a prediction of the current residual frame. In an implementation, the residual latent may be quantized and entropy coded.

[0019] The method may combine the residual prediction with the initial predicted reconstruction of the second frame to obtain a predicted second frame (predicted reconstruction of the current frame), which may be used as a reconstruction of the first frame when processing two consecutive sequential frames of a video sequence.

[0020] The compressed video for two sequential frames of a video sequence may include optionally quantized and / or entropy coded flow latents and optionally quantized and / or entropy coded residual latents. Thus, a compressed video sequence of frames may include a series of such flow and residual latents for a consecutive set of two sequential frames of the video sequence.

[0021] In some implementations, although not necessarily, the generator network of the second autoencoder neural network is a component of a generative adversarial neural network (GANN), i.e., the generator network may have been trained using an adversarial loss, with a value that depends, among other things, on the classifier score (produced by the classifier neural network) that defines the likelihood that a video frame was produced using the generator network of the second autoencoder.

[0022] In some implementations, although not necessarily, the first frame of a video sequence may be (separately) encoded using an I-frame compression system, such as a third autoencoder neural network. For example, an encoder network of the third autoencoder neural network may generate one or more latents representing the first video frame, and a generator network of the third autoencoder neural network may generate a reconstruction of the first video frame. Again, the one or more latents representing the first video frame may be quantized and entropy coded.

[0023] According to one aspect, a method for compressing video implemented by a data processing apparatus includes receiving a video sequence of frames; generating an optical flow between two sequential frames using a flow prediction network, the two sequential frames including a first frame and a second frame following the first frame; generating from the optical flow a predicted optical flow between the first frame and the second frame and a confidence mask using a first autoencoder neural network; warping a reconstruction of the first frame according to the predicted optical flow, followed by taking the initial predicted reconstruction of the second frame. applying a blurring operation according to a confidence mask to obtain a second frame; generating a residual prediction using a second autoencoder neural network, the residual prediction being the difference between the second frame and an initial predicted reconstruction of the second frame; and combining the initial predicted reconstruction of the second frame and the residual prediction to obtain a predicted second frame, wherein each of the first autoencoder neural network and the second autoencoder neural network comprises an encoder network and a generator network, respectively, and the generator network of the second autoencoder neural network is a component of a generative adversarial neural network (GANN).

[0024] In some implementations, the first frame and the second frame follow a third frame, the third frame being an initial frame in the video sequence, and the method includes, before processing the second frame and the third frame, generating a predicted reconstruction of the third frame from the third frame using a third autoencoder neural network; generating an optical flow between the third frame and the first frame using a flow prediction network; generating a predicted optical flow between the third frame and the first frame and a confidence mask using the first autoencoder neural network from the optical flow; The method further includes warping the reconstruction of the third frame, followed by applying a blurring operation according to the confidence mask to obtain an initial predicted reconstruction of the first frame; using a second autoencoder neural network to generate a residual prediction, which is the difference between the first frame and the initial predicted reconstruction of the first frame; and combining the initial predicted reconstruction of the first frame and the residual prediction to obtain a predicted first frame, wherein the third autoencoder neural network includes an encoder network and a generator network, and the third generator network of the third autoencoder neural network is a component of a generative adversarial neural network (GANN).

[0025] In some implementations, the method further includes encoding the residual using a second autoencoder neural network to obtain a residual latent; obtaining a free latent by encoding an initial prediction of the second frame using a third encoder neural network; and concatenating the free latent and the residual latent, wherein generating a prediction of the residual using the second autoencoder neural network includes generating a predicted residual by the second autoencoder neural network using the concatenation of the free latent and the residual latent.

[0026] In some implementations, the method further includes entropy encoding a quantization of the residual latent, wherein the entropy encoded quantization of the residual latent is included in compressed video data representing the video.

[0027] In some implementations, encoding the residual to obtain a residual latent includes processing the residual using an encoder neural network of the second autoencoder neural network to generate a residual latent.

[0028] In some implementations, obtaining the free potential by encoding the initial prediction of the second frame includes processing the initial prediction of the second frame using an encoder neural network to generate the free potential.

[0029] In some implementations, generating a prediction of the residual includes processing a concatenation of the free latent and the residual latent using a generator neural network of the second autoencoder neural network to generate a prediction of the residual.

[0030] In some implementations, combining the initial predicted reconstruction and residual prediction of the second frame to obtain a predicted second frame includes generating the predicted second frame by summing the initial predicted reconstruction and residual prediction of the second frame.

[0031] In some implementations, generating a predicted optical flow between the first frame and the second frame includes processing the optical flow generated by the flow prediction network using an encoder network of the first autoencoder network to generate a flow latent representing the optical flow, and processing a quantization of the flow latent using a generator neural network of the first autoencoder neural network to generate the predicted optical flow.

[0032] In some implementations, the method further includes a step of entropy encoding the quantization of the flow latent, wherein the entropy encoded quantization of the flow latent is included in compressed video data representing the video.

[0033] In some implementations, the first autoencoder neural network and the second autoencoder neural network are trained on a set of training videos to optimize an objective function that includes an adversarial loss.

[0034] In some implementations, for one or more video frames of each training video, the adversarial loss is based on a classifier score, and the classifier score is generated by operations including: generating inputs to a classifier neural network, where the inputs include reconstructions of video frames generated using the first autoencoder neural network and the second autoencoder neural network; and providing the inputs to the classifier neural network, where the classifier neural network is configured to receive inputs including the input video frames and process the inputs to generate output classifier scores that define the likelihood that the video frames were generated using the first autoencoder neural network and the second autoencoder neural network.

[0035] According to another aspect, a method implemented by one or more computers for decompressing video is provided, the method including: receiving a compressed representation of the video, wherein the compressed representation of the video defines, for each video frame after a first video frame in the video, a quantized flow latent representing optical flow between the video frame and a preceding video frame; generating a reconstruction of each video frame in the video, wherein, for each video frame after the first video frame in the video, obtaining a reconstruction of a preceding video frame in the video; processing the quantized flow latent for the video frame using a flow generator neural network to generate optical flow between the preceding video frame and the video frame; and generating a reconstruction of the video frame using (i) the reconstruction of the preceding video frame and (ii) the optical flow between the preceding video frame and the video frame.

[0036] In some implementations, the step of generating a reconstruction of a video frame using (i) a reconstruction of a preceding video frame and (ii) an optical flow between the preceding video frame and the video frame includes the steps of generating an initial reconstruction of the video frame by warping the reconstruction of the preceding video frame using the optical flow between the preceding video frame and the video frame, and generating a reconstruction of the video frame using the initial reconstruction of the video frame.

[0037] In some implementations, the step of generating a reconstruction of a video frame using an initial reconstruction of the video frame includes the steps of generating a reconstruction of a residual video frame, where the residual video frame is defined by a difference between (i) the video frame and (ii) the initial reconstruction of the video frame, and generating the reconstruction of the video frame by combining the reconstruction of the residual video frame with the initial reconstruction of the video frame.

[0038] In some implementations, the compressed representation of the video further includes, for each video frame after the first video frame in the video, a quantized residual potential representing a residual video frame, and generating a reconstruction of the residual video frame includes processing the quantized residual potential for the video frame using a residual generator neural network to generate a reconstruction of the residual video frame.

[0039] In some implementations, the compressed representation of the video defines a latent representing a first video frame in the video, and generating a reconstruction of the first video frame includes processing the latent representing the first video frame using an I-frame generator neural network to generate a reconstruction of the first video frame.

[0040] According to another aspect, there is provided a computer storage medium encoded with a computer program, the program including instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform the operations of the methods described herein.

[0041] According to another aspect, there is provided a system including a data processing apparatus and a computer storage medium encoded with a computer program, the program including instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations of the methods described herein.

[0042] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0043] Compression systems described herein can generate compressed video data by generating optical flow data that defines optical flow between video frames in a video. The compression system then compresses the optical flow data, including a compressed representation of the optical flow data within a compressed representation of the video. For example, because a significant portion of the optical flow data may have constant values that reflect smooth and predictable motion between video frames, the optical flow data may, in some cases, be compressed more efficiently than the original video frames. Thus, representing video frames in terms of optical flow allows the video to be compressed more efficiently.

[0044] In addition to generating optical flow data representing optical flow between video frames in the video, the compression system can further generate residual video frames corresponding to the video frames in the video. The residual video frames ("residuals") corresponding to the video frames represent errors in the reconstruction of the video frames generated using the optical flow data. The compression system can compress the residual video frames and include a compressed representation of the residual video frames in the compressed representation of the video. For example, because the residual video frames may be substantially composed of small values close to zero, the residual video frames may, in some cases, be compressed more efficiently than the original video frames. Thus, representing video frames in terms of optical flow and residual video frames can allow the video to be efficiently compressed while enabling high-fidelity reconstruction of the video.

[0045] The compression system can include a neural network that is trained using an adversarial loss that prompts the compression system to generate compressed video data that can be reconstructed to generate realistic video data that is free of unnatural artifacts that often result from decompressing video data using conventional systems, for example.

[0046] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0047] [Figure 1] FIG. 1 illustrates an exemplary compression system. [Figure 2] FIG. 1 illustrates an exemplary P frame compression system. [Figure 3] 1 is a flow diagram of an exemplary process for compressing video. [Figure 4] FIG. 1 illustrates an exemplary decompression compression system. [Figure 5] FIG. 1 illustrates an exemplary P frame decompression system. [Figure 6] 1 is a flow diagram of an exemplary process for decompressing video. [Figure 7] FIG. 1 illustrates an exemplary training system. DETAILED DESCRIPTION OF THE INVENTION

[0048] Like reference numbers and designations in the various drawings indicate like elements.

[0049] This specification describes a compression system for compressing video data, a decompression system for decompressing video data, and a training system for training neural networks included in the compression and decompression systems. The compression system is described in more detail with reference to Figures 1-3, the decompression system is described in more detail with reference to Figures 4-6, and the training system is described in more detail with reference to Figure 7.

[0050] 1 illustrates an exemplary compression system 100. Compression system 100 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0051] Compression system 100 is configured to receive video 104 including a sequence of video frames 106, e.g., 106-A, 106-B, 106-C, 106-D, etc. Compression system 100 processes video 104 to generate compressed video data 114, i.e., data that occupies less space in memory (in some cases, by one or more orders of magnitude) than the original video and allows for (an approximate or exact) reconstruction of the original video 104. (In some cases, video 104 may be a proper subset of a larger video, e.g., video 104 may be the first 1,000 frames of a larger video containing over 100,000 frames.)

[0052] A video can have any suitable number of video frames, e.g., 10 video frames, 1,000 video frames, or 1,000,000 video frames. Each video frame in a video may be represented as an array of pixels, e.g., a two-dimensional (2D) array of pixels, where each pixel is represented by one or more numerical values, e.g., red-green-blue (RGB) values. The video may be obtained from any suitable source. For example, the video may be provided by the compression system 100 by a user, e.g., via an application programming interface (API) made available by the compression system. As another example, the video may be read from memory.

[0053] The compressed video data 114 produced by the compression system 100 may be decompressed by a decompression system to reconstruct the original video 104, as described in more detail below with reference to Figure 4. Once produced, the compressed video data 114 may, for example, be stored in memory, transmitted over a data communications network (e.g., the Internet), or used for any other suitable purpose.

[0054] The compression system 100 generates compressed video data 114 by sequentially compressing video frames in the video starting with the first video frame.

[0055] The compression system 100 includes an I-frame compression system 102, a P-frame compression system 200, and an encoding engine 112, each of which is described below.

[0056] The I-frame compression system 102 processes a first video frame in a video 104 to generate (i) one or more latency 110 representing the first video frame and (ii) a reconstruction 108 of the first video frame. Exemplary operations that may be performed by the I-frame compression system 102 are described in more detail below with reference to step 302 of FIG.

[0057] For each video frame after the first video frame, the P frame compression system generates an output that includes (i) one or more potentials 110 that represent the current video frame and (ii) a reconstructed version 108 of the current video frame. The P frame compression system 200 generates the output by processing (i) the current video frame, (ii) the preceding video frame, and (iii) a reconstruction of the preceding video frame. An example of a P frame compression system 200 is described in more detail below with reference to FIG. 2.

[0058] The coding engine 112 is configured to process each latent 110 generated by the I-frame compression system 102 (e.g., for the first frame in the video) and the P-frame compression system 200 for each (subsequent) video frame 106 to generate an encoded representation of the latent 110. The coding engine 112 may use a coding technique such as an entropy coding technique, e.g., Huffman coding or arithmetic coding, to generate the encoded representation of the latent 110. The encoded representation of the latent 110 forms part or all of the compressed video data 114. The compressed video data 114 may be represented in any suitable numerical format, for example, as a bitstream, i.e., as a sequence of bits.

[0059] 2 illustrates an exemplary P frame compression system 200. P frame compression system 200 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0060] For each video frame after the first video frame in the video, the P frame compression system 200 is configured to receive inputs including: (i) a current video frame 220, (ii) a previous video frame 218, and (iii) a reconstruction 212 of the previous video frame.

[0061] A current video frame 220 and a previous video frame 212 are extracted from the original video.

[0062] The reconstruction 212 of the previous video frame is obtained as a previous output of one of the I-frame compression systems 102 of the P-frame compression system 200. More specifically, if the current video frame 220 is the second video frame in the video, the reconstruction 212 of the previous video frame is obtained as a previous output of the I-frame compression system. If the current video frame 220 is after the second video frame in the video, the reconstruction 212 of the previous video frame is obtained as a previous output of the P-frame compression system 200.

[0063] P frame compression system 200 processes the input to generate (i) a latent representing a current video frame 220, including flow latent 206 and residual latent 208, and (ii) a reconstruction 216 of the current video frame. The latent representing the current video frame is coded (e.g., entropy coded) as described with reference to FIG. 1 and forms part of the compressed video data. Reconstruction 216 of the current video frame is subsequently provided as input to P frame compression system 200 for use in generating a latent representing a next video frame and a reconstruction of the next video frame.

[0064] The P frame compression system includes a flow coding engine 202 and a residual coding engine 112, each of which is described next.

[0065] The flow encoding engine 202 generates optical flow data that defines the optical flow between a previous video frame 218 and a current video frame 220. The flow encoding engine 202 processes the optical flow data to generate a flow potential 206 that represents the optical flow data. The flow encoding engine 202 further processes the flow potential 206 to generate a reconstruction of the optical flow data (a "predicted optical flow") and warps a reconstruction 212 of the previous video frame using the reconstructed optical flow data to generate an initial reconstruction 210 of the current frame (an "initial predicted reconstruction").

[0066] For example, exemplary operations that may be performed by the flow encoding engine 202 to generate the flow latency 206 and the initial reconstruction 210 of the current frame are described in more detail with reference to FIG.

[0067] The residual coding engine 112 generates a current residual frame as the difference between (i) an initial reconstruction of the current frame 210 and (ii) the current frame 220. The residual coding engine 112 processes the current residual frame 214 to generate a residual latent 208 that represents the current residual frame 214. The residual coding engine 112 further processes the residual latent to generate a reconstruction of the current residual frame 214 (a "residual prediction") and combines the reconstruction of the current residual frame 214 with the initial reconstruction 210 of the current frame to generate a reconstruction 216 of the current frame (a "predicted reconstruction"), which may be referred to as a "predicted second frame."

[0068] For example, exemplary operations that may be performed by the residual coding engine 112 to generate the residual latency 208 and the current frame reconstruction 216 are described in more detail with reference to FIG.

[0069] The P frame compression system 200 provides flow latents 206 and residual latents 208 to be encoded, e.g., entropy coded, and included in compressed video data representing the video. The P frame compression system 200 provides a reconstruction 216 of the current frame for processing as part of generating a latent representing the next video frame.

[0070] 3 is a flow diagram of an exemplary process 300 for compressing video. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a compression system, such as compression system 100 of FIG. 1, suitably programmed in accordance with this specification, can perform process 300.

[0071] The system generates a latent representing a first video frame in the video and a reconstruction of the first video frame in the video (302). More specifically, the system processes the first video frame in the video using a neural network conveniently referred to as an I-frame encoder neural network to generate the latent representing the first video frame in the video. The system quantizes the latent representing the first video frame by, among other things, quantizing each numerical value in the latent representing the first video frame. The system then processes the (quantized) latent representing the first video frame using a neural network conveniently referred to as an I-frame generator neural network to generate a reconstruction of the first video frame in the video. (The I-frame encoder neural network and the I-frame generator neural network may be understood to collectively define an autoencoder neural network.)

[0072] The system performs steps 304-312 sequentially for each video frame in the video, starting with the second video frame. For convenience, steps 304-312 are described as being performed with respect to the "current" video frame in the video.

[0073] The system generates a flow latency for the current video frame (304). More specifically, to generate the flow latency, the system generates optical flow data that defines the optical flow between a previous video frame and the current video frame in the video. The system can generate the optical flow data using any of a variety of techniques. For example, the system can process the previous and current video frames using a neural network, conveniently referred to as a flow prediction neural network, that is configured through training to generate an output that defines the optical flow between the previous and current video frames. An example of a flow prediction neural network is described with reference to Rico Jonschkowski et al., "What matters in unsupervised optical flow," arXiv:2006.04902, 1(2):3, 2020. As another example, the system can generate the optical flow between the previous and current video frames using the Lucas-Kanade method.

[0074] After generating the optical flow between the previous video frame and the current video frame, the system processes the data defining the optical flow using a neural network, conveniently referred to as a flow encoder neural network, to generate a flow latent for the current video frame. The system also quantizes the flow latent representing the optical flow by, among other things, quantizing each numerical value in the flow latent.

[0075] The system uses the (quantized) flow latents to generate an initial reconstruction of the current video frame (306). More specifically, the system processes the flow latents using a neural network conveniently referred to as a flow generator neural network to generate a reconstruction of the optical flow between the previous video frame and the current video frame. (The flow encoder neural network and the flow generator neural network may be understood to collectively define an autoencoder neural network.) In some implementations, in addition to generating a reconstructed optical flow, the flow generator neural network also generates a belief mask. The belief mask includes a respective value, conveniently referred to as a confidence value, for each pixel in the previous video frame. Intuitively, for each pixel, the confidence value for the pixel can characterize the accuracy of the reconstructed optical flow in the pixel's neighborhood.

[0076] The system may, for example, obtain a reconstruction of a preceding video frame previously generated by the system and warp the reconstruction of the preceding video frame according to the reconstructed optical flow to generate an initial reconstruction of the current video frame. Optionally, as part of generating the initial reconstruction of the current video frame, the system may apply a blurring operation according to a belief mask. The amount of blurring to be applied to each pixel in the initial reconstruction of the current video frame is defined by a belief value for that pixel.

[0077] The system may use any suitable warping technique to warp the reconstruction of the preceding video frame according to the reconstructed optical flow. For example, the system may warp the initial reconstruction of the current video frame according to the reconstructed optical flow.

[0078]

number

[0079] where x' is the initial reconstruction of the current video frame, x is the reconstruction of the preceding video frame,

[0080]

number

[0081] is the reconstructed optical flow, σ is the confidence mask, Warp(·) is a bilinear or trilinear warp operation (e.g., as described with reference to E. Agustsson et al., “Scale-space flow for end-to-end optimized video compression,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8503–8512, 2020), and AB(·,σ) defines the scale-space blur operation (i.e., the blur operation, where σ defines the blur scale) according to the confidence mask σ.

[0082] The system generates a residual latent for the current video frame (308). More specifically, to generate the residual latent, the system generates a residual video frame as the difference (i.e., error) between (i) the current video frame and (ii) an initial reconstruction of the current video frame. For example, the system may generate the residual video frame by subtracting the current video frame from the initial reconstruction of the current video frame. The system then processes the residual video frame using a neural network, conveniently referred to as a residual encoder neural network, to generate the residual latent. The system quantizes the residual latent by, among other things, quantizing each numerical value in the residual latent.

[0083] The system generates a reconstruction of the current video frame (310). More specifically, to generate the reconstruction of the current video frame, the system processes inputs including the (quantized) residual latents for the current video frame using a neural network conveniently referred to as a residual generator neural network to generate a reconstruction of a residual video frame. (The residual encoder neural network and the residual generator neural network may be understood to collectively define an autoencoder neural network.)

[0084] In some implementations, the system generates a latent representing an initial reconstruction of the current video frame, conveniently referred to as a "free" latent. For example, the system may process the initial reconstruction of the current video frame using an encoder neural network (e.g., an I-frame encoder neural network) to generate the free latent. The system may then include both (i) the quantized residual latent and (ii) the free latent in the input processed by the residual generator neural network to generate a reconstruction of the residual video frame. For example, the system may concatenate the quantized residual latent and the free latent and then provide the concatenation as an input to the residual generator neural network. In addition, feeding in the free latent extracted from the initial reconstruction of the current video frame can significantly increase the amount of detail synthesized in the residual video frame thanks to the additional information and context provided by the free latent. Furthermore, because the decompression system can directly compute the free latent from the initial reconstruction of the current video frame (hence the latent is "free"), the free latent does not need to be encoded into the compressed video data.

[0085] After generating the reconstructed residual video frame, the system may generate a reconstruction of the current video frame by combining (e.g., summing) (i) the reconstructed residual video frame and (ii) the initial reconstruction of the current video frame. The reconstructed residual video frame may thus be understood to correct any errors in the initial reconstruction of the current video frame that was generated by warping the reconstruction of the preceding video frame. If the current video frame is not the last video frame, the system may subsequently use the reconstruction of the current video frame to generate the initial reconstruction of the next video frame, e.g., as described in step 306.

[0086] The system determines (312) whether the current video frame is the last video frame in the video.

[0087] In response to determining that the current video frame is not the final video frame in the video, the system advances to the next video frame and returns to step 304 .

[0088] In response to determining that the current video frame is the final video frame, the system generates compressed video data representing the video from at least the quantized latents representing the video frames (316). More specifically, the system generates compressed video data from at least (i) the quantized latents representing the first video frame and (ii) respective quantized flow latents and quantized residual latents for each video frame after the first video frame in the video.

[0089] For example, the system may compress the quantized latents representing the video frames using an entropy coding technique, e.g., Huffman coding or arithmetic coding. The system may compress the quantized latents using a predefined probability distribution over a set of possible quantized numeric values or using an adaptive probability distribution determined based on the quantized latents. Exemplary techniques for determining adaptive probability distributions for entropy coding are described with reference to D. Minnen et al., "Joint autoregressive and hierarchical priors for learned image compression," Advances in Neural Information Processing Systems, pp. 10771-10780, 2018. The entropy coded representations of the quantized latents representing the video frames collectively form part or all of the compressed video data representing the video.

[0090] 4 illustrates an exemplary decompression system 400. Decompression system 400 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0091] The decompression system 400 is configured to process the compressed video data 114 produced by the compression system 100 to reconstruct the original video 104 represented by the compressed video data 114. The compressed video data 114 includes coded (e.g., entropy coded) quantized latents for each video frame in the video, as described above with reference to FIG.

[0092] The decompression system 400 includes a decoding engine 410, an I-frame decompression system 402, and a P-frame decompression system 404, each of which is described below.

[0093] The decoding engine 410 is configured to entropy decode the compressed video data 114 to generate a decoded representation of quantized latents for each video frame in the video. In particular, the decoding engine 410 generates an I-frame latent 408-A representing a first frame in the video and respective P-frame latents 408-B-D representing each video frame after the first video frame in the video. The P-frame latents 408-B-D for the video frames include flow latents and residual latents, as described above with reference to FIG. 3.

[0094] The I-frame decompression system 400 is configured to process the I-frame potential 408-A to generate a reconstruction 406 of a first video frame in the video. Exemplary operations that may be performed by the I-frame decompression system 400 to generate a reconstruction of the first video frame are described in more detail in FIG.

[0095] For each video frame after the first video frame in the video, P frame decompression system 400 is configured to process (i) the reconstruction 406 of the preceding video frame and (ii) the P frame potentials 408-B-D for the current video frame to generate a reconstruction 406 of the current video frame. An example of a P frame decompression system 400 is described in more detail with reference to FIG. 5.

[0096] The reconstruction 406 of the video frames of the video collectively define the original video 104 .

[0097] 5 illustrates an exemplary P frame decompression system 400. P frame decompression system 400 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0098] The P frame decompression system is configured to generate a reconstruction 506 of a current video frame in a video by processing (i) a reconstruction 512 of a previous video frame and (ii) a flow latent 514 and a residual latent 516 for the current video frame.

[0099] The reconstruction of the previous video frame is obtained as the previous output of the I-frame decompression system or the P-frame decompression system. More specifically, if the current video frame is the second video frame in the video, the reconstruction of the previous video frame 512 is obtained as the output of the I-frame decompression system. If the current video frame is after the second video frame in the video, the reconstruction of the previous video frame 512 is obtained as the previous output of the P-frame decompression system.

[0100] The P frame decompression system includes a flow decoding engine 502 and a residual decoding engine 504, each of which is described next.

[0101] The flow decoding engine 502 is configured to process the reconstruction 512 of the previous video frame and the flow latent 514 to generate an initial reconstruction 508 of the current video frame. More specifically, the flow decoding engine 502 processes the flow latent to generate a reconstruction of the optical flow between the previous video frame and the current video frame. The flow decoding engine 502 then warps the reconstruction of the previous video frame according to the optical flow to generate the initial reconstruction 508 of the current video frame.

[0102] The residual decoding engine 504 is configured to process the residual latency 516 to generate a reconstruction 510 of a residual video frame.

[0103] The P frame decompression system then combines, eg, sums, the initial reconstruction 508 of the current video frame with the reconstruction 510 of the residual video frame to generate a reconstruction 506 of the current video frame.

[0104] 6 is a flow diagram of an exemplary process 600 for decompressing video. For convenience, process 600 is described as being performed by one or more computer systems located at one or more locations. For example, a decompression system, such as decompression system 400 of FIG. 4 , suitably programmed in accordance with this specification, can perform process 600.

[0105] The system receives (602) compressed video data representing a video. The compressed video data may be received, for example, over a data communications network or retrieved, for example, from a memory. The compressed video data may be generated by a compression system, for example, as described with reference to FIG. 3. The compressed video data includes, for each video frame in the video, one or more coded (e.g., entropy coded) quantized latents that represent the video frame.

[0106] The system decodes the compressed video data to recover, for each video frame in the video, one or more quantized latents that represent the video frame (604). The system may, for example, decode the quantized latents that represent the video frame using any suitable entropy decoding technique. For each video frame after the first video frame, the system decodes (i) the quantized flow latent and (ii) the quantized residual latent for the video frame.

[0107] The system generates 606 a reconstruction of a first video frame in the video. More specifically, the system processes the quantized latents representing the first video frame using an I-frame generator neural network. The I-frame generator neural network shares the same parameter values as the I-frame generator neural network implemented by the compression system, e.g., as described with reference to step 302 of FIG. 3 .

[0108] The system performs steps 608-612 for each video frame after the first video frame in the video. For convenience, steps 608-612 are described with respect to the "current" video frame.

[0109] The system generates an initial reconstruction of the current video frame using the quantized flow latent for the current video frame (608). More specifically, the system processes the quantized flow latent using a flow generator neural network to generate a reconstruction of the optical flow between the previous video frame and the current video frame. The flow generator neural network shares the same parameter values as the flow generator neural network implemented by the compression system, e.g., as described with reference to step 306 of FIG. 3. The system then warps the reconstruction of the previous video frame using the reconstructed optical flow to generate an initial reconstruction of the current video frame.

[0110] The system generates a reconstruction of the current video frame using the initial reconstruction of the current video frame and the residual latent for the current video frame (610). More specifically, the system processes an input including the quantized residual latent for the current video frame using a residual generator neural network to generate a reconstruction of the residual video frame. The residual generator neural network shares the same parameter values as the residual generator neural network implemented by the compression system, e.g., as described with reference to step 310 of FIG. 3. In some implementations, the input to the residual generator neural network further includes a free latent representing the initial reconstruction of the current video frame. The system can generate the free latent, for example, by processing the initial reconstruction of the current video frame using an encoder neural network, e.g., the I-frame encoder neural network described with reference to step 302 of FIG. 3.

[0111] After generating the reconstruction of the residual video frame, the system generates a reconstruction of the current video frame using (i) the residual video frame and (ii) the initial reconstruction of the current video frame. For example, the system can generate the reconstruction of the current video frame as a sum of the residual video frame and the initial reconstruction of the current video frame.

[0112] The system determines (612) whether the current video frame is the last video frame in the video.

[0113] If the current video frame is not the last video frame in the video, the system advances to the next video frame (614) and returns to step 608.

[0114] If the current video frame is the last video frame in the video, the system outputs the reconstructed video, ie, the video includes a reconstruction of each video frame in the video.

[0115] 7 illustrates an exemplary training system 700. Training system 700 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0116] The training system is configured to train neural networks included in the compression and decompression systems on a set of training videos to optimize an objective function.

[0117] More specifically, the training system is an I-frame encoder neural network E I , I-frame generator neural network G I , flow encoder neural network E flow , the flow generator neural network G flow , residual encoder neural network E res , and the residual generator neural network G res Training. E I , G I , E flow , G flow , E res , and G res The operations performed by are described above, for example, with reference to Figures 3 and 6, and these operations may be implemented, for example, using a convolutional neural network having the capacity indicated by the relative sizes in the figures. The training system may be implemented using an I-frame classifier neural network D, as described in more detail below. I and P frame classifier neural network D P The neural networks in the compression and decompression systems are jointly trained together with the I and D P is only active during training, and SG exhibits gradient stopping behavior.

[0118] For each training video, the training system generates a quantized latent y I , and then process the first video frame in the training video using an I-frame encoder neural network to generate a reconstruction of the first video frame.

[0119]

number

[0120] The quantized latents representing the first video frame can be processed using an I-frame generator neural network to generate a quantized latent representing the first video frame. The training system then processes the latents representing the first video frame and the reconstruction of the first video frame using the I-frame classifier neural network to generate an I-frame classifier score. The I-frame classifier neural network is configured to process the latents representing the video frames and the video frames to generate an I-frame classifier score that defines the likelihood that the input video frame was generated by the I-frame generator neural network.

[0121] The training system is, for example,

[0122]

number

[0123] We train an I-frame encoder neural network and an I-frame generator neural network to optimize the objective function given by:

[0124]

number

[0125] measures the distance (e.g., L1 or L2 distance) between the first video frame and the reconstruction of the first video frame;

[0126]

number

[0127] denotes the I-frame classifier score generated by the I-frame classifier neural network by processing the reconstruction of the first video frame and the latent representing the first video frame, and R(y I ) represents the number of bits (bit rate) required to store the latent representing the first video frame. The term in the objective function that depends on the classifier score is sometimes called the adversarial loss term. (The I-frame encoder neural network and the I-frame generator neural network, as a result of being trained using an objective function that includes an adversarial loss, may be understood to collectively define a generative adversarial neural network.)

[0128] The training system is, for example,

[0129]

number

[0130] We train an I-frame classifier neural network to optimize the objective function given by

[0131]

number

[0132] denotes the I-frame classifier score generated by the I-frame classifier neural network by processing the reconstruction of the first video frame and the latent representing the first video frame, and D I (x I ,yI ) denotes the I-frame classifier score generated by the I-frame classifier neural network by processing the first video frame and the latent representing the first video frame.

[0133] For each video frame after the first video frame, the training system calculates (e.g., using a flow prediction neural network UFlow) the optical flow F between the current video frame and the preceding video frame. t and generate a quantized flow latent y t,f The optical flow data is processed using a flow encoder neural network to generate the reconstructed optical flow data

[0134]

number

[0135] and the confidence mask σ t The quantized optical flow latents are processed using a flow generator neural network to generate an initial reconstruction of the current video frame.

[0136]

number

[0137] Reconstruction of the preceding video frame using a warping operation with adaptive blur to generate

[0138]

number

[0139] ,process the reconstructed optical flow data and the confidence mask.

[0140] The training system generates a residual video frame r as the difference between the initial reconstruction of the current video frame and the current video frame. t generating a quantized residual latent by processing the residual video frame using a residual encoder neural network; and reconstructing the residual video frame by processing an input including the residual latent using a residual generator neural network.

[0141]

number

[0142] Optionally, the training system generates a free latent vector representing an initial reconstruction of the current video frame.

[0143]

number

[0144] To generate the residual video frame, an I-frame encoder neural network can be used to process the initial reconstruction of the current video frame, and the free latent can be included as an input to a residual generator neural network. The training system generates the reconstruction of the current video frame by summing the initial reconstruction of the current video frame and the reconstruction of the residual video frame.

[0145]

number

[0146] can be generated.

[0147] The training system then trains the P-frame classifier neural network D to generate P-frame classifier scores. PThe P-frame classifier neural network is configured to process an input including a video frame to generate a P-frame classifier score that defines the likelihood that the input video frame was generated using the flow generator neural network and the residual generator neural network. In some implementations, both the I-frame classifier neural network and the P-frame classifier neural network may use spectral normalization.

[0148] The training system is, for example,

[0149]

number

[0150] We train a flow encoder neural network, a flow generator neural network, a residual encoder neural network, and a residual generator neural network to optimize the objective function given by, where t indexes the video frames from the second video frame to the last video frame, T is the number of video frames, and λ, β, k flow , and k TV is a hyperparameter,

[0151]

number

[0152] is t - video frame x t and the reconstruction of the t-th video frame

[0153]

number

[0154] represents the error between

[0155]

number

[0156] represents the P-frame classifier score generated by the P-frame classifier neural network by processing the reconstruction of the t-th video frame and the input to the residual generator neural network for the t-th video frame, and SG(σ t ) is the confidence mask σ for the t-th video frame t represents the gradient stopping action acting on

[0157]

number

[0158] represents the L2 error between the optical flow for the t-th video frame and the reconstructed optical flow for the t-th video frame, and TV(σ t ) is σ t represents the total variation of t,r ) is the time it takes for video frame x t represents the number of bits (bit rate) required to store the latent representation of the discriminator score. The term in the objective function that depends on the classifier score is sometimes called the adversarial loss term. (The residual encoder neural network and the residual generator neural network, as a result of being trained using an objective function that includes an adversarial loss, can be understood to collectively define a generative adversarial neural network.)

[0159] The training system is, for example,

[0160]

number

[0161] A P-frame classifier neural network is trained to optimize an objective function given by: where t indexes the video frames from the second video frame to the last video frame, T is the number of video frames,

[0162]

number

[0163] Reconstruct the t-th video frame using a P-frame classifier neural network

[0164]

number

[0165] and the input y to the residual generator neural network t,r is the P frame classifier score generated by processing D P (x t ,y t,r ) is the P-frame classifier neural network used to classify the t-th video frame x t and the input y to the residual generator neural network t,r is the P frame classifier score generated by processing

[0166] The training system can pre-train a flow prediction neural network UFlow to perform optical flow prediction, and can optionally freeze parameter values of the flow prediction neural network during training of other neural networks included in the compression and decompression systems.

[0167] This specification uses the term "configured" in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. A computer program or programs configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0168] Embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and structural equivalents thereof, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiver apparatus suitable for execution by a data processing apparatus.

[0169] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or further include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can optionally include code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0170] A computer program, which may also be called or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0171] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed on and operate on the same one or more computers.

[0172] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, e.g., an FPGA or ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0173] A computer suitable for running a computer program may be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive and / or transfer data therefrom. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0174] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0175] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0176] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of the machine learning training or production, i.e., inference, workload, for example.

[0177] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0178] Embodiments of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as data servers, or middleware components, e.g., application servers, or front-end components, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0179] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device for the purpose of displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device at the server.

[0180] While this specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from that combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0181] Similarly, although operations may be shown in the figures or recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be incorporated together in a single software product or packaged in multiple software products.

[0182] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0183] 100 Compression System 102 I-frame compression system 104 videos, original videos 106, 106-A, 106-B, 106-C, 106-D Video Frames 108 Reconstructed, Restructured Version 110 Potential 112 Coding engine, residual coding engine 114 Compressed Video Data 200 P Frame Compression System 202 Flow Encoding Engine 206 Flow Potential 208 Residual latent 210 Initial reconfiguration 212 Reconstruction 214 Current Residual Frame 216 Reconstruction 218 preceding video frames 220 Current Video Frame 300 processes 400 Decompression System 402 I-Frame Decompression System 404 P Frame Decompression System 406 Reconfiguration 408-AI Frame Latency 408-B~DP Frame Potential 410 Decryption Engine 502 Flow Decryption Engine 504 Residual Decoding Engine 506 Reconstruction 508 Initial reconstruction of the current video frame 510 Residual Video Frame Reconstruction 512 Reconstruction of Previous Video Frames 514 Flow Potential 516 residual latent 600 processes 700 Training System

Claims

1. 1. A method for compressing video implemented by a data processing apparatus, comprising: receiving a video sequence of frames; generating optical flow between two sequential frames using a flow prediction network, the two sequential frames including a first frame and a second frame following the first frame; From the optical flow, a first autoencoder neural network is used to a predicted optical flow between the first frame and the second frame; and Trust Mask and generating a warping the reconstruction of the first frame according to the predicted optical flow, followed by applying a blurring operation according to the confidence mask to obtain an initial predicted reconstruction of the second frame; using a second autoencoder neural network to generate a residual prediction, which is the difference between the second frame and the initial predicted reconstruction of the second frame; combining the prediction of the residual with the initial predicted reconstruction of the second frame to obtain a predicted second frame; Including, the first autoencoder neural network and the second autoencoder neural network each include an encoder network and a generator network, respectively; the generator network of the second autoencoder neural network is a component of a generative adversarial neural network (GANN); method.

2. the first frame and the second frame follow a third frame, the third frame being an initial frame in the video sequence; before processing the second frame and the third frame; generating a predicted reconstruction of the third frame using a third autoencoder neural network from the third frame; generating optical flow between a third frame and the first frame using the flow prediction network; From the optical flow, using the first autoencoder neural network, a predicted optical flow between the third frame and the first frame; and Trust Mask and generating a warping the reconstruction of the third frame according to the predicted optical flow, followed by applying a blurring operation according to the confidence mask to obtain an initial predicted reconstruction of the first frame; using the second autoencoder neural network to generate a residual prediction, the residual being the difference between the first frame and the initial predicted reconstruction of the first frame; combining the prediction of the residual with the initial predicted reconstruction of the first frame to obtain a predicted first frame; further comprising the third autoencoder neural network includes an encoder network and a generator network; a third generator network of the third autoencoder neural network is a component of a generative adversarial neural network (GANN); The method of claim 1.

3. encoding the residuals using the second autoencoder neural network to obtain residual latent variables; using the third autoencoder neural network to obtain free latent variables by encoding an initial prediction of the second frame; concatenating the free latent variables and the residual latent variables; further comprising generating the prediction of the residual using the second autoencoder neural network includes generating the predicted residual using the concatenation of the free latent variables and the residual latent variables with the second autoencoder neural network. The method of claim 2.

4. The method described in claim 3, further comprising a step of entropy encoding the quantized residual latent variables, wherein the entropy encoded, quantized residual latent variables are included in compressed video data representing the video.

5. encoding the residuals to obtain the residual latent variables, processing the residuals using an encoder neural network of the second autoencoder neural network to generate the residual latent variables; 4. The method of claim 3, comprising:

6. obtaining the free latent variables by encoding the initial prediction of the second frame, processing the initial prediction of the second frame using an encoder neural network to generate the free latent variables; 4. The method of claim 3, comprising:

7. generating the prediction of the residual, processing the concatenation of the free latent variables and the residual latent variables using a generator neural network of the second autoencoder neural network to generate the prediction of the residual; 4. The method of claim 3, comprising:

8. combining the prediction of the residual with the initial predicted reconstruction of the second frame to obtain the predicted second frame, generating the predicted second frame by summing the initial predicted reconstruction of the second frame and the prediction of the residual; 2. The method of claim 1, comprising:

9. generating the predicted optical flow between the first frame and the second frame, processing the optical flow generated by the flow prediction network using the encoder network of the first autoencoder neural network to generate a flow latent variable representing the optical flow; quantizing the flow latent variables using a generator neural network of the first autoencoder neural network to generate the predicted optical flow; 2. The method of claim 1, comprising:

10. The method described in claim 9, further comprising a step of entropy encoding the quantized flow latent variable, wherein the entropy encoded, quantized flow latent variable is included in compressed video data representing the video.

11. 2. The method of claim 1 , wherein the first autoencoder neural network and the second autoencoder neural network are trained on a set of training videos to optimize an objective function that includes an adversarial loss.

12. For one or more video frames of each training video, the adversarial loss is based on a classifier score, the classifier score being: generating inputs to a classifier neural network, the inputs including reconstructions of the video frames generated using the first autoencoder neural network and the second autoencoder neural network; providing the input to the classifier neural network, wherein the classifier neural network: receiving an input comprising an input video frame; processing the input to generate an output classifier score that defines the likelihood that the video frame was generated using the first autoencoder neural network and the second autoencoder neural network; and configured to provide The method of claim 11 , wherein the method is generated by operations including:

13. 1. A method implemented by one or more computers for decompressing video, comprising: receiving a compressed representation of the video, the compressed representation of the video defining, for each video frame after a first video frame in the video, a quantized flow latent variable representing optical flow between the video frame and a preceding video frame; generating a reconstruction of each video frame in the video, for each video frame after the first video frame in the video: obtaining a reconstruction of a previous video frame in the video; processing the quantized flow latent variables for the video frame using a flow generator neural network to generate optical flow between the previous video frame and the video frame; (i) generating the reconstruction of the previous video frame and (ii) generating the reconstruction of the video frame using the optical flow between the previous video frame and the video frame; Including, (i) generating the reconstruction of the previous video frame and (ii) generating the reconstruction of the video frame using the optical flow between the previous video frame and the video frame, generating an initial reconstruction of the video frame by warping the reconstruction of the preceding video frame using the optical flow between the preceding video frame and the video frame; generating the reconstruction of the video frame using the initial reconstruction of the video frame; Including, generating the reconstruction of the video frames using the initial reconstruction of the video frames, generating a reconstruction of a residual video frame, the residual video frame being defined by the difference between (i) the video frame and (ii) the initial reconstruction of the video frame; generating the reconstruction of the video frame by combining the reconstruction of the residual video frame with the initial reconstruction of the video frame; A method comprising:

14. the compressed representation of the video further includes, for each video frame after the first video frame in the video, a quantized residual latent variable representing a residual video frame; generating the reconstruction of the residual video frame, processing the quantized residual latent variables for the video frame using a residual generator neural network to generate the reconstruction of the residual video frame.

14. The method of claim 13, comprising:

15. the compressed representation of the video defines a latent variable representing the first video frame in the video; generating the reconstruction of the first video frame, processing the latent variables representing the first video frame using an I-frame generator neural network to generate the reconstruction of the first video frame; 14. The method of claim 13, comprising:

16. 14. The method of claim 13, wherein the compressed representation of the video is generated by the method of claim 1.

17. 17. A computer program configured, when said computer program is executed by a data processing apparatus, to cause said data processing apparatus to perform a method according to any one of claims 1 to 16.

18. a data processing device; a computer readable storage medium storing a computer program, the computer program being configured, when executed by the data processing apparatus, to cause the data processing apparatus to perform a method according to any one of claims 1 to 16; A system including:

Images