Video compression based on long-range end-to-end deep learning

Abstract

The present disclosure presents at least a method and apparatus for efficiently encoding or decoding a video. For example, a plurality of frames are provided to a motion estimator to generate an output comprising estimated motion information. The estimated motion information is provided to an auto-encoder or an auto-decoder to generate an output comprising a reconstructed motion field. The reconstructed motion field and one or more decoded frames of the plurality of frames are provided to a deep neural network to generate an output comprising a refined bi-directional motion field. The video is encoded or decoded based on the refined bi-directional motion field.

Description

TECHNICAL FIELD

[0001] At least one of the embodiments in this implementation relates generally to a method or apparatus for video encoding or decoding, and more particularly, to a method or apparatus for efficiently providing video compression and / or decompression based on long-range end-to-end deep learning or deep neural network. BACKGROUND

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) and transform to exploit the spatial and temporal redundancy in the video content. Generally, intra- or inter-frame prediction is used to exploit the intra- or inter-frame correlation, followed by transform, quantization, and entropy coding of the differences between the original image and the predicted image, often denoted as prediction error or prediction residual. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to the entropy coding, quantization, transform, and prediction.

[0003] Recent additions to video compression technology include various industry standards, versions of reference software, and / or documents such as the Joint Exploration Model (JEM) developed by the JVET (Joint Video Exploration Team) group and the subsequent VTM (Versatile Video Coding (VVC) Test Model). The aim is to further improve the existing HEVC (High Efficiency Video Coding) standard. SUMMARY

[0004] The disadvantages and deficiencies of the prior art are addressed and overcome by the general aspects described herein.

[0005] According to a first aspect, a method is provided. The method comprises video encoding by: providing a to-be-encoded region and one or more reconstructed regions to a motion estimator to produce an output comprising an estimated bi-directional motion field of the to-be-encoded region; providing the estimated bi-directional motion field to an autoencoder to produce an output comprising a bitstream representing an encoded region and a reconstructed bi-directional motion field; providing the reconstructed bi-directional motion field and the one or more reconstructed regions to a deep neural network to produce an output comprising a refined bi-directional motion field of the to-be-encoded region; providing the refined bi-directional motion field and the one or more reconstructed regions to a motion compensator to produce an output comprising a reconstructed version of the to-be-encoded region.

[0006] According to another aspect, a second method is provided. The method comprises video decoding by: providing a bitstream representing an encoded region to an auto-decoder to produce an output comprising a reconstructed bi-directional motion field; providing the reconstructed bi-directional motion field and one or more reconstructed regions to a deep neural network to produce an output comprising a refined reconstructed bi-directional motion field; providing the refined reconstructed bi-directional motion field and the one or more reconstructed regions to a motion compensator to produce an output comprising a reconstructed version of the encoded region, the output corresponding to a decoded region.

[0007] According to another aspect, an apparatus is provided. The apparatus includes one or more processors, where the one or more processors are configured to implement a method for video encoding according to any of its variants. According to another aspect, an apparatus for video encoding includes means for providing a to-be-encoded region and one or more reconstructed regions to a motion estimator to produce an output comprising an estimated bi-directional motion field of the to-be-encoded region; means for providing the estimated bi-directional motion field to an autoencoder to produce an output comprising a bitstream representing an encoded region and a reconstructed bi-directional motion field; means for providing the reconstructed bi-directional motion field and the one or more reconstructed regions to a deep neural network to produce an output comprising a refined bi-directional motion field of the to-be-encoded region; and means for providing the refined bi-directional motion field and the one or more reconstructed regions to a motion compensator to produce an output comprising a reconstructed version of the to-be-encoded region.

[0008] According to another aspect, another apparatus is provided. The apparatus includes one or more processors, where the one or more processors are configured to implement a method for video decoding according to any of its variants. According to another aspect, an apparatus for video decoding includes means for providing a bitstream representing an encoded region to an auto-decoder to produce an output comprising a reconstructed bi-directional motion field; means for providing the reconstructed bi-directional motion field and one or more reconstructed regions to a deep neural network to produce an output comprising a refined reconstructed bi-directional motion field; and means for providing the refined reconstructed bi-directional motion field and the one or more reconstructed regions to a motion compensator to produce an output comprising a reconstructed version of the encoded region, the output corresponding to a decoded region.

[0009] According to another general aspect of at least one embodiment, the motion compensator performs differentiable motion compensation based on the refined bi-directional motion field and the one or more reconstructed regions.

[0010] According to another general aspect of at least one embodiment, the bi-directional motion field comprises two uni-directional motion fields.

[0011] According to another general aspect of at least one embodiment, the motion correction is determined by the deep neural network, the motion correction being applied symmetrically to the two uni-directional motion fields of the reconstructed bi-directional motion field to produce the refined bi-directional motion field.

[0012] According to another general aspect of at least one embodiment, the two motion corrections are determined by the deep neural network, the two motion corrections being applied to each of the two uni-directional motion fields of the reconstructed bi-directional motion field to produce the refined bi-directional motion field.

[0013] According to another general aspect of at least one embodiment, the two refined uni-directional motion fields are based on 1) uni-directional motion fields between a region of the first reference frame and the to-be-coded region or the to-be-decoded region of the subsequent frame, and 2) uni-directional motion fields between the to-be-coded region or the to-be-decoded region of the subsequent frame and a region of the next subsequent frame, respectively. Thus, bi-prediction is bi-directional prediction (from past and future).

[0014] According to another general aspect of at least one embodiment, the plurality of frames forms a group of pictures (GOP) in a video sequence.

[0015] According to another general aspect of at least one embodiment, providing the one or more reconstructed regions to the deep neural network is performed by upsampling the one or more reconstructed regions at an input of the deep neural network.

[0016] According to another general aspect of at least one embodiment, the residual is further provided to an autoencoder or an autodecoder.

[0017] According to another general aspect of at least one embodiment, the at least one pair of reconstructed regions is used to provide a training procedure for the deep neural network.

[0018] According to another general aspect of at least one embodiment, the region is one of a block or a frame.

[0019] According to another general aspect of at least one embodiment, a mask for weighting the motion-compensated predicted region is further provided to the autoencoder or the autodecoder. Thus, the method is compatible with advanced prediction.

[0020] According to another general aspect of at least one embodiment, the mask for weighting the motion-compensated predicted region is further refined by a DNN.

[0021] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments; and at least one of (i) an antenna configured to receive a signal comprising a video block; (ii) a frequency band limiter configured to limit the received signal to a frequency band comprising the video block; or (iii) a display configured to display an output representative of the video block.

[0022] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium containing data content generated according to any of the encoding embodiments or variants.

[0023] According to another general aspect of at least one embodiment, a signal is provided, the signal comprising video data generated according to any of the encoding embodiments or variants.

[0024] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the encoding embodiments or variants.

[0025] According to another general aspect of at least one embodiment, a computer program product is provided, the computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the encoding / decoding embodiments or variants.

[0026] These and other aspects, features, and advantages of the general aspects will become apparent to those of ordinary skill in the art through reading the following detailed description, with appropriate reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A block diagram illustrating an embodiment of a video encoder is shown.

[0028] Figure 2 A block diagram illustrating an embodiment of a video decoder is shown.

[0029] Figure 3 A pictorial example showing coding tree units and coding tree concepts to represent compressed HEVC pictures is shown.

[0030] Figure 4 A pictorial example showing coding tree unit splitting into coding units, prediction units, and transform units is shown.

[0031] Figure 5 An example of hierarchical coding using groups of pictures (GOPs) is shown.

[0032] Figure 6 An example of an encoder or encoding process according to an embodiment is shown.

[0033] Figure 7 An annotation of a decoder-side DNN for motion field refinement according to an embodiment is shown.

[0034] Figure 8 An example of bi-directional motion field refinement according to an embodiment is shown.

[0035] Figure 9 An example of an enlarged motion compensation input block according to an embodiment is shown.

[0036] Figure 10 Another example of bi-directional motion field refinement according to an embodiment is shown.

[0037] Figure 11 Another example of bidirectional motion field refinement is shown, in accordance with an embodiment.

[0038] Figure 12 An example of a decoder or decoding process is shown, in accordance with an embodiment.

[0039] Figure 13 An example of an overall training process is shown, in accordance with an embodiment.

[0040] Figure 14 An example of a motion refinement training process is shown, in accordance with an embodiment.

[0041] Figure 15 An example of an encoder or encoding process with advanced prediction is shown, in accordance with an embodiment.

[0042] Figure 16 An example of a decoder or decoding process with advanced prediction is shown, in accordance with an embodiment.

[0043] Figure 17 An example of an encoder or encoding process with advanced prediction and mask refinement is shown, in accordance with an embodiment.

[0044] Figure 18 An example of an encoder or encoding process with advanced prediction and considering residuals is shown, in accordance with an embodiment.

[0045] Figure 19 An example of a decoder or decoding process with advanced prediction and considering residuals is shown, in accordance with an embodiment.

[0046] Figure 20 An example of a motion refinement training process considering residuals is shown, in accordance with an embodiment.

[0047] Figure 21 A block diagram of a system in which aspects of the present embodiments can be implemented is shown. DETAILED DESCRIPTION

[0048] Figure 1 An example video encoder 100, such as a High Efficiency Video Coding (HEVC) encoder, is shown. Figure 1 An encoder in which improvements to the HEVC standard are made or an encoder employing similar techniques to HEVC, such as a VVC (Versatile Video Coding) encoder developed by the JVET (Joint Video Exploration Team) can also be shown.

[0049] In the present application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "encoding" or "coded" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably.

[0050] Before encoding, the video sequence can undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution more resilient to compression (e.g., using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing and attached to the bitstream.

[0051] In HEVC, for encoding a video sequence of one or more pictures, the pictures are partitioned (102) into one or more slices, where each slice can comprise one or more slice segments. The slice segments are organized into coding units, prediction units and transform units. The HEVC specification distinguishes between "blocks" and "units", where a "block" processes a specific area in a sample array (e.g., luma, Y), and a "unit" comprises all collocated blocks of a coding color component (e.g., Y, Cb, Cr or monochrome), syntax elements and prediction data (e.g., motion vectors) associated with the block.

[0052] For encoding, the pictures are partitioned into coding tree blocks (CTB) of square shape with configurable size, and a consecutive set of coding tree blocks is grouped into a slice. A coding tree unit (CTU) contains the CTBs of a coding color component. A CTB is the root of a quadtree partitioning into coding blocks (CB), and a coding block can be partitioned into one or more prediction blocks (PB) and forms the root of a quadtree partitioning into transform blocks (TB). Corresponding to coding blocks, prediction blocks and transform blocks, coding units (CU) comprise prediction units (PU) and a set of tree structure transform units (TU), PUs comprise prediction information for all color components, and TUs comprise residual coding syntax structures for each color component. The size of CBs, PBs and TBs for the luma component are adapted to the corresponding CU, PU and TU. In the present application, the term "block" can be used to refer to any of CTU, CU, PU, TU, CB, PB and TB. In addition, "block" can also be used to refer to macroblocks and partitions as specified in H.264 / AVC or other video coding standards, and more generally to data arrays of various sizes.

[0053] In the example encoder 100, a picture is encoded by the encoder elements as follows. A picture to be encoded is processed in units of CUs. Each CU is encoded using either intra mode or inter mode. When a CU is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use for encoding the CU, and indicates the intra / inter decision by a prediction mode flag. Prediction residuals are calculated by subtracting (110) the prediction block from the original image block.

[0054] The prediction residuals are then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can also skip the transform and directly apply quantization on the untransformed residual signal based on 4x4 TUs. The encoder can also bypass both the transform and quantization, i.e., directly encode the residuals without applying the transform or quantization processes. In direct PCM encoding, no prediction is applied and the encoded unit samples are directly encoded into the bitstream.

[0055] The encoder decodes the encoded blocks to provide references for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residuals. The decoded prediction residuals and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed picture to perform, e.g., deblocking / SAO (sample adaptive offset) filtering to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (180).

[0056] Figure 2 A block diagram of an example video decoder 200, such as a HEVC decoder, is shown. In the decoder 200, a bitstream is decoded by the decoder elements as follows. The video decoder 200 generally performs a decoding pass that is reciprocal to the encoding pass as described above with respect to the encoder 100, the decoding pass performing video decoding as part of encoding video data. Figure 1 Figure 2 A decoder in which the HEVC standard is improved upon or a decoder employing techniques similar to HEVC, such as a VVC decoder, can also be shown.

[0057] ​In particular, the input of the decoder comprises a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, picture partitioning information and other coded information. The picture partitioning information indicates the size of CTUs and the way the CTUs are partitioned into CUs (and possibly PUs, if applicable). The decoder can thus partition (235) the picture into CTUs and each CTU into CUs according to the decoded picture partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals.

[0058] The decoded prediction residuals are combined (255) with the predicted blocks to reconstruct the image blocks. The predicted blocks can be obtained (270) from intra prediction (260) or motion-compensated prediction (i.e. inter prediction) (275). In the case of bi-prediction, two motion-compensated predictions can be combined with a weighted sum. Loop filters are applied (265) to the reconstructed images. The filtered images are stored in the reference picture buffer (280).

[0059] The decoded pictures can also undergo post-decoding processing (285), e.g. inverse color transform (e.g. from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding processing can use metadata derived in the pre-encoding process and signaled in the bitstream.

[0060] In the HEVC video compression standard, pictures are partitioned into so-called coding tree units (CTU) of typically 64x64, 128x128 or 256x256 pixels. Each CTU is represented by a coding tree in the compression domain. As shown in Figure 3 each leaf is called a coding unit (CU). Each CU is then given some intra or inter prediction parameters (e.g. prediction information). To this end, it is spatially partitioned into one or more prediction units (PU), each PU being assigned some prediction information. As shown in Figure 4 the intra or inter coding mode is assigned at the CU level.

[0061] The field of application of the present disclosure is image and video compression / decompression and deep neural networks (DNN). The aim is to improve the compression efficiency compared to existing image / video compression systems.

[0062] More specifically, the domain is the encoding / decoding of a video sequence using a GOP (Group of Pictures) structure with temporal hierarchy, similar to traditional video codecs (typical random access GOP structure, see for example "High Efficiency Video Coding (HEVC) standard", ITU-T H.265, October 2014). The video sequence is divided into several GOPs, where each picture within a GOP (past and future of the current frame to be encoded) can use already decoded pictures for prediction.

[0063] The present invention proposes to introduce the use of deep learning or DNNs (Deep Neural Networks) within this coding scheme to improve the coding efficiency. For example, the network is trained end-to-end, but with encoder side information.

[0064] Deep Neural Networks (DNNs) have shown advanced performance in various domains such as computer vision, speech recognition, natural language processing, etc. However, this performance comes at a huge computational cost, as DNNs tend to have a large number of parameters, often in the order of millions, and sometimes even billions. This leads to very high inference complexity (computational cost of using a trained DNN on test data for inference). The main challenge of this high inference complexity is to bring the performance of DNNs to mobile or embedded devices with resource constraints (e.g. in terms of battery size, computational power and memory capacity).

[0065] Deep learning based compression systems employ machine learning techniques, such as trained convolutional networks, in order to automatically discover the internal structure or representation of the input images or video frames of an input video. In contrast to hard-coded techniques, machine learning techniques enable the automatic discovery of structure, which enables a more efficient representation of the input images. That is, the input images encoded by the deep learning techniques achieve an improved reconstruction quality and an improved compression rate compared to conventional techniques. For example, one or more models can be trained once based on the machine learning techniques, but the trained models can be applied to input images irrespective of the size of the input images and the required target bitrate, and in response to an increase in bits available for compression, the image reconstruction quality of the one or more trained models is progressively improved. In deep learning based compression schemes, an autoencoder is defined as a network with two parts: a first part, called the encoder, takes an input and processes it in order to produce a representation (typically of lower size or entropy compared to the input). A second part uses this latent representation and aims at recovering the original input.

[0066] The problem addressed by the present invention is to be able to use DNNs for long-range motion estimation in video coding schemes, thus allowing temporal hierarchical coding. In conventional video codecs, long-range (e.g., using GOPs of size 16 or 32 frames) motion compensation is exploited to improve compression performance. Indeed, the longer the picture range (assuming good prediction quality), the more video sequence can be compressed with less bitrate and processed.

[0067] In previous and existing compression systems, DNN-based motion estimation (or frame interpolation) is only applied to short motion ranges. For example, in “Learning Video Compression” (Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G. Anderson, Lubomir Bourdev, https: / / arxiv.org / pdf / 1811.06981.pdf), an end-to-end DNN-based compression scheme is proposed, but only using short-range motion estimation (i.e., motion between the current frame and the previous one). In the same artifact, using previously available information (image reconstruction, motion field or even latent variables), the motion field and the residual are jointly optimized and sent to the decoder, but the proposed scheme still relies only on short-range motion, as its DNN-based approach does not handle large motions. Moreover, the same proposed system does not allow the use of bi-directional frames in order to reduce the latency.

[0068] In recent video codecs (e.g., VVC / H266), decoder-side motion refinement has been introduced (see, for example, Decoder-side Motion Vector Refinement (DMVR) or even Bi-directional Optical Flow (BDOF) processes in “Versatile Video Coding (Draft 5)” (JVET-N1001, 14thMeeting: Geneva, Switzerland, 19-27 March 2019). This technique allows an automatic refinement at the decoder side of the bi-directional motion field sent by the encoder. However, in this case, the encoding and refinement of the motion field are done separately.

[0069] Another potential problem of end-to-end approaches is that the result is completely dependent on the input (input images) and it is very difficult to control the output (e.g., motion field).

[0070] Therefore, the present embodiments solve the above-mentioned problems and deficiencies of existing compression systems by:

[0071] - by considering decoder-side refinement in the encoder-side motion field encoding, incorporating the signal motion field (encoder-side information) in the process;

[0072] - training the motion field encoding network based on frame interpolation and known motion;

[0073] - Use decoder-side motion refinement and auto-coded motion field to train full inter prediction; and / or

[0074] - Add residual coding in the scheme to allow full end-to-end optimization.

[0075] In the following, we assume that the video sequence is coded using a temporal hierarchical approach, similar to the random access structure in HEVC. As a general example, Figure 5 A GOP (Group of Pictures) of 5 frames being coded is shown. First, frame 0 (10) is coded, then frame 4 (14) is coded using data from frame 0 (10), then frame 2 (12) is coded using data from the past (frame 0) and the future (frame 4), and so on.

[0076] In the following example, we assume a GOP of size 3, comprising frame 0 (10), frame 1 (11) and frame 2 (12), but the principle is applicable to any size of GOP. In this example, frame 0 (10) is coded, then frame 2 (12) is coded using data from frame 0 (10), then the following frame 1 (11) is coded using data from the past (frame 0) and the future (the next following frame 2). The skilled person will easily extend this example to a GOP of size 2 or any size of GOP, where coding frame, frame 3 using previously decoded frame 0 and frame 4 will also follow the same process and / or architecture. Figure 5

[0077] In the following, the following notations will be used:

[0078] - X: original data

[0079] - Reconstructed data (after full decoding process), typically prediction + residual

[0080] - Predicted data (using decoded data)

[0081] - Refined data (using decoded data)

[0082] - I k : image at time k

[0083] - mv i→j : motion field from time i to time j (motion compensation using mv i→j and I i gives predicted I j )

[0084] ​One main idea of the present disclosure is to predict 2 frames or more using an encoder-side computed motion field in combination with an encoder / decoder on the decoder side DNN (Deep Neural Network). The whole process is denoted as DNN:

[0085] - Autoencoder with bottleneck entropy (aiming at compressing the input) to encode the motion field (see e.g. “End-to-end optimized image compression”, Johannes Ballé, Valero Laparra, Eero P. Simoncelli, https: / / arxiv.org / abs / 1611.01704, example of autoencoder for compression purposes).

[0086] - DNN for motion field refinement.

[0087] - Differentiable motion compensation (see e.g. “Spatial Transformer Networks”, Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu, https: / / arxiv.org / abs / 1506.02025).

[0088] The whole chain can be trained unsupervised (given a succession of images of triplets) and jointly. The main advantage is that the motion field encoding takes into account further decoder side refinements (as described in the training section below) and that the motion field refinement is not limited by large motions in the images.

[0089] We first propose a simple framework, then additional improvements to this framework are proposed in this document.

[0090] Associated encoding process / encoder

[0091] Figure 6 The overall process / block diagram for encoding 600 a frame I1 according to an embodiment is shown. First, in step 601, 2 reconstructed frames (thus available on the decoder side) and and the original frame I1 to be encoded are input to the encoder side motion estimator MF. According to a variant, the estimator MF is a bi-directional block matching estimator. The output of the estimator MF is the motion information representing the motion between the reconstructed frames and the original frame I1 and the motion between the reconstructed frames and the original frame I1. The output of the estimator MF is thus called the estimated bi-directional motion field. Typically, in a GOP example, the reconstructed frame is the previous frame of the original frame I1 and the reconstructed frame is a subsequent frame of the original frame I1, and thus the motion information is bi-directional motion information (motion from the past and the future). According to embodiments, the bi-directional motion field comprises two uni-directional motion fields mv 0→1 and mv 2→1 However, the present principles are not limited to literal bi-prediction, but are more generally compatible with bi-prediction. For example, according to another example commonly used for low-delay compression, the reconstructed frames and the reconstructed frame are both previous frames of the original frame I1. The output of the estimator MF is motion information representing block-based motion, including adaptive block size, and is thus referred to as a motion field at the frame level. However, the present principles are more generally compatible with any motion representation, such as block-based motion represented by block-level motion vectors, sub-block motion information represented by block-level motion fields, up-to-pixel wise motion information.

[0092] The estimator outputs a bi-directional motion field (mv 0→1 , mv 2→1 ), and then auto-encodes this motion field in step 602 using an auto-encoder, e.g. as described in the above-mentioned “End-to-end optimized image compression”. The generated bitstream transmits this motion field to the decoder. The decoder part of the auto-encoder generates a reconstructed motion field In step 603, the reconstructed motion field is then used, together with the already reconstructed frames and , to produce a refined bi-directional motion field (labeled ) using a DNN (see below for details).

[0093] In step 604, motion compensation is performed using the reconstructed and refined motion field and the already reconstructed frames to produce a prediction of the current frame (I1). In this simple example, no additional residual is added, making the prediction become the reconstruction (I1) (see below for an exemplary process including a residual). In embodiments more specifically related to the training phase, differentiable motion compensation is used, e.g. similarly as described in the above-mentioned “Spatial Transformer Networks”. With differentiable motion compensation, the skilled person will understand that there is a motion compensation function for which a derivative function exists. For simplicity, we assume here that the prediction is the average of 2 predictions (see below for alternative embodiments). The reconstructed frame (I1) can then be used as the reconstructed frame and for encoding the subsequent frames of the GOP. As explained previously, the joint training of the DNN motion refinement and autoencoder allows performing long-range motion estimation in the frames. Moreover, the DNN-based bi-directional motion field refinement is implemented in the decoder side of the compression scheme and is advantageously implemented in the same way in the encoding process / encoder or in the corresponding decoding process / decoder (to be described hereafter).

[0094] In a variant, we assume that the image to be encoded is provided to the entire DNN zone by zone (typically block). For example, each block is independently sent to the same network. In another variant, the entire image comprising several blocks is provided to the DNN. According to the implementation of the variant, the zone is one of the blocks or frames.

[0095] Example of bi-directional motion refinement

[0096] We also propose a possible variant of decoder-side DNN-based bi-directional motion field refinement. Figure 7 The annotations and inputs / outputs of the process are shown: the inputs are 2 images, 2 uni-directional motion fields (from the first image to the intermediate image; last image to the intermediate image). The output is the refined uni-directional motion field.

[0097] Example 1

[0098] As Figure 8 shown, in this variant, we assume for simplicity that the DNN is applied block by block. Using each input block and the associated motion field (in the case of a block-based approach, the motion field is simply a single motion vector), each input block is motion compensated.

[0099] Taking into account the imprecision of the input motion vector, the input block is enlarged by N pixels. For example, if the input block is a 64x64 block, the block is enlarged to 96x96 (16 pixel border) to create the motion compensated block (for details on the input and output block sizes, see the exemplary training process below).

[0100] In Figure 9 , we show the current block B that is being processed. The motion vector associated with each input block is used to motion compensate the block while enlarging the block size: (resp. ) is the enlarged and motion compensated block B using mv 0→1 and image I0. These 2 blocks are then provided to the DNN described above.

[0101] The DNN output is a motion correction δmv 0→2 on the already motion compensated block, applied in a symmetrical way on each uni-directional motion field. According to the implementation, the motion correction δmv 0→2is the offset scaled by 1 / 2 and -1 / 2, the resulting motion correction δmv 0→1 and δmv 2→1 are added to the input motion field mv 0→1 and mv 2→1 ( actually reconstructed motion field ) to produce the refined bi-directional motion field Note that while the input motion field is a single vector (i.e. a uniform motion field for each pixel of the block), the output is a pixel-based motion field.

[0102] According to an embodiment, the input motion field has a pixel precision, in order to avoid image interpolation of the input motion compensated block during the motion compensation process.

[0103] Example 2

[0104] Figure 10 Another variant is illustrated. In this variant, the DNN also gets as input the encoder side motion field mv 0→1 and mv 2→1 allows to process sub-block motion field as input. Using each input block and the associated motion field (in case of block-based approach, the motion field is just a single motion vector), each input block is motion compensated.

[0105] Example 3

[0106] Figure 11 Another variant is illustrated. In this variant, the bi-directional motion field is obtained by separately computing each unidirectional motion field: for each input image and associated motion field mv 0→1 , mv 2→1 , two motion corrections δmv 0→1 and δmv 2→1 are computed respectively. In both cases, the network is the same, only the order of the input is different (see example training phase below). The resulting motion corrections δmv 0→1 and δmv 2→1 are then added to the input unidirectional motion field mv 0→1 and mv 2→1 ( actually reconstructed unidirectional motion field to produce the refined bi-directional motion field According to this variant, each unidirectional motion gets one motion correction.

[0107] Associated decoding process / decoder

[0108] In Figure 12In the following, we show an exemplary decoding process / decoder 1200 associated with the previously described process. The decoding process / decoder implements the steps / modules of the decoder side of the encoding process / encoder according to any of its variants. The bitstream is sent to an auto-decoder 1202, which is the decoder part of the auto-encoder 602, the auto-decoder produces a reconstructed bidirectional motion field Figure 6 As previously mentioned, the terms bidirectional and bi-prediction can be used interchangeably, and the terms motion field, motion vector and motion information can be used interchangeably. In step 1203, the reconstructed motion field is used with the already decoded (or reconstructed) frames to produce a refined bidirectional motion field (labeled Using the reconstructed and refined motion field and the already decoded frames, motion compensation 1204 is performed to produce a prediction of the current frame In a variant, the motion compensation 1204 is a differentiable motion compensation. In a variant, where no residual information is sent from the encoder to the decoder, the predicted frame is the reconstructed frame In the following, variant implementations of the decoding process / decoder are described.

[0109] Associated training process - overall network

[0110] An example of the training process of the whole DNN is shown in Figure 13 and described in the following. For a pair of reconstructed images (alternatively, original images can be used during training) and a given initial motion field, the loss L = ||I1 - (I1)||1 + λR is minimized, where I1 is the original frame to be encoded. The LI norm ||.||1 is preferred over the L2 norm because the residual cost (see below) needs to be minimized. The R parameter is given by an approximation of the rate of the motion field auto-encoder, as mentioned above in “End-to-end optimized image compression”. The λ parameter is fixed for a given rate-distortion trade-off. Note that other parameters (like the quantization step in the auto-encoder) also need to be chosen during training. According to a variant implementation, each part of the network is trained separately (auto-encoder and motion field refinement). The motion field auto-encoder is trained by using a loss directly on the output motion field error and the motion field encoding cost; the motion field refinement is trained using a given motion input and image pair to minimize the reconstruction loss of the current frame, and then the two networks are fine-tuned together.

[0111] According to other implementations, optional regularization terms can be added to the process (weight specifications, etc.).

[0112] ​​As mentioned above, the process is performed by blocks with a uniform initial motion field. Alternatively, a non-uniform initial motion field can be used according to the motion field refinement DNN type. The output of the network is a pixel-based bi-directional prediction

[0113] Motion refinement training process

[0114] Figure 14 An exemplary overall training procedure is shown for all variants of the above motion refinement network. According to an embodiment, for each pair of input images and associated initial motion field, a bi-directional motion field is output For creating the bi-directional prediction This bi-directional prediction is used in the final loss together with the original image I1. According to another embodiment, the supervised network is trained by directly using the ground truth bi-directional field. According to other embodiments, an optional regularization term can be used during training.

[0115] Advanced bi-directional prediction

[0116] In the following examples, the main procedure is the same as above, only the differences are described.

[0117] Associated encoding process / encoder

[0118] Figure 15 An exemplary encoding procedure / encoder for the bi-directional prediction is shown. In Figure 15 We show a more advanced procedure where the motion field is not only composed of 2 unidirectional motion fields mv 0→1 , mv 2→1 but also has a mask (denoted here by M 02 ) indicating the relative weight of the prediction from I0 and from I2:

[0119]

[0120] The values of the mask are binary (0 or 1) or continuous (between 0 and 1 or in other ranges, such as [-0.25 to 1.25]). In the current example, the mask is computed by the motion estimator MF before refinement (encoder-side computation) and sent to the decoder using the autoencoder. Thus, the autoencoder is further provided with a mask for weighting the motion-compensated prediction.

[0121] In a variant, the mask values are block-coded (i.e., the mask values are uniform on each block). In another variant, the values are pixel-based.

[0122] Then in the refinement process, the mask is used (decoder-side computation of either of the encoding or decoding processes).

[0123] Associated decoding process / decoder

[0124] Figure 16 An exemplary decoding process / decoder for bi-prediction is shown. As Figure 16 shown, the decoding or decoder is similar to the one presented earlier, only the decoded mask is used to form the bi-prediction.

[0125] Additional examples

[0126] Figure 17 A variant of the above process is shown, where not only the motion field, but also the mask is refined at the decoder side. Assuming a block-based mask input, the output is a pixel-wise refined mask. The associated training is similar to the one described above, as the final loss does not change (only the prediction computation is modified by the mask information).

[0127] In the following, the main process is the same as above, only the differences are detailed.

[0128] Associated encoding process / encoder

[0129] Figure 18 A more advanced process is shown, in which the whole network also includes residual coding. The residual is computed as the difference between the prediction and the original image I1. This residual is then auto-encoded and transmitted in the bitstream together with the motion field.

[0130] In a variant, the same auto-encoder is used to encode the motion field together with the residual.

[0131] Associated decoding process / decoder

[0132] Figure 19 An exemplary decoding process / decoder for residual-based reconstruction is shown. As Figure 19 shown, the decoder is similar to the one presented earlier, only the decoded error residual is added to the prediction.

[0133] Figure 20 An example of a motion refinement training process taking into account the residual according to an embodiment is shown. This training process is similar to the one already described, only a new term appears in the loss, related to the coding cost of the residual. The loss is then expressed as:

[0134]

[0135] where R m is the coding cost of the motion field and R r is one of the residuals. The quantization steps associated with each network can be fixed independently or jointly.

[0136] For the training process, other implementations are also possible. According to a variant implementation, each part of the network is trained separately (autoencoder and motion field refinement): the motion field autoencoder is trained by using a loss directly against the output motion field error and the motion field encoding cost; the motion field refinement is trained using a given motion input and image pair to minimize the reconstruction loss of the current frame; the residual autoencoder is trained by using a loss directly against the input image error and the image encoding cost; then all networks are fine-tuned together.

[0137] According to other implementations, optional regularization terms are added to the process (weight norms, etc.).

[0138] As mentioned above, the process is performed by blocks with uniform initial motion fields. Alternatively, non-uniform initial motion fields are used according to the motion field refinement DNN type. The output of the network is a pixel-based bi-prediction

[0139] Based on the above process, additional other variants are also feasible, e.g., using more than two previously reconstructed frames as input.

[0140] Figure 21 A block diagram illustrating an example of a system 2000 in which various aspects and implementations can be implemented is shown. The system 2000 can embody a device that includes the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of the system 2000 can individually or collectively be embodied in a single integrated circuit (IC), multiple ICs, and / or a

[0141] The system 2000 includes at least one processor 2010 configured to execute instructions loaded thereinto for implementing the various aspects described in this document, for example. Processor 2010 can include embedded memory, input output interface, and various other circuitry as known in the art. The system 2000 includes at least one memory 2020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 2000 includes a storage device 2040, which can include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disks, and / or optical disks. The storage device 2040 can include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0142] The system 2000 includes an encoder / decoder module 2030 configured, for example, to process data to provide encoded video or decoded video, and the encoder / decoder module 2030 can include its own processor and memory. The encoder / decoder module 2030 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 2030 can be implemented as a separate element in the system 2000, or can be incorporated in the processor 2010 as a combination of hardware and software as known to those skilled in the art.

[0143] Program code to be loaded onto processor 2010 or encoder / decoder 2030 to perform the various aspects described in this document can be stored in the storage device 2040 and then loaded onto the memory 2020 for execution by the processor 2010. In accordance with various embodiments, one or more of the processor 2010, the memory 2020, the storage device 2040, and the encoder / decoder module 2030 can store one or more of various items during the performance of the processes described in this document. Such storage items can include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0144] In several embodiments, the processor 2010 and / or memory internal to the encoder / decoder module 2030 is used to store instructions and provide working memory for processing needed during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 2010 or the encoder / decoder module 2030) is used for one or more of these functions. The external memory can be the memory 2020 and / or the storage device 2040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0145] Input to the elements of the system 2000 can be provided through various input devices as shown in block 2005. Such input devices include, but are not limited to: (i) an RF portion that receives RF signals transmitted, for example, by a broadcaster over the air,

[0146] (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0147] In various embodiments, the input devices of block 2005 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements needed to (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band which can be referred to as a channel in certain embodiments, for example, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF portion of various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band-limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner to perform various ones of these functions, including, for example, down-converting a received signal to a lower frequency (such as an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium, and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements, such as amplifiers and analog-to-digital converters, between existing elements. In various embodiments, the RF portion includes an antenna.

[0148] Further, the USB and / or HDMI terminals can include respective interface processors for connecting the system 2000 to other electronic devices across USB and / or HDMI connections. It will be appreciated that various aspects of input processing (such as Reed-Solomon error correction) can be implemented as desired, such as within separate input processing ICs or within the processor 2010. Similarly, aspects of USB or HDMI interface processing can be implemented as desired within separate interface ICs or within the processor 2010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 2010 and the encoder / decoder 2030, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on output devices.

[0149] The various elements of the system 2000 can be disposed within an integrated housing. Within the integrated housing, the various elements can be interconnected and transmit data between these elements using suitable connection arrangements 1140 (such as internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0150] The system 2000 includes a communication interface 2050 that enables communication with other devices via a communication channel 2090. The communication interface 2050 can include, but is not limited to, a transceiver configured to transmit and to receive data over the communication channel 2090. The communication interface 2050 can include, but is not limited to, a modem or network card, and the communication channel 2090 can be implemented, for example, within a wired and / or a wireless medium.

[0151] In various embodiments, data is streamed to the system 2000 using a Wi-Fi network, such as IEEE 802.11. The Wi-Fi signals of these embodiments are received through the communication channel 2090 and the communication interface 2050, which are adapted for Wi-Fi communication. The communication channel 2090 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, for allowing streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 2000, which delivers the data through the HDMI connection of the input block 2005. Still other embodiments use the RF connection of the input block 2005 to provide streaming data to the system 2000.

[0152] The system 2000 can provide output signals to various output devices, including a display 2065, speakers 2075, and other peripheral devices 2085. In various examples of embodiments, the other peripheral devices 2085 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 2000. In various embodiments, control signals are communicated between the system 2000 and the display 2065, the speakers 2075, or the other peripheral devices 2085 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 2000 via dedicated connections through respective interfaces 2060, 2070, and 2080. Alternatively, the output devices can be connected to the system 2000 using the communication channel 2090 via the communication interface 2050. The display 2065 and the speakers 2075 can be integrated in a single unit with other components of the system 2000 in an electronic device, such as a television. In various embodiments, the display interface 2060 includes a display driver, such as a timing controller (T Con) chip.

[0153] Alternatively, if the RF portion of the input 2005 is a part of a separate set top box, the display 2065 and the speakers 2075 can be separate from the other components. In various implementations in which the display 2065 and the speakers 2075 are external components, the output signals can be provided via dedicated output connections including, for example, HDMI ports, USB ports, or COMP outputs.

[0154] The implementations can be performed by computer software using, for example, hardware that is suitable for technical environments. The implementations can be implemented using one or more integrated circuits. The memory 2020 can be of any type appropriate for technical environments and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 2010 can be of any type appropriate for technical environments, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on multi-core architecture, as non-limiting examples.

[0155] Various implementations are involved in decoding. As used in this application, “decoding” can encompass all or a portion of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various implementations, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various implementations, such processes also include or alternatively include processes performed by decoders of the various implementations described in this application.

[0156] As further examples, in one implementation, “decoding” refers only to entropy decoding, in another implementation, “decoding” refers only to differential decoding, and in yet another implementation, “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” refers specifically to a subset of operations or refers broadly to a more extensive decoding process will be clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0157] Various implementations are involved in encoding. In a similar manner as discussed above with respect to “decoding,” as used in this application, “encoding” can encompass all or a portion of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various implementations, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transform, quantization, and entropy encoding. In various implementations, such processes also include or alternatively include processes performed by encoders of the various implementations described in this application.

[0158] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" refers specifically to a subset of operations or more broadly to a wider encoding process will be clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0159] Note that the syntax elements used herein are descriptive terms. Thus, they do not exclude the use of other syntax element names.

[0160] When the drawings present a flow diagram, it is understood that this also provides a block diagram of a corresponding apparatus. Similarly, when the drawings present a block diagram, it is understood that this also provides a flow diagram of a corresponding method / process.

[0161] Various embodiments refer to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account constraints of computational complexity. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solve the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a full evaluation of their encoding cost as well as the relevant distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to reduce the encoding complexity, in particular the computation of an approximate distortion based on a prediction or predicted residual signal rather than the reconstructed residual signal. A mix of these two approaches can also be used, such as by using an approximate distortion for only some of the possible encoding options, while using a full distortion for other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the encoding cost and the relevant distortion.

[0162] The specific implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), implementation of the discussed features in other forms (for example, an apparatus or program) is possible and intended. An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.

[0163] The processing device includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, a computer, a cell phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0164] Reference to“one implementation” or“an implementation” or“one specific implementation” or“a specific implementation,” along with phrasings similarly employing an implementation, refer to a particular featured, structure, characteristic, property, etc., being described in conjunction with an implementation. Thus, appearances of the phrases“in one implementation” or“in an implementation” or“in one specific implementation” or“in a specific implementation,” as well as any other variations of those phrases throughout this document, are not necessarily all referring to the same implementation.

[0165] Moreover, this document can refer to“determining” various pieces of information. Determining the information can include one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0166] Moreover, this document can refer to“accessing” various pieces of information. Accessing the information can include one or more of receiving the information, retrieving the information (e.g., from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0167] Moreover, this document can refer to“receiving” various pieces of information. As with“accessing,” receiving is intended to be a broad term. Receiving the information can include one or more of accessing the information or retrieving the information (e.g., from memory). Moreover,“receiving” is typically involved in one way or another in operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0168] It will be understood that, e.g., in the case of“A / B,”“A and / or B,” and“at least one of A and B,” any of the following is intended to be covered: selection of the first listed option (A) only, or the second listed option (B) only, or both options (A and B). As a further example, in the case of“A, B, and / or C” and“A, B, and at least one of C,” such phrasing is intended to cover the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended, as is apparent to one of ordinary skill in this and related arts, for as many items in a list as is desired for a given implementation.

[0169] Also, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in certain implementations, an encoder signals a particular one of a plurality of parameters. As such, in one implementation, the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to a decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with others, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, bit savings are achieved in various implementations. It should be understood that signaling can be achieved in various ways. For example, in various implementations, information is signaled to a corresponding decoder using one or more syntax elements, flags, etc. While the foregoing involves the verb form of the word "signal," the word "signal" can also be used as a noun herein.

[0170] It will be apparent to those skilled in the art that a specific embodiment can generate various signals having information formatted in accordance with a variety of communication protocols or conventions. For example, a signal can be formatted to carry information for use by one of the specific embodiments, or to carry data generated by one of the specific embodiments. For instance, a signal can be formatted to carry a bitstream of a described implementation. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of spectrum) or a baseband signal. The formatting can include, for example, encoding with a codec and modulating a carrier with the encoded data. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired and wireless links. A signal can be stored on a processor-readable medium.

[0171] We describe multiple embodiments. Features of the embodiments can be provided alone or in any combination. In addition, embodiments can include one or more of the above-described exemplary features, devices, or aspects, alone or in any combination, across various claim classes and types.

[0172] Throughout this disclosure, various other broad and specific aspects, embodiments and claims will also be supported. For example, the various methods and other aspects described in this application can be used to modify modules of video encoder 100 and decoder 200 (e.g., motion compensation and estimation modules (170, 175, and 275)), as described in Figure 1 and Figure 2Aspects of the disclosure are illustrated by reference to FIGS. 1-3. In addition, aspects of the disclosure are not limited to VVC or HEVC, and can apply to, for example, other standards and recommendations (whether preexisting or future-developed) and extensions of any such standards and recommendations (including VVC and HEVC). Aspects described in this application can be used alone or in combination, unless otherwise indicated or technically precluded.

Claims

1. A method for video encoding, comprising: providing a to-be-encoded region and one or more reconstructed regions to a motion estimator to produce an output comprising an estimated bi-predictive motion field for the to-be-encoded region, the one or more reconstructed regions belonging to at least two frames in a video sequence; providing the estimated bi-predictive motion field to an autoencoder to produce an output comprising video data representing an encoded region and a reconstructed bi-predictive motion field; providing the reconstructed bi-predictive motion field and the one or more reconstructed regions to a deep neural network to produce an output comprising a refined bi-predictive motion field for the to-be-encoded region; and providing the refined bi-predictive motion field and the one or more reconstructed regions to a motion compensator to produce an output comprising a reconstructed version of the to-be-encoded region.

2. A method for video decoding, comprising: providing video data representing an encoded region to an auto-decoder to produce an output comprising a reconstructed bi-predictive motion field; providing the reconstructed bi-predictive motion field and one or more reconstructed regions to a deep neural network to produce an output comprising a refined reconstructed bi-predictive motion field, the one or more reconstructed regions belonging to at least two frames in a video sequence; and providing the refined bi-predictive motion field and the one or more reconstructed regions to a motion compensator to produce an output comprising a reconstructed version of the encoded region, the output corresponding to a decoded region.

3. The method of claim 2, wherein the motion compensator performs differentiable motion compensation based on the refined bi-predictive motion field and the one or more reconstructed regions.

4. The method of claim 2, wherein a bi-predictive motion field comprises two uni-directional motion fields, and wherein a motion correction is determined by the deep neural network, the motion correction being applied symmetrically to the two uni-directional motion fields of the reconstructed bi-predictive motion field to produce the refined bi-predictive motion field.

5. The method of claim 2, wherein a bi-predictive motion field comprises two uni-directional motion fields, and wherein two motion corrections are determined by the deep neural network, the two motion corrections being applied to each of the two uni-directional motion fields of the reconstructed bi-predictive motion field, respectively, to produce the refined bi-predictive motion field.

6. The method of claim 2, wherein the refined bi-predictive motion field is based on 1) a uni-directional motion field between a region of a first reference frame and the decoded region of a subsequent frame, and 2) a uni-directional motion field between the decoded region of the subsequent frame and a region of a next subsequent frame, respectively.

7. The method of claim 2, wherein the one or more reconstructed regions belong to a plurality of frames forming a group of pictures in a video sequence.

8. The method of claim 2, wherein providing the one or more reconstructed regions to the deep neural network is performed by upsampling the one or more reconstructed regions at an input of the deep neural network.

9. The method of claim 2, wherein a residual is further decoded by the auto-decoder. ​ ​ 10. The method of claim 2, wherein at least one pair of reconstructed regions is used to provide a training procedure for the deep neural network.

11. The method of claim 2, wherein a region is one of a block or a frame.

12. The method of claim 2, wherein a mask for weighting motion compensated prediction regions is further provided to the auto-decoder.

13. The method of claim 12, wherein the mask for weighting motion compensated prediction regions is further refined.

14. A non-transitory computer readable medium comprising program code instructions stored thereon for performing the method of claim 2 when the program is executed by a processor.

15. An apparatus for video encoding, comprising: a motion estimator that receives a to-be-encoded region and one or more reconstructed regions, and produces an output comprising an estimated bi-prediction motion field for the to-be-encoded region, the one or more reconstructed regions belonging to at least two frames in a video sequence; an auto-encoder that receives the estimated bi-prediction motion field, and produces an output comprising video data representing the encoded region and a reconstructed bi-prediction motion field; a deep neural network that receives the reconstructed bi-prediction motion field and one or more reconstructed regions, and produces an output comprising a refined bi-prediction motion field for the to-be-encoded region; and a motion compensator that receives the refined bi-prediction motion field and one or more reconstructed regions, and produces an output comprising a reconstructed version of the to-be-encoded region.

16. An apparatus for video decoding, comprising: an auto-decoder that receives video data representing an encoded region, and produces an output comprising a reconstructed bi-prediction motion field; a deep neural network that receives the reconstructed bi-prediction motion field and one or more reconstructed regions, and produces an output comprising a refined bi-prediction motion field, the one or more reconstructed regions belonging to at least two frames in a video sequence; and a motion compensator that receives the refined bi-prediction motion field and one or more reconstructed regions, and produces an output comprising a reconstructed version of the encoded region.

17. The apparatus of claim 16, wherein the motion compensator performs differentiable motion compensation based on the refined bi-prediction motion field and the one or more reconstructed regions.

18. The apparatus of claim 16, wherein the auto-decoder performs receiving and decoding a residual.

19. The apparatus of claim 16, wherein the auto-decoder performs receiving a mask for weighting motion compensated prediction regions.

Images