Inter-coding using deep learning in video compression

Abstract

Methods, systems, and bitstream syntax for inter-frame coding using an end-to-end neural network used in image and video compression are described. The inter coding method includes one or more of the following: joint luma-chroma motion compensation for a YUV picture; coding the joint brightness-chrominance residual error of the YUV picture; using an attention layer; starting a time motion prediction network to carry out motion vector prediction; performing motion vector decoding using a cross-domain network in which the motion vector and the residual information are combined; decoding the residual error by using a cross-domain network; compensating inter prediction using weighted motion; and using only temporal features, only spatial features, or both temporal and spatial features in entropy decoding. A method for improving training of a neural network for inter-frame coding is also described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to Indian Provisional Patent Application No. 202241037461 filed on June 29, 2022 and Indian Provisional Patent Application No. 202341026932 filed on April 11, 2023, each of which is incorporated by reference in its entirety. Technical Field

[0003] This document relates generally to images. More particularly, embodiments of the invention relate to inter-frame coding using deep learning in video compression. Background Art

[0004] In 2020, the MPEG expert group in the International Organization for Standardization (ISO) jointly with the International Telecommunication Union (ITU) released the first version of the Universal Video Coding standard (VVC), also known as H.266. Recently, the same joint team (JVET) and the Still Picture Compression Experts (JPEG) have begun working on the development of the next generation coding standard, which will improve coding performance compared to existing image and video coding techniques. As part of this research, coding techniques based on artificial intelligence and deep learning are also investigated. As used herein, the term "deep learning" refers to a neural network with at least three layers, and preferably more than three layers.

[0005] As appreciated by the inventors herein, improved techniques for encoding images and videos based on neural networks are described herein.

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to have been recognized in any prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0008] Figure 1 Describes an example framework for using neural networks in video coding;

[0009] Figure 2A Describes an example of a separate luma-chroma motion compensation (MC) network in YUV420 video coding;

[0010] Figure 2B Describes an example of a joint luma-chroma motion compensation (MC) network in YUV420 video coding;

[0011] Figure 2C Depicted in Figure 2B An example neural network (NN) implementation of a joint luma-chroma MC network in YUV420 video coding as depicted in;

[0012] Figure 3A depicts an example neural network model for end-to-end image and video encoding according to the prior art;

[0013] Figure 3B depicts an example neural network model for end-to-end image and video encoding according to an embodiment of the present invention;

[0014] Figure 3C depicts an example of an adaptation block;

[0015] Figure 4A depicts an example NN for temporal motion vector prediction according to an embodiment of the present invention;

[0016] Figure 4B and Figure 4C Describes examples of applying flow prediction in temporal, delta, and motion vector coding of P and B frames, respectively;

[0017] Figure 4D depicts an example of a multi-frame motion prediction network with warp alignment according to an embodiment of the present invention;

[0018] Figure 5A A network architecture for cross-domain encoding of motion vectors using previously reconstructed image data according to an embodiment of the present invention is depicted;

[0019] Figure 5B A network architecture for cross-domain encoding of residual data using reconstructed motion vectors according to an embodiment of the present invention is described;

[0020] Figure 6 Describes a network architecture for a time-space entropy model according to an embodiment of the present invention; and

[0021] Figure 7 An example architecture for weighted motion compensated inter prediction according to an embodiment of the present invention is depicted. DETAILED DESCRIPTION

[0022] This document describes example embodiments for inter-frame coding when using neural networks in image and video coding. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. However, it is apparent that various embodiments of the invention can be practiced without these specific details. In other cases, well-known structures and devices are not described in detail in order to avoid unnecessarily obscuring, obscuring, or confusing embodiments of the invention. Summary of the invention

[0023] Example embodiments described herein relate to image and video encoding using neural networks. In an embodiment, a processor receives an encoded video sequence and a high-level syntax indicating enabling inter-frame coding adaptation for decoding a current picture, the processor:

[0024] parsing the high-level syntax to extract inter-coding adaptation parameters; and

[0025] Decoding the current picture based on the inter-frame coding adaptation parameters to generate an output picture, wherein the inter-frame coding adaptation parameters include one or more of the following:

[0026] Joint luma-chroma motion compensation enable flag, which indicates to use the joint luma-chroma motion compensation network in decoding when the input picture is in the YUV color domain;

[0027] a joint luma-chroma residual encoding enable flag indicating the use of a joint luma-chroma residual network in decoding when the input picture is in the YUV color domain;

[0028] Attention layer enable flag, which indicates to use attention network layer in decoding;

[0029] Temporal motion prediction enabled flag, which indicates the use of a temporal motion prediction network for motion vector prediction in decoding;

[0030] A cross-domain motion vector enable flag indicating that a cross-domain network combining motion vector and residual information is used to decode motion vectors during decoding;

[0031] A cross-domain residual enable flag indicating that the residual is decoded using a cross-domain network that combines motion vector and residual information in decoding; and

[0032] The time-space-entropy flag indicates whether entropy decoding uses only spatial features, only temporal features, or a combination of spatial and temporal features.

[0033] In a second embodiment, in a system including a processor for training a neural network for inter-frame coding, the processor may employ one or more of the following:

[0034] Large motion training, where for a training sequence with a total of n pictures, large motion training is performed using random P frame jumps from 1 to n-1;

[0035] Time distance modulation loss, where the rate-distortion loss is calculated as

[0036] Loss = w*λ*MSE+rate,

[0037] where rate represents the achieved bitrate and MSE measures the distortion between the original picture and the corresponding reconstructed picture, and the weight parameter “w” is initialized based on the temporal frame spacing as:

[0038]

[0039] where index i represents the iteration count in N training iterations.

[0040] In a third embodiment, a method for processing uncompressed video frames using one or more neural networks is presented, the method comprising:

[0041] Based on the uncompressed input video frame (x t ) to generate motion vector and spatial map information (α) of the uncompressed input video frame;

[0042] generating a motion compensated frame based at least on a motion compensation network and a reference frame used to generate the motion vector information

[0043] applying the spatial map information to the motion compensated frame to generate a weighted motion compensated frame;

[0044] generating a residual frame by subtracting the weighted motion compensated frame from the uncompressed input video frame;

[0045] Generate reconstructed residual frames based on residual encoder analysis and decoder synthesis network wherein the residual encoder analysis network generates a coded frame based on quantization of the residual frame; and

[0046] A decoded approximation of the encoded frame is generated by adding the weighted motion compensated frame to the reconstructed residual frame.

[0047] Example encoding model using deep learning

[0048] Deep learning based image and video compression methods are becoming increasingly popular and are an active area of ​​research. Figure 1An example of a basic deep learning-based framework is depicted (reference [1]). The example contains several basic components (e.g., motion compensation, motion estimation, residual coding, etc.) found in traditional codecs (such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc.). The main difference is that all these components use neural network (NN) based methods, such as motion vector (MV) decoder network (Net), motion compensation (MC) net, residual decoder net, etc. The framework also includes several encoder-only components, such as optical flow net, MV encoder net, residual encoder net, quantization, etc. This framework is often referred to as an end-to-end deep learning video coding (DLVC) framework.

[0049] Note that unlike traditional encoder architectures, this end-to-end deep learning (DL) network does not have an inverse quantization block (inverse Q). This end-to-end network does not require inverse Q. This is because a simple half-round-based quantization is performed on the latent at the encoder size, which does not require any inverse Q at the decoder side. The network is trained for different λ (different QPs) (e.g., loss = λ*MSE + rate) to generate a model for each λ.

[0050] Compared to traditional coding schemes, the state-of-the-art DLVC method can achieve similar image coding performance, but there is still a big gap in inter-frame coding when compared with VVC. The embodiments described herein will focus on improving the training, coding efficiency and coding complexity of neural networks for inter-frame (or inter-frame) coding.

[0051] YUV 4:2:0 encoding

[0052] In a typical DLVC implementation, Figure 1 The framework operates on images in the RGB domain. Given the correlation between chrominance components, it may be more efficient to operate in a luma-chrominance space (such as YUV, YCbCr, etc.) in the 4:2:0 domain (simply and without limitation denoted as YUV420), where 4:2:0 means that the chrominance components are subsampled by a factor of 2 in both horizontal and vertical resolution compared to luma.

[0053] In order to operate in the YUV420 domain, several modifications are proposed to enable more efficient YUV420 encoding. Since luminance motion and chrominance motion are highly correlated, in an embodiment, motion estimation and motion coding of luminance and chrominance are completed using a modified YUV optical flow network and a MV encoder network-MV decoder network, respectively. However, motion compensation and residual coding of the luminance component and chrominance component of YUV420 can be handled in a variety of ways as follows:

[0054] Using separate luma and chroma motion compensation (MC) networks or using a joint luma-chroma MC network;

[0055] Separate and joint motion compensation (MC) networks for YUV420 coding using separate luma and chroma residual coding networks or using a joint luma-chroma residual coding network

[0056] The MC network designed for RGB images assumes that all image channels have the same dimension. For YUV420 inter-frames, separate MC networks can be designed to adapt to the dimensions of the Y channel and UV channel, such as Figure 2A This, however, adds complexity and carries the risk that the joint information present in the Y and UV channels is not effectively exploited, and that the motion compensation for the channels may be slightly different, resulting in artifacts in the reconstructed image. Figure 2A The input of the brightness MC network in the separate MC network is the decoded motion of the current frame Reference Frame The brightness component of The output of the luma MC network is a motion compensated luma frame. Similarly, Figure 2A The input of the chrominance MC network of the separate MC network is the decoded motion of the current frame Reference Frame chroma components, and using downsampling and downscaling chroma motion Bilinear interpolation chroma prediction component warp Then, the chroma MC network outputs the motion compensated chroma components

[0057] For example, Figure 2B As shown, the joint luma-chroma MC network can effectively exploit cross dependencies as long as the sizes of the Y and UV references and the warped frame channels are properly handled. Figure 2B The input of the joint luminance-chrominance MC network is the decoded motion of the current frame Reference Frame The brightness component, expressed as warp Bilinear interpolation brightness prediction frame, reference frame chroma components, and using downsampling and downscaling chroma motion Bilinear interpolation chroma prediction component warp The joint luminance-chrominance MC network outputs the motion-compensated luminance component and chrominance components

[0058] Figure 2C An example embodiment of a neural network for joint luma-chroma MC is depicted. A typical MC neural network (such as references [1] and [7]) consists of an initial convolutional layer and residual blocks operating on the current frame spatial dimensions; followed by a) a series of average pooling layers that reduce the spatial dimensions of the predicted frame features by a factor of 2 and b) residual blocks. A series of residual blocks are then used to process the predicted frame features of lower spatial dimension, upsample them and add them back to higher dimensional features to enhance the quality of inter-frame prediction. Since the resolution of the chroma components of YUV420 is half that of the luma, motion compensation for the luma inter-frame prediction components and the chroma inter-frame prediction components is performed in a unified manner by merging the chroma channels at the appropriate luma pooling layer whose resolution matches. The proposed method saves computational cost and improves performance while also reducing memory usage.

[0059] exist Figure 2C In the joint luma-chroma MC network in , the luma and chroma bilinear interpolated frames are partially processed independently using convolution and residual blocks in the initial stage (before 205 and 210). Then, the chroma inter-frame prediction features (205) are added to the luma inter-frame prediction after the first luma pooling layer (215) because chroma is half the luma resolution. This ensures that the luma prediction and chroma prediction are jointly processed thereafter, which can reduce complexity compared to separate MC networks and can also exploit cross-channel dependencies. The chroma inter-frame prediction features are separated from the joint inter-frame prediction features before the final upsampling layer (235) and processed separately from (240) to output the final motion compensated chroma inter-frame prediction After the final upsampling (layer 225), the luma inter prediction features are processed independently to output the final motion compensated luma inter prediction

[0060] like Figure 2C As used in , the term Conv(K,C,S) denotes a convolutional network with a K×K kernel, C output channels, and a stride of S (S=1 means no upsampling or downsampling). The number of inputs is not explicitly stated, as the notation assumes that the number of outputs from a given level is equal to the number of inputs into the next level. For example, in column 240, Conv(3,64,1) is followed by Conv(3,2,1). This means that the last layer Conv(3,2,1) receives 64 input channels from the previous layer Conv(3,64,1) and outputs two channels, which correspond to the chroma MC prediction outputs Separate and joint residual coding networks for YUV420 encoding

[0061] Similar to the considerations of the MC network, the luma residual and chroma residual of the inter-frame can be encoded separately or jointly. Separate residual coding can improve the coding performance of chroma. However, if the possible cross-correlation in the luma residual channel and the chroma residual channel is not effectively utilized, separate residual coding will increase the complexity and increase the coding overhead. Separate luma / chroma residual coding networks are novel for inter-frame coding. The joint luma-chroma residual coding network can effectively exploit the cross-dependency of the residuals while reducing the complexity of the residual network and entropy coding. The current joint residual coding architecture is based on references [5-6].

[0062] Adding attention layer to inter-coding

[0063] Figure 3A An example of a process pipeline (300) for video encoding using a four-layer neural network architecture for encoding and decoding of latent features (reference [7]) is depicted. As used herein, the term "latent feature" or "latent variable" refers to a feature or variable that is not directly observable but is inferred from other observable features or variables (e.g., by processing the directly observable variables). In image and video coding, the term 'latent space' can refer to a representation of compressed data in which similar data points are closer together. In video coding, examples of latent features include representations of transform coefficients, residuals, motion representations, syntax elements, model information, etc. In the context of neural networks, latent spaces are very useful for learning data features and finding simpler representations of image data for analysis.

[0064] like Figure 3A As depicted, given an input image x (302) of h×w resolution, in the encoder (300E), the input image is processed by a series of convolutional neural network blocks (also referred to as convolutional networks or convolution blocks), each of which is followed by a nonlinear activation function (305, 310, 315, 320). At each such layer (which may include multiple convolutional network sublayers and activation functions), its output is typically reduced (e.g., reduced by a factor of 2 or more, commonly referred to as the "stride", where stride = 1 means no downsampling, stride = 2 means downsampling by a factor of 2 in each direction, etc.). For example, using stride = 2, the output of the L1 convolutional network (305) will be h / 2×w / 2. The final layer (e.g., 320) generates output latent coefficients y (322), which are further quantized (Q) and entropy encoded (e.g., by an arithmetic encoder AE) before being sent to the decoder (300D). The hyper-prior network and the spatial context model network (not shown) are also used to generate a probability model of the latent point (y).

[0065] In the decoder (300D), the process is reversed. After arithmetic decoding (AD), given the decoded latent A series of deconvolution layers (325, 330, 335, 340) (each deconvolution layer combines a deconvolution neural network block and a non-linear activation function) are used to generate the output To approximate the input (302). In the decoder, the output resolution of each deconvolution layer is typically increased (e.g., by a factor of 2 or more) to match the downsampling factor in the corresponding convolution level in the encoder 300E so that the input image and the output image have the same resolution.

[0066] In an embodiment, Figure 3B As depicted, for encoding and decoding of P-frames and B-frames, it is proposed to add "attention blocks". Attention blocks (e.g., block 355) are used to enhance certain data more than other data. As an example, an attention block can be added after two layers. In another embodiment, an attention block can be added after each layer; however, the improvement in performance may not justify the increase in complexity.

[0067] The reason behind using the adaptation block is that traditional video codecs benefit significantly from their block-level adaptation to local image / video characteristics. Therefore, DLVC should also benefit from local adaptation. The attention block is one of the ways to locally adapt the layer by weighting the filter response with spatially varying weights learned end-to-end with the filter. The attention block can also be applied to MV nets and / or residual nets and / or MC nets, etc. The use of the attention block in a specified neural network can be signaled to the decoder using high-level syntax elements. An example of such a syntax element is provided later in this specification. Using the proposed architecture, experimental results using YUV420 data have shown that the BD rate is improved by 10%-14% for Y and 0%-26% for U or V.

[0068] Temporal motion (flow) prediction

[0069] In current P-frame and B-frame models, the total bits used to encode motion information and residual information represent a large portion of the total bit rate. The motion field is correlated in both time and space. In reference [1], the motion field generated by the optical flow network exploits spatial correlation; however, temporal correlation has not been exploited. In reference [4], multiple previously decoded frames are used as input to explore temporal information. In an embodiment, it is proposed to explore temporal correlation in DLVC.

[0070] In an example embodiment, it is proposed to use temporal information based on a flow prediction network that uses the motion fields of one or more previous frames as input. Experimental results show that using two frames can achieve a good trade-off between complexity and performance, reaching a BD rate gain of about 2%. Figure 4A Depicted is an example of a NN for temporal MV prediction.

[0071] like Figure 4A As depicted, the proposed NN (400) comprises a stream buffer, a convolutional 2D network, a series of ResBlock64 layers (405) and a final convolutional 2D network, which are used for current frame motion prediction using the decoded stream of the previously decoded frame.

[0072] Figure 4B and Figure 4C Examples of applying flow prediction in temporal, delta, and motion vector coding of P frames and B frames, respectively, are described. Figure 4B shows the decoded flow using three past (L0) reference frames assuming no hierarchical P frame layer For reference P frame X t Temporal motion prediction. Figure 4C shows the decoding flow using past (L0) and future (L1) reference frames assuming no hierarchical B-frame layer For reference B frame X t On the encoder side, the temporally predicted motion stream is subtracted from the motion estimation stream and the delta motion is encoded using the MV encoder network. On the decoder side, the MV decoder network decodes the delta motion and adds back the temporally predicted motion to reconstruct the final motion. The decoded stream is used to warp the reference frame using bilinear interpolation and perform the final inter-frame prediction using MC-net.

[0073] although Figure 4A The architecture of produces coding gain, but the prediction may be suboptimal because in the presence of significant amounts of motion, the two previous motion fields and the current frame may not correspond to each other spatially, and a network with a limited perceptual field size may have difficulty inferring spatial correspondence and aligning it internally to make a good prediction of the current motion field. To address this limitation, in another embodiment, it is proposed to align the motion fields before using them as input to the prediction network. If the motion fields of the previous two time instants and the current time instant are represented in time order as and The flow field can be Backward warping to and Align, thus obtaining and The concatenation of needs to be aligned with the current frame, and this can be achieved by estimating the approximate motion field between time t-1 and time t. Assuming that the motion from time t-1 to t has the same magnitude as the motion from time t-1 to t-2, then the motion from t-1 to t can be estimated by A forward shift is performed to obtain the aligned cascade flow field. The shifted flow is used as the input to the motion predictor network. Figure 4D Depicts the proposed motion predictor network. The motion predictor consists of a sequence of three residual layers: The reverse twist, through The forward distortion of Figure 4A The motion prediction network (400) shown in FIG. Figure 4D middle, Indicated in Figure 4B and Figure 4C The quantized motion vector Δ value at the output of the MV decoder block.

[0074] Cross-domain fusion for motion and residual coding

[0075] In reference [1], motion and residual coding are performed independently. In an embodiment, it is proposed to exploit the potential cross-correlation between motion features and residual features. For example, motion discontinuity at the object boundary can be used to encode the residual features more effectively.

[0076] In an embodiment, it is proposed to use cross-domain fusion for motion vector (MV) coding. Figure 5A In the depicted embodiment, the previous frame reconstructed samples (502) are used to enhance the MV encoding (505) at the encoder using optical flow (e.g., as in block 400). At the decoder, the previous frame residual latent values ​​(508) are additionally used for MV compensation (MC) of the current frame to exploit motion cross-dependencies on the residual. The motion vector decoder block (510) uses the previous frame residual latent points (508) and the motion vector latent points (506) to apply cross-domain fusion. The motion vector encoder (505) uses the cross-domain fusion as an additional input to the motion vector encoding process based on the previous frame image (502). The fusion is done in the latent domain at the decoder and in the spatial domain at the encoder. This fusion method attempts to exploit any cross-dependencies of the current frame motion on the intensity of the current image or residual image. As an example, non-zero residuals at object boundaries may coincide with motion boundaries, which can help improve the coding efficiency of motion information.

[0077] In another embodiment, cross-domain fusion can be applied to residual coding. Figure 5BAs depicted, the reconstructed motion vectors may be used to guide residual encoding (520). At the decoder, the residual decoder (525) utilizes the motion vector latent point (507) and the residual latent point. The residual decoder block (525) applies cross-domain fusion by using the current frame motion vector latent point (507). The residual encoder (520) applies cross-domain fusion using the current frame reconstructed motion as an additional input to the residual encoding process. Fusion is done in the latent domain at the decoder and in the spatial domain at the encoder. This fusion method attempts to exploit any cross-dependencies of the current frame residual on the current frame motion in the same area. As an example, changes in motion field at object boundaries may coincide with non-zero residuals, which can help improve the coding efficiency of the residual information.

[0078] Entropy Coding Based on Temporal and Spatial Priors

[0079] In an embodiment, it is desirable to enable the entropy NN model to use features from previous frames or from spatially neighboring frames. As in reference [8], the core idea is to enable the entropy model to estimate spatiotemporal redundancy in latent space rather than at the pixel level, which significantly reduces the complexity of the framework.

[0080] In inter-frame coding, the residual intensity map undergoes roughly the same motion as the current image. Since the encoder CNN network is shift invariant, the latent feature map also undergoes roughly the same motion transformation, although its amplitude is reduced by the downsampling ratio experienced by the network layers. If the residual latent map of the previous frame is warped by the image motion field (appropriately downsampled and scaled), it will be a good prediction of the current latent to be transmitted. The entropy model of the latent can be conditioned on the predicted latent in addition to the super-prior latent and the already decoded current frame latent. This will significantly reduce the bits required to transmit the residual latent. Figure 6 Depicted is an example of the proposed entropy model, where a temporal entropy model is added. The spatial context model uses the decoded neighboring latent features of the current frame To estimate the spatial model parameters Super prior decoding features To estimate the hyper-prior parameter ψ t , and use the prior feature γ for estimating time t The potential characteristics of the current frame decoded motion on the previously decoded frame The three features are used together to estimate the parameters of the Gaussian or Laplace or multi-mixture entropy model, such as the mean and variance of the next potential points in the current frame. Note that the motion field of the current frame Scaling and downsampling are required to match these The spatial resolution of the dive site matches that of the

[0081] Training Improvements

[0082] In order to improve the inter-frame coding efficiency, the following training process is proposed:

[0083] 1) Large motion training: In order to handle large motion well, in the training step, it is necessary to add large motion training using random P frame jumps from 1 to n-1, where n represents the total number of frames in the training segment (e.g., n=7).

[0084] 2) Time distance modulation loss: The principle is to use higher weights for distant P frames:

[0085] For the time-distance modulation loss, the rate-distortion (RD) loss can be formulated as follows:

[0086] Loss = w*λ*MSE+rate,

[0087] where rate represents the achieved bitrate (e.g., bits per pixel), and MSE measures the L2 loss between the original frame and the reconstructed frame. The weight parameter "w" is initialized based on the temporal distance (t) between frames, and uses the following distance-based weights:

[0088]

[0089] The weight w i (where index i represents the iteration count (i.e., from 1 to 200k)) is the same for every frame in a group of pictures (GOP) and increases monotonically from 0 to 1 over a period of 200k iterations.

[0090] 3) MV Entropy Modulation Loss: The principle is to give higher weights to low-probability latent points (samples that are difficult to encode) than to high-probability latent points. The motivation comes from the focal loss of object detection (reference [3]). In object detection, there is always an imbalance between background samples and foreground samples, and the network always confuses the background with the foreground. In reference [3], the authors proposed a solution to this problem, namely, applying fixed weights to the cross entropy loss in order to increase the weights of hard samples, where hard samples are background samples in the image.

[0091] In an embodiment, the formula for weighted entropy loss is as follows:

[0092] Entropy loss = -log(p t )

[0093] Modulation entropy loss = (1-p t ) b *(-log(p t )),

[0094] in,

[0095] p t is the estimated probability of the latent symbol t

[0096] b decreases monotonically from 5.0 to 0.0 over a period of 200k iterations. At b = 0, the modulation entropy loss decreases to the normal entropy loss.

[0097] The overall coding gain is approximately 1.5% to 2.5% due to the improvements in the training process.

[0098] Syntax Examples

[0099] The proposed tools may be communicated from the encoder to the decoder using a high level syntax (HLS), which may be part of a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), slice header (SH), or as part of supplementary metadata such as supplementary enhancement information (SEI) data. An example syntax is depicted in Table 1. Alternatively, such signaling may not be required if a particular architecture or tool is predetermined and known to both the encoder and decoder.

[0100] Table 1 Example of high-level syntax for inter-frame coding adaptation

[0101]

[0102] inter_coding_adaptation_enabled_flag equal to 1 specifies that inter-coding adaptation is enabled for the decoded picture. inter_coding_adaptation_enabled_flag equal to 0 specifies that inter-coding adaptation is not enabled for the decoded picture.

[0103] joint_LC_MC_NN_enabled_flag equal to 1 specifies the use of a joint luma-chroma MC network to decode signals in the YUV domain. joint_LC_MC_NN_enabled_flag equal to 0 specifies the use of separate MC networks to decode signals in the YUV domain.

[0104] joint_LC_residue_NN_enabled_flag equal to 1 specifies the use of a joint luma-chroma residual network to decode signals in the YUV domain. joint_LC_residue_NN_enabled_flag equal to 0 specifies the use of a separate residual network to decode signals in the YUV domain.

[0105] attention_layer_enabled_flag equal to 1 specifies that the attention layer is enabled for the decoded picture. attention_layer_enabled_flag equal to 0 specifies that the attention layer is not enabled for the decoded picture.

[0106] attention_layer_MV_enabled_flag equal to 1 specifies that the attention layer is enabled for MV decoding. attention_layer_MV_enabled_flag equal to 0 specifies that the attention layer is not enabled for MV decoding.

[0107] attention_layer_residue_enabled_flag equal to 1 specifies that the attention layer is enabled for residual decoding. attention_layer_residue_enabled_flag equal to 0 specifies that the attention layer is not enabled for residual decoding.

[0108] temporal_motion_prediction_idc equal to 0 specifies not to use the temporal motion prediction network module to decode motion vectors. temporal_motion_prediction_idc equal to 1 specifies to use the temporal motion prediction and simple level network modules to decode motion vectors. temporal_motion_prediction_idc equal to 2 specifies to use the temporal motion prediction network module with warping of reference pictures to decode motion vectors.

[0109] num_ref_pics_minus1 plus 1 specifies the number of reference pictures used for the temporal motion prediction network module.

[0110] cross_domain_mv_enabled_flag equal to 1 specifies that cross-domain network motion vector decoding is enabled. cross_domain_mv_enabled_flag equal to 0 specifies that cross-domain network motion vector decoding is not enabled.

[0111] cross_domain_residue_enabled_flag equal to 1 specifies that cross-domain network residual decoding is enabled. cross_domain_residue_enabled_flag equal to 0 specifies that cross-domain network residual decoding is not enabled.

[0112] temporal_spatio_entropy_idc equal to 0 specifies that neither temporal nor spatial features are used for entropy decoding of inter frames. temporal_spatio_entropy_idc equal to 1 specifies that only spatial features are used for entropy decoding of inter frames. temporal_spatio_entropy_idc equal to 2 specifies that only temporal features are used for entropy decoding of inter frames. temporal_spatio_entropy_idc equal to 3 specifies that both temporal and spatial features are used for entropy decoding of inter frames.

[0113] Weighted Motion Compensated Inter Prediction

[0114] Figure 7 Depicted is an example process for weighted motion compensated prediction according to an embodiment. Figure 1 compared to, Figure 7 The following changes are depicted: replacing the MV encoder network with a MV+Inter weight map encoder network; replacing the MV decoder network with a MV+Inter weight map decoder network; and adding a "hybrid inter" network.

[0115] The motivation for this architecture is to allow both intra- and inter-coding in inter-frames without explicitly signaling the binary intra / inter flag used in traditional block-based coding. Explicit binary mode signaling is easier in traditional block-based codecs, but may not be directly effective for deep learning-based codecs (which encode spatially overlapping features in the latent domain).

[0116] Instead of using binary intra / inter coding flags, the basic idea here is to encode the explicit spatial weight map together with the motion information. Figure 7 As shown, these weights are used as point-wise mixing factors (α or α t ) (where α is in [0, 1]) is applied to the motion compensated (MC) inter-prediction samples Then the residual coding (r t ),in,

[0117] -x t is the source frame (uncompressed input)

[0118] -v t is the estimated uncompressed motion flow on the encoder side

[0119] - is the reconstructed motion flow output by the MV decoder network

[0120] - α is the spatial weight map encoded together with the motion information (output by the MV decoder network)

[0121] - is a motion compensated (MC) inter-predicted frame derived from the compressed stream using forward warping of a reference frame and a motion compensation network

[0122] - Represents a weighted MC inter-frame prediction frame obtained by point-by-point multiplication of motion compensated samples with the corresponding alpha weights

[0123] -r t is the residual signal obtained after subtracting the weighted MC frame from the source

[0124] - is the reconstructed residual on the decoder side after quantization and synthesis of the residual latent points

[0125] - is to reconstruct the residual The reconstructed frame obtained by adding the weighted MC inter-frame prediction

[0126] The process of weighted motion compensation prediction can be expressed as:

[0127] a.

[0128] b. as well as

[0129] c.

[0130] Among them, g a and g s is the residual encoder analysis network (e.g., Figure 1 and Figure 7 The residual encoder network in ) and the residual decoder synthesis network (e.g., the residual decoder network) (reference [1]).

[0131] Note that motion latent points carry information of both the compression flow and the spatial weight map.

[0132] The output of the motion compression network also includes a spatial weight map (α or α t ), the spatial weight map is used to blend motion compensation before residual compression. The residual is the difference between the original frame and the motion compensated pixels scaled by alpha at the pixel resolution level.

[0133] When α is 1, the decoded stream and the corresponding MC inter prediction are highly reliable, so the residual is purely inter-coded. When α is zero, the decoded stream and the corresponding MC inter prediction are highly unreliable, so the residual is purely intra-coded. When α is between 0 and 1, a mixture of intra and inter information can be encoded based on the quality of the inter prediction, which is similar to the combined intra-interprediction (CIIP) used in traditional codecs such as VVC.

[0134] The network is trained in an end-to-end manner for a joint rate-distortion (RD) loss function (e.g., loss = λ*MSE + rate), during which a stochastic gradient descent algorithm (such as an ADAM optimizer) is used to learn network parameters for optimal encoding of motion information, alpha weights, and residual information. The network is trained for large video datasets such as Vimeo-90k, using a batch size of 4, 8, or 16. A network trained on a generalized video dataset may not fully understand the selection of RD optimal motion information, alpha weights, and residual information for the actual source content to be tested. In an embodiment, this can be alleviated by content-specific encoder optimization, such as overfitting the encoder network or the encoded potential of a given source video through an iterative refinement process. This can help optimize the alpha weights, motion information, and residual information, thereby minimizing the RD loss of a given content to be tested as the encoder complexity increases.

[0135] In order to accurately train and encode the spatial weight map, additional inputs are included (enhanced) in the motion and alpha compression network (e.g., MV+inter-weight map encoder network), such as: using the previously reconstructed reference frame, using the current original input frame, and using a warped reference frame generated using uncompressed optical flow motion vectors. For simplicity, the warping process is not shown in the figure.

[0136] The residual compression architecture remains the same as in previous versions of DLVC (see, for example, Figure 1 ). For motion compensated blending, as discussed previously, additional variations can be implemented by passing this information (instead of the warped frames) to the decoder to improve training, i.e. by using an argument to the syntax variable mv_aug_type, e.g. with the following options:

[0137] 1.mv_aug_type=prev_com_res: Use the previous frame (reference frame) and the residual latent point of the reference frame as the enhancement input

[0138] 2.mv_aug_type = input: source frame (input), reference frame and reference frame residual latent point are used as augmentation input

[0139] 3.mv_aug_type=warped: source frame (input), reference frame, warped reference frame (using uncompressed stream), and residual latent of reference frame are used as augmentation input

[0140] 4.mv_aug_type=mc: Use source frame (input), reference frame, motion compensated reference frame (using uncompressed stream), and residual latent point of reference frame as enhancement input

[0141] Experimental results show that the proposed scheme can improve the compression efficiency by at least 2% and up to 5% compared with DLVC v.4.4 (reference [1]), depending on the category of the test image.

[0142] References

[0143] Each of the references listed herein is incorporated by reference in its entirety.

[0144] [1] Guo Lu et al., “DVC: ​​An end-to-end deep video compression framework”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[0145] [2] Z. Guo et al., “Soft then Hard: Rethinking the Quantization in Neural Image Compression”, Proceedings of ICML 2021, arXiv:2104.05168v1, April 12, 2021.

[0146] [3] Tsung-Yi Lin et al., “Focal Loss for Dense Object Detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, arXiv:1708.02002v2, February 7, 2018.

[0147] [4] Z. Hu et al., “FVC: A New Framework towards Deep Video Compression in Feature Space,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, arXiv:2105.09600v1, May 20, 2021.

[0148] [5] AK Singh et al., “A Combined Deep Learning based End-to-End Video Coding Architecture for YUV Color Space”, CVPR 2021, arXiv:2104.00807v1, April 1, 2021.

[0149] [6] HE Gilmez et al., “Transform Network Architectures for Deep Learningbased End-to-End Image / Video Coding in Subsampled Color Spaces”, arXiv:2103.01760v1, February 27, 2021.

[0150] [7] A. Mohananchettiar et al., “Multi-level latent fusion in neural networks for image and video coding,” U.S. Provisional Patent Application Serial No. 63 / 257,388, filed on October 19, 2021.

[0151] [8] Z. Sun et al., “Spatiotemporal entropy model is all you need for learned video compression”, arXiv:2104.06083 (2021).

[0152] [9] Z. Cheng et al., “Learched Image Compression with Discriedized Gaussianmixmix Liknules and Oced Modules,” CVPR 2020, arXiv:2001.01568v3, March 30, 2020.

[0153] Example Computer System Implementation

[0154] Embodiments of the invention may be implemented using a computer system, a system configured with electronic circuit systems and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to inter-frame coding using a neural network for image and video coding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to inter-frame coding using a neural network for image and video coding described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0155] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement methods related to inter-frame coding using a neural network for image and video coding as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transient tangible medium that carries a set of computer-readable signals including instructions, which, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention may take any of a variety of non-transient and tangible forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0156] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents of the components (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0157] Equivalents, extensions, alternatives and miscellaneous

[0158] Thus, example embodiments involving inter-frame coding using neural networks for image and video coding are described. In the foregoing description, embodiments of the invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the sole and exclusive indicator of the invention and the applicant's inventive intent is the set of claims issued in the specific form of this application, wherein such claim issuance includes any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes that are not expressly recited in a claim should not limit the scope of such claim in any way. Therefore, the specification and drawings should be viewed in an illustrative and not restrictive sense.

Claims

1. A method for processing a coded video sequence using one or more neural networks, the method comprising: receiving high-level syntax indicating enabling of inter-coding adaptation for decoding of a current picture; parsing the high-level grammar to extract inter-frame coding adaptation parameters; as well as Decoding the current picture based on the inter-frame coding adaptation parameters to generate an output picture, wherein the inter-frame coding adaptation parameters include one or more of the following: Joint luma-chroma motion compensation enable flag, which indicates to use the joint luma-chroma motion compensation network in decoding when the input picture is in the YUV color domain; a joint luma-chroma residual encoding enable flag indicating the use of a joint luma-chroma residual network in decoding when the input picture is in the YUV color domain; Attention layer enable flag, which indicates to use attention network layer in decoding; Temporal motion prediction enable flag, which indicates the use of a temporal motion prediction network for motion vector prediction in decoding; A cross-domain motion vector enable flag indicating that a cross-domain network combining motion vector and residual information is used in decoding to decode motion vectors; A cross-domain residual enable flag indicating that the residual is decoded using a cross-domain network that combines motion vector and residual information in decoding; and The time-space-entropy flag indicates whether entropy decoding uses only spatial features, only temporal features, or a combination of spatial and temporal features.

2. The method of claim 1, wherein: When joint luma-chroma motion compensation is enabled, Generate a motion compensated luminance component of the current picture using a joint luminance-chrominance motion compensation network having an input and chrominance components The input includes: The decoded motion of the current picture Luminance component of the previous reference picture Depend on The bilinear interpolation luma prediction picture represented by the previous reference picture, the chroma components of the previous reference picture and using downsampling and downscaling chroma motion Depend on Represents the bilinear interpolated chroma prediction component.

3. The method of claim 1, wherein: When using the attention network layer, an attention block layer is inserted between two deconvolution layers when decoding a P picture or a B picture, each of which consists of a deconvolution layer with upsampling followed by a nonlinear activation block.

4. The method of claim 3, wherein: Either insert an attention block layer after two consecutive deconvolution layers with no attention block in between, or insert an attention block layer after each deconvolution layer.

5. The method of claim 1, wherein: When using the temporal motion prediction network for motion vector prediction, the flow prediction neural network includes: A stream buffer receiving decoded stream information, followed by a first convolutional 2D network, followed by a series of one or more ResBlock 64 layers, followed by a second convolutional 2D network.

6. The method of claim 5, wherein: For a P picture, the output motion of the current picture is generated in the decoder include: Decoded stream motion using three past reference pictures ( and ) as an input to the flow prediction network to generate a first output; receiving quantized delta motion values ​​from an encoder; processing the quantized delta motion value using a motion vector decoder network to generate an input motion prediction value; and adding the first output to the input motion prediction value to generate an output motion vector for the current picture, Wherein, in the encoder, generating the quantized delta motion value comprises: Use current image (X t ), reference pictures and a motion estimation block to generate a second output; subtracting the first output from the second output to generate a delta output; and The delta output is processed through a motion vector encoding network and then quantized to generate the delta motion value.

7. The method of claim 5, wherein: For a B picture, the output motion of the current picture is generated in the decoder include: Decoded stream motion using two past reference pictures and one future reference picture ( and ) as an input to the flow prediction network to generate a first output; receiving quantized delta motion values ​​from an encoder; processing the quantized delta motion value using a motion vector decoder network to generate an input motion prediction value; and adding the first output to the input motion prediction value to generate an output motion vector for the current picture, Wherein generating the quantized delta motion value in the encoder comprises: Use current image (X t )、Reference pictures( and ), and a motion estimation block to generate a second output; subtracting the first output from the second output to generate a delta output; and The delta output is processed through a motion vector encoding network and then quantized to generate the delta motion value.

8. The method of claim 6, wherein: The input of the flow prediction network is preceded by a warping network, which includes: With motion vector input ( and ) is used to generate a first network through make Reverse twisted output; Cascade network, which is used for cascading and the output of the first network to generate a cascade network output; and The second network forward warps the cascaded network outputs to generate the input to the flow prediction network.

9. The method of claim 1, wherein: When a cross-domain network is used to decode motion vectors, the decoding includes: receiving residual latent points and motion vector latent points from an encoder; combining the residual latent point and the motion vector latent point in a motion vector decoder network to generate a motion vector to be used for motion compensation, wherein generating the motion vector latent point in the encoder comprises: Pixel values ​​of the current picture and previous reference pictures are used as input to an optical flow network and a motion vector encoder network to generate the motion vector latent points.

10. The method of claim 1, wherein: When a cross-domain network is used to decode the residual, the decoding includes: receiving residual latent points and motion vector latent points from an encoder; and combining the residual latent point and the motion vector latent point in a residual decoder network to generate a residual pixel value, wherein generating the residual latent point in the encoder comprises: Accessing residual pixel values ​​of the current picture and a previous reference picture; Access motion vector dive points; applying a motion vector decoder to the motion vector potential point to generate a reconstructed motion vector; and The residual latent point is generated based on the residual pixel value and the generated reconstructed motion vector.

11. The method of claim 1, wherein when the entropy decoding uses spatiotemporal features, the entropy decoding comprises: The decoded neighboring latent features of the current picture Applied to the spatial context model to generate spatial model parameters estimates; The super prior decoded features Applied to the super decoder to estimate the super prior parameters (ψ t ); The latent features of the previously decoded picture and the motion decoded from the current picture is applied to the warped block to estimate the temporal prior features (γ t );as well as Entropy model parameters of subsequent potential points are generated based on the spatial model parameters, the temporal prior features and the hyper priori parameters.

12. A method for improving the training of a neural network employed in inter-frame coding, the method comprising one or more of the following: Large sports training, including For a training sequence with a total of n pictures, large motion training is performed using random P frame jumps from 1 to n-1; Time distance modulation loss, where the rate-distortion loss is calculated as Loss = w*λ*MSE+rate, where rate represents the achieved bitrate and MSE measures the distortion between the original picture and the corresponding reconstructed picture, and the weight parameter "w" is initialized based on the temporal frame spacing as: where index i represents the iteration count in N training iterations.

13. The method of claim 12, wherein: Calculating the modulation entropy loss involves: Modulation entropy loss = (1-p t ) b *(-log(p t )), in, p t represents the estimated probability of the latent symbol t, and b decreases monotonically from 5.0 to 0.0 over the N training iterations.

14. A method for processing uncompressed video frames using one or more neural networks, the method comprising: Based on the uncompressed input video frame (x t ) to generate motion vector and spatial map information (α) of the uncompressed input video frame; Generate a motion compensated frame based at least on a motion compensation network and a reference frame used to generate the motion vector information applying the spatial map information to the motion compensated frame to generate a weighted motion compensated frame; generating a residual frame by subtracting the weighted motion compensated frame from the uncompressed input video frame; Generate reconstructed residual frames based on residual encoder analysis and decoder synthesis network wherein the residual encoder analysis network generates a coded frame based on quantization of the residual frame; as well as A decoded approximation of the encoded frame is generated by adding the weighted motion compensated frame to the reconstructed residual frame.

15. The method of claim 14, wherein: The spatial map information includes weights in [0, 1], where 0 indicates a preference for intra-frame coding only, 1 indicates a preference for inter-frame coding only, and weights between 0 and 1 represent hybrid intra-frame and inter-frame coding.

16. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, the instructions for executing the method according to any one of claims 1 to 15 using one or more processors.

17. An apparatus comprising a processor and configured to perform any of the methods of claims 1 to 15.

Images