Inter-coding prediction tools in video coding using deep learning

Abstract

Methods and systems are described for end-to-end video coding and decoding using deep learning. Proposed fusion schemes for reference pictures and motion vector features allow single-prediction models for deep-learning video coding (DLVC) to be extended to support multi-reference prediction coding as well. Example high-level syntax to support a variety of deep learning models for bi-prediction is described as well.

Description

INTER-CODING PREDICTION TOOLS IN VIDEO CODING USING DEEP LEARNING CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application may be related to Indian Provisional Patent Applications Ser. No.202411099633, filed on 16 December 2024, Ser. No.202411065408, filed on 29 August 2024, and 202411043070, filed on 3 June 2024, each of which is incorporated by reference herein in its entirety. TECHNOLOGY

[0002] The present document relates generally to images and video coding. More particularly, embodiments of the present invention relate to inter-coding prediction in video coding using deep learning tools. BACKGROUND

[0003] In 2020, the MPEG group in the International Standardization Organization (ISO), jointly with the International Telecommunications Union (ITU), released the first version of the Versatile Video Coding Standard (VVC), also known as H.266. More recently, the same group has been working on the development of the next generation coding standard that provides improved coding performance over existing video coding technologies. As part of this investigation, new coding techniques are also examined.

[0004] As used herein the term “end-to-end image compression neural network” refers to a neural network that jointly optimizes all components of a video compression system, from the point of input of uncompressed input to the encoder, to the point of output of a reconstructed output image of the decoder, using a single loss function.

[0005] As used herein, the terms “latent features” or “latent variables” denote features or variables that are not directly observable but are rather inferred from other observable features or variables, e.g., by processing the directly observable variables. In image and video coding, the term ‘latent space’ may refer to a representation of the compressed data in which similar data points are closer together. In video coding, examples of latent features include the representation of the transform coefficients, the residuals, the motion representation, syntax elements, model information, and the like. In the context of neural networks, latent spaces are useful for learning data features and for finding simpler representations of the image data for analysis.

[0006] As appreciated by the inventors here, improved techniques for video coding using deep learning are desired, and they are described herein. As used herein, the term inter- coding refers to coding a picture by using as reference other pictures in the same video sequence to take advantage of temporal redundancies among neighboring pictures.

[0007] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not be assumed to have been recognized in any prior art on the basis of this section, unless otherwise indicated. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] An embodiment of the present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0009] FIG.1A depicts an example framework for end-to-end video compression using a deep learning-based image and video compression (DLVC) architecture according to prior art;

[0010] FIG.1B depicts an example of a neural-networks model for image and video coding using a hyper prior model as part of its entropy model according to prior art;

[0011] FIG.2A depicts an example framework for P- and B-based coding according to an embodiment of this invention;

[0012] FIG.2B depicts an example of multi-frame-based context prediction according to an embodiment of this invention;

[0013] FIG.2C depicts an example of a multi-frame cross-group fusion module according to an embodiment of this invention;

[0014] FIG.2D depicts an example of a multi-frame motion-coding module according to an embodiment of this invention;

[0015] FIG.2E depicts an example of fusing reference motion features according to an embodiment of this invention;

[0016] FIG.2F depicts an example of fusing features of two reference frames to generate a fused reference frame according to an embodiment of this invention;

[0017] FIG.2G depicts an example of fusing features using separate uni-feature and bi- feature fusion adaptors according to an embodiment of this invention;

[0018] FIG.3 depicts an example end-to-end B-frame architecture for video coding using a vision sequence model according to an embodiment of this invention;

[0019] FIG.4 depicts an example workflow to generate fused reference pictures in the vision sequence model according to an embodiment of this invention;

[0020] FIG.5 depicts a coding example using open group of pictures (GOPs);

[0021] FIG.6 depicts an example of a criterion to select between forward and reverse coding of a GOP based on bits per pixel and PSNR; and

[0022] FIG.7 depicts an example process for bidirectional-open-GOP selection during coding. DESCRIPTION OF EXAMPLE EMBODIMENTS

[0023] Example embodiments that relate to applying inter-coding prediction tools in video coding using deep learning are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of the present invention. It will be apparent, however, that the various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating embodiments of the present invention. SUMMARY

[0024] Example embodiments described herein relate to applying video coding techniques using deep learning. Embodiments described herein describe fusion schemes for reference pictures and motion vector features which allow single-prediction models for deep- learning video coding (DLVC) to be extended to support multi-reference prediction coding as well. Example high-level syntax to support a variety of deep learning models for bi- prediction is described as well. DEEP LEARNING BASED VIDEO CODING (DLVC)

[0025] Deep learning-based image and video compression approaches (DLVC) are increasingly popular and it is an area of active research. Early DLVC models followed a traditional coding approach using the residual coding-based framework (Ref. [1]). Lately, aconditional coding framework using Deep Contextual Video Compression (DCVC) was proposed (Ref. [2]) (see FIG.1A). Improvements to this model include: DCVC-TCM (temporal context mining) (Ref. [3]), DCVC-HEM (hybrid spatial-temporal entropy modeling) (Ref.4]), DCVC-DC (diverse contexts) (Ref. [5]), and DCVC-FM (feature modulation) (Ref. [6]). The DCVC series of models supports a low delay P (LDP) coding structure as in the conventional codecs, such as AVC, HEVC, VVC, and the like. DCVC-FM not only outperforms VVC but also has comparable performance against ECM (Enhanced Compression Model) in JVET Exploratory Experiment when ECM uses low delay B (LDB) test condition. As is known, random access (RA) condition which employs B frames shows more than 20% gain compared against the LDP case. The characteristics of B frame type are still missing in the DCVC series. Herein, new embodiments that take advantage of B-frame type characteristics are proposed.

[0026] In a conventional codec, such as HEVC and VVC, the following definitions are common: • bi-predictive (B) slice: A slice that is decoded using intra prediction or using inter prediction with at most two motion vectors and reference indices to predict the sample values of each block. • predictive (P) slice: A slice that is decoded using intra prediction or using inter prediction with at most one motion vector and reference index to predict the sample values of each block.

[0027] FIG.1A depicts an example framework for end-to-end video compression using a deep contextual video compression (DCVC) architecture, where only a single (P) reference is being used (Ref. [2]). In FIG.1A, traditional residual coding (where residuals are generated using motion estimation and motion compensation between the current frame and a reference frame) is being replaced by context modeling, based on feature extraction from decoded frames.

[0028] Using a contextual encoder (105), input ^^(102) is encoded into latent codes ^^(106) which are quantized as ^^^(107) using a rounding operation. Reconstructed data ^^^(137) are generated via the contextual decoder (135).

[0029] In modern codecs, the distinction between B slices and P slices is rather blurred, with B slices allowing more than one inter prediction. As used herein, the term ‘B frame type characteristics’ is defined as describing: 1) a temporal feature and / or other feature propagation from more than one reference; 2) that the decoding order and the display order can be different. As used herein, the term ‘fusion’ denotes some form of merging of multiple inputs to generate a single output. An example of fusion is pixel interpolation among consecutive frames to generate a new frame. Fusion can be performed using both traditional image-processing techniques and deep learning networks. Fusion can happen in the feature domain and / or the pixel domain.

[0030] As an example, consider a group of pictures (GOP) with nine frames in display order denoted as: I0, B1, B2, B3, B4, B5, B6, B7, and P0, where I0 is coded on its own in intra mode, P0 uses I0 as a reference and is coded in P-mode, and each B frame uses two neighboring frames as a reference to be coded in B-mode. For example, for a given inter-prediction process, in order to guarantee to a decoder that all required references are available for inter-prediction, a possible coding order may be: I0, P0, B4, B2, B1, B3, B6, B5, B7.

[0031] In practice, higher temporal layers are coded with progressively higher QP than lower temporal layers. For example, when using rate control based on a Lagrangian-basedcriterion, such as ^ ^ ^ ^ ^^, where D denotes distortion (e.g., mean square error (MSE)), Rdenotes bit rate, and λ denotes a Lagrangian multiplier, which used for an encoder side Rate- Distortion (RD) cost optimization, then λ can be adjusted as: ^^ 0.57 x 2(QP-12) / 3 (8-bitdepth coding), or^ ^ 0.57 x 2QP / 3 (10-bitdepth coding).where QP denotes a quantization parameter. Revised random access configuration with improved QP setting for intra frames, B-frame QP setting based on QP offset and QP offset model parameters for GOP=8 and GOP=16, were proposed in Ref. [8] for the HEVC HM encoder and subsequently also used for VVC with higher GOP size of 32. Fusion Module for B frames

[0032] Consider a DLVC model, such as DCVC-DC of Ref. [5] (see also FIG.1A), trained using a P-frame architecture. In an embodiment, this trained model can be applied to also code B frames as follows. Consider a sequence of I, P, and B frames denoted (in display order) as I0, B1, P2, B3, P4, with a coding order of I0, P2, B1, P4, and B3. For coding P2 and P4, the conventional P-coding model can be applied. For coding B1 and B3 the following methods can be applied: During inference: B1 can be predicted by the fusion of I0 and P2, and B3 can be predicted by the fusion of P2 and P4. A fusion module could be trained or derived using the conventional methods.

[0033] In one embodiment, a conventional encoding method is applied. Using B1 as an example, picture level inference generates P1 (inferred from I0) and P2 (inferred from P2). Then for a given coding block c in B1: - get the corresponding coded block cp1 from P1 and compute the distortion between cp1 and block c in B1 as d1 (for example, using the MSE (mean square error) criterion and the like). Compute the bitrate r1 and distortion j1 = d1 + λ *r1. - get the corresponding coded block cp2 from P2 and compute the distortion from block c in B1 as d2. Compute the bitrate r2 and distortion j2 = d2 + λ *r2. - Generate a weighted block cp3 = w1*cp1+w2*cp2, (for example, w1 = w2 = ½), then the distortion from B1 is computed as d3. Compute the bitrate r3 and distortion j3 = d3 + λ *r3. - Finally, compare j1, j2, j3, and select the coded block (e.g., c1, c2, or c3) with the minimum distortion. The disadvantage of the above method is that one needs to signal the low-level syntax for each block in the bitstream to indicate whether prediction is from P1 or P2, and also whether bi-prediction is used. If no signaling is needed, one can always use cp3.

[0034] In another embodiment, as depicted in FIG.2F, a neural-network fusion model (250) may be trained to generate a fused reference picture from two reference pictures (P1, P2). The input is reference pictures P1 and P2, the output is the final prediction P. E1 and E2 denote feature extraction blocks which extract features F1 and F2 from P1 and P2. Each feature extraction block can be a CNN block. The extracted features are then concatenated and sent to a fusion block. The fusion block can be a multiscale CNN block or a transformer block. The trained model may be called a fusion model. During inference, the I model, the Pmodel and the new fusion model are used together to generate B pictures. The advantage of this method is that no signaling for individual frames is needed.

[0035] It should be noted that separate such neural-network fusion models (250) may be used to fuse distinct features of the reference frames. For example, in an embodiment, separate neural network fusion models may be used for: the fusion of motion-compensated warped reference frames, the fusion of reference-frame motion features, the fusion of reference-frame latent features, and the fusion of reference-frame temporal context features.

[0036] In another embodiment, as depicted in FIG.2G, the neural network fusion block (250) performs fusion of reference frame features using either uni-feature adaptors or a bi- feature fusion adaptor (260) based on the availability of specific features in the reference frames. For example, if one of the reference frames is an intra frame (I-frame) and the other reference frame is an inter frame (say P-frame), many of the reference features for the I- frame, such as motion features, motion latents, or temporal context features, are not available (set to ‘None’). In such a case, the fusion using concatenation of reference features (F0 and F1) shown in FIG 2F either cannot be performed or needs to be performed with F0 features being set to ‘zero’ followed by concatenation of the P-frame features (F1) and fusion. Fusion of reference features using ‘zeros’ for non-available reference features in FIG.2F results in lower performance at the GOP boundary as the same convolution layer block is used for multiple cases (IP, PI and PP). Hence, as depicted in FIG.2G, it is proposed to apply different feature adaptors, based on the availability of features for the different types of reference frames.

[0037] For example, when reference features for both reference frames are not available (262) (e.g., when both reference frames are Intra frames), the fusion model outputs ‘None.” In another case, when only one of the reference frame features (either F1 or F2) is available (264, 266) (e.g., when one frame is an Intra-frame and the other one is a P-frame), the concatenation and fusion steps are skipped and only the uni-feature adaptor is used. Finally (266), when both reference frame features (F1 and F2) are available (e.g., when both reference frames are P-frames), then, the reference features are concatenated and fused in the bi-feature fusion adaptor. In an example implementation, using separate uni-feature and bi- feature fusion adaptors for open GOP B-frame coding (e.g., GOP=32) provided an additional coding gain of 1% to 3%.

[0038] Compared to the bi-feature adaptor network which fuses distinct features of two reference frames for a B-frame, the uni-feature adaptor serves as a separate dedicated network to translate and propagate the reference features from a single inter reference frameto predict the current B-frame. Keeping distinct uni-feature and bi-feature adaptor networks helps in improving coding performance at open GOP and closed GOP boundaries having mixed reference frame types. For example, a simple bi-feature fusion adaptor network can be built using a simple feed-forward CNN layer with input channels being twice the number of input reference features after concatenation, while an equivalent uni-feature fusion adaptor network uses the same number of input channels as the number of reference features of a single frame. In both the cases, the number of output features would remain the same. The bi- feature adaptor network works as a fusion or blending network for the reference frame features while the uni-feature adaptor network works as a translatory propagation network of reference features.

[0039] In an embodiment, the neural network fusion model is trained separately from the P-DCVC model. For example, in an embodiment, a simplified training procedure can be performed as follows. Consider a pretrained I-model and P-model, trained using training data (say, N frames in total for each training clip, e.g., N=7), using a test pattern, say, in display order, I0, B1, P2, B3, P4, B5, P6, and in coding order of I0, P2, B1, P4, B3, P6 and B5. It is desired to train the fusion model only without changing parameters in the trained I-model and P-model. One can train all lambda together following P-DCVC model training. - during a first-round training iteration, only B1 training data is used for training the fusion model. For example, B1 is predicted based on I0 and P2; - during a second-round training iteration, both B1 and B3 training data are used for further refining the fusion model. For example, B3 may be predicted using P2 and P4; - during a third-round iteration, B1, B3 and B5 training data are used for further refining the fusion model. For example, B5 may be predicted using P4 and P6; - during a fourth-round iteration, learning rate is reduced, B1, B3 and B5 training data are used for further refining the fusion model.

[0040] In another embodiment, the neural network fusion model may be trained together with the P-DCVC model to improve its overall coding efficiency. For example, a simplified training procedure can be performed as follows.

[0041] Consider a pretrained I-model and P-model, trained as described earlier using the same frame coding order cadence, I0, P2, B1, P4, B3, P6 and B5. - during a first-round iteration, the P-model is fixed and B1 training data is used for training the fusion model, say by using I0 and P2 to predict B1;- during a second-round training iteration, the P-model is fixed, and both B1 and B3 training data are used for further refining the fusion model. For example, B3 may be predicted using P2 and P4; - during a third-round training iteration, the P-model is fixed, B1, B3 and B5 training data are used for further refining the fusion model; - for a fourth-round iteration, the fusion model is fixed, and P2, B1, P4, B3, P6 and B5 training data are used for refining the P model; - for a fifth-round iteration P2, B1, P4, B3, P6 and B5 training data are used for refining both the P model and the fusion model; - for a sixth round iteration, learning rate is reduced, and P2, B1, P4, B3, P6 and B5 training data are used for refining P model and fusion model.

[0042] In another embodiment, a training procedure may include these sequential steps: using a pretrained P-model, first train the fusion model, then refine the P model, then refine both the P model and the fusion model. One may also repeat this core training sequence every time one increases the number of B frames in training, such as when using B1 only, when using B1 and B3, and when using B1, B3, and B5. The above training methods can be extended using any GOP sequence that combines I, P, and B frames.

[0043] Outside of the post-processing fusion module, this approach uses existing DCVC models as is. Next, a modified DCVC model supporting B-frame encoding is proposed. DCVC B-Model

[0044] In an embodiment, the existing DCVC model is extended to natively support multiple references. Fusion in multi-reference inter prediction is handled through an explicitly coded β map which represents a weight map that captures the relative reliability of one reference versus the other in predicting the corresponding pixel of the current frame. The same multi-reference architecture can be also used for regular low-delay B and low-delay P configurations based on whether the β map is used as weight map for bi-prediction or alternatively used as a reference frame index map by using it as one-hot coding map which indicates choice of specific reference frame for P-frame uni-prediction. While, for clarity and brevity, embodiments described here refer or depict only two references frames, the proposed architectures can be easily extended to support more than two reference frames without anyloss of generality. Furthermore, while the proposed enhancements are discussed in the context of DCVC, they are equally applicable to other end-to-end deep learning architectures.

[0045] FIG.2A depicts an example of generalized P / B picture codec framework, where ^^^^^^^^215^and ^^^^^^(210) are respectively the end-to-end trained multi-reference motion and conditional frame coders (Ref. [5]). As discussed in Ref. [5] “Motion coder,” ^^^^^^^, “Frame coder,” ^^^^^^, and “Temporal context extractor,” ^^^^^^^^^, in P-frame coding, comprise the three key steps in a coding pipeline as follows. First, ^^^^^^^estimates the current frame motion vector^(142) using an optical flow network and current reference frame (140) (^^^!"^, then^(142) is encoded into the quantized motion latentrepresentation, ^^^$^ , and decoded as ^^ (144). Next, using ^^ and a propagated feature %^!"(146) from a previous frame, %^^^^^^^^(205) extracts a motion-aligned temporal context feature &^. Finally, conditioned on &^, ^^^^^^encodes ^^into the quantized latent representation ^^^(107) and is decoded as ^^^(137).

[0046] A similar approach is followed in FIG.2A but takes also into consideration that multiple frames (as an example here, with no limitation, two) may be used now. Therefore, now, ^^^^^^^uses two feature motion vectors, %"^$and %'^$. These are higher dimensional features from which the motion vectors are decoded. These features are used in conditioning the coding of the next frame’s motion vectors for which their frame is a reference. These are extracted using the current (^^ frame and two reference frames (^"and ^'^ to generate two estimated motion vectors ^"and ^'and a feature motion vector %^^$for the current frame ^. Hence, consider F denoting the reference features for frame coding whereas %^^$denotes reference features for motion vector coding.

[0047] Next, now, %^^^^^^^^ uses two sets of (^^, ^^ , and %^ ) parameters to generatecontext feature C, which, as before, is used, together with input ^, as input to ^^^^^^(210) to generate feature F , predicted input ^^ and latents ^^.

[0048] Each reference frame, ^^^, chosen from previously decoded frames, has associated with it a propagated reference feature, %^, and decoded motion vectors, ^^. The temporal context prediction module ^^^^^^^^^(205) fuses the information fromreference frames to produce the temporal context features, C. Though the figure shows only one scale for clarity, C can be generated at more than one scale for capturing a richer context, in which case the fusion can be done at each scale separately. Multiple references should produce more informative temporal context features, resulting in more efficient compression. The motion fields corresponding to the reference frames are computed and coded by the ^^^^^^^mv coder(215) depicted in FIG.2D a). It jointly encodes the motion fields conditioned on the combined temporal context information, %^$, to exploit their redundancy.

[0049] FIG.2B depicts an example workflow for the %^^^^^^^^block (205) of FIG.2A according to an embodiment. As depicted in FIG.2B, it includes two instantiations of a P- only %^^^^^^^^workflow, but combined using a weighted map, ). For the i-th reference, the reference frame ^^^, and reference features %^are warped by the corresponding motion field to do motion compensation. For each reference frame, motion offsets (*^) and maps (+^) are produced to refine the corresponding motion field. Features obtained by cross-group fusion from each refence are combined using a convolutional network that depends on the temporal id (tid) corresponding to the level of the B-frame hierarchy. Optionally, a weight map β in [0, 1] may be used to capture the relative reliability of one reference frame versus the other in predicting the corresponding pixel of the current frame. A beta map is defined at the resolution of the motion y latents. The ),-maps are obtained by upsampling the β maps transmitted as additional channels to the hyper prior information for the motion latents. In the case where there are more than two reference frames used, there will be as many β maps, one for each reference. Upsampling of a beta map may be done via bilinear interpolation or other techniques known in the art.

[0050] FIG.2C depicts an example of the multi-frame cross-group fusion module (220) which does the fusion of temporal context information from multiple reference frames. The design follows the same architecture as the P-only cross-group fusion module in Ref. [5], except that now two separate cross-fusion paths are merged using the ),-maps. That is, features from each reference are weighted respectively by the upsampled weight maps ),-and 1. ),- before the final fusion to generate C. In the case of more than two referenceframes, features from each reference frame will have their own weight map. The contextual encoder and decoder utilize this richer context information to compress and decompress video frames more efficiently. Other aspects of their architectures remain unchanged.

[0051] During compression of each frame, the number of motion fields to be computed and compressed equals the number of reference frames. Compression of motion fields is generated by the ^^^^^^^module (215). FIG.2D depicts an example of its implementation according to an embodiment. The module computes the optical flow of the current frame with respect to each of the reference frames ( ^^^) to generate motion vectors^. Next, the conditional autoencoder, ^^$(225), generates quantized motion vectors ( ^^) and their corresponding %^^$information. Since there is considerable amount of redundancy betweenthese motion fields, it is more efficient to compress them jointly, conditioned on the combined temporal context information, C, obtained from the fused motion features, %^$, to exploit the redundancy.

[0052] The fusion of the reference motion features corresponding to each of the reference frames is achieved by a convolutional layer whose details are shown in FIG.2E; however, sequence-based methods, such as Mamba (Ref. [9]), to be discussed later on, are also viable. Optionally, the different reference views can be weighted based on the corresponding weight maps, )^,-$, which indicate their reliability in predicting the current motion vectors. The )^,-$maps are by upsampling the maps transmitted as additional channels to the latents. In an embodiment, weight maps, )^prior for the frame,-$, areby a fixed weight / ^ ^ 1 / 1, for i = 1, 2, …, N, where N denotes the number of referencesbeing used. Hence, for two reference frames / " ^ / ' ^ 0.5. This the design,and since motion take up only a relatively small percentage of the total bits, this shouldn’t significantly affect the overall bit-rate performance. As depicted in FIG.2E, given a first motion feature (%"^$^ for the first reference picture, and a second motion feature (%'^$^ for the second reference picture, the output fused motion feature (%^$) is generated as: %^$ =2*324563456^ )^$,- ∗ %^$" , 81 . )^$,- 9 ∗ %^$' ).

[0053] on the hierarchylevel (tid) of the B-frame prediction. This helps in adapting the prediction to the different motion directions and magnitudes at different levels of the hierarchy. The motion vectors are then compressed and decompressed conditioned on this fused motion context information, (input C to the ^^$module), which should result in more efficient compression of motion vectors. Positional encoding modulation

[0054] In another embodiment, a positional encoding modulation that is invariant to the input-size coined canonical positional embedding is proposed. This modulation is to be applied to the image features before they are processed by any fusion module, to better represent the spatial relationships in the data. The concept of sinusoidal positional embeddings can be found in Ref.

[0012] . In Ref.

[0012] , the embedding uses an absolute reference to the token position in the context of transformers. That kind of embedding, if used as originally suggested, can lead to model overfitting. This is because only a limited set of videoresolutions may be available during training, which will result in the positional embedding generating unknown embedding patterns for arbitrary resolutions during deployment. This issue is addressed by normalizing the positional ranges based on the feature’s width and height.

[0055] In detail, let the feature tensor %"have dimensions (L, H, W), where L, H, and W represent the number of layers, height, and width, respectively. Then, the feature modulationwill produce %:" ^ %" ^ ;, where ; is the proposed modulation tensor that depends on thefeature tensor dimensions and is defined as: KLMNO^PQQQ ^ìsin F'GHI × 6! QR S , if i %2=0 and i<L / 21]1]1]cxd of the M tensor can be 32x32, 64x64, 128x128, or 256x256. In an example embodiment, without limitation, L= 64 to merge the features and the latent of the motion vectors, L=128 to merge the latent of the input frame, and L=12 to merge the warped frames. Open Group of Pictures (GOP) configurations

[0056] In addition to the described B-frame architecture, this section proposes a number of auxiliary techniques that can improve coding efficiency in end-to-end trained architectures. One of such strategies is the open-GOP configuration in which frames inside a group of pictures (GOP) are allowed to access reference frames and features from frames that are present in a neighboring GOP. One example embodiment of this idea is presented in FIG.5, which depicts three GOPs, each with four frames. The numbers underneath the coding type (e.g., I, P, B) denote encoding order. The other numbers denote the display order. Under the closed-GOP model, no frame in a GOP may use as references frames from another GOP. In this open-GOP example, the last frame of GOP1 may use as a reference the I-frame of GOP2, and the last frame of GOP2 may use as a reference the I-frame of GOP3. Using pseudocode, this illustrative example may be described as: Open-GOP (IBPB):num_gops ← input / / Number of input GOPs

[0057] The above scheme can easily be generalized to any arbitrary IPB-based GOP coding configuration. While the majority of frames will be using references only within the GOP, tail frames of a GOP could use as references frames from the next GOP.

[0058] The above scheme assumes that the coding order follows the typical display order, that is, “left to right.” In another embodiment, coding gains may be observed by allowing a bidirectional coding configuration, where pictures in a GOP are coded using either the traditional forward order or a reverse (or backward) coding order. For example, in FIG.5, in GOP 1, in a reverse coding order, starting from frame 4, coded as an (I,1) frame, frames 3, 2, and 1 may be coded as (B,3), (P,2), and (B,4). With this configuration each GOP is encoded twice, first, in the forward direction and next, in the reverse direction. The reverse direction encoding can be achieved by flipping the input GOP along the time dimension, thus coding the GOP as if reproduced in reversed order. After both coded versions are available to the encoder, a decision is made regarding which of the two encodings to transmit. Such decisionis achieved by using the input GOP as reference and extracting rate and distortion metrics for each of the encodings. With the help of a discriminating function, the encoding with the best rate-distortion pair is selected.

[0059] An example embodiment of the discriminating function is presented in FIG.6, where bits per pixel (BPP) and PSNR may be used as metrics for bit rate and distortion, respectively. As an example, a linear decision boundary (605) is created based on the forwardpass values (ghhi , hj1^i) representing the anchor (e.g., 0,0). If the relative position of thebackward (or reverse) metrics (hj1^k , ghhk) falls to the left of the boundary 605, then thereverse pass will be chosen, otherwise, the forward pass will be selected. In an embodiment, this boundary may be expressed as hj1^l= w ghhl, where w (e.g., w = 1) denotes a weight between coding gain in bits per pixel and PSNR,hj1^l = hj1^k − hj1^i , and ghhl = ghhk − ghhi. Thus, the decision criterion maybe expressed as: if hj1^l> w ghhlapply reverse coding; else apply forward coding.For example, for ghhl = 0, reverse coding is applied only if hj1^k > hj1^i .

[0060] Alternative linear or non-linear discrimination (e.g., log-based) functions may also be defined based on experimental data.

[0061] In order for the decoder to recover the video in the original order, an additional bit is sent with high-level syntax or metadata indicating whether or not the GOP should be flipped when reassembling the video. Using pseudocode, an example bidirectional coding configuration can be described as: Bidirectional GOP coding:encode_metrics_backward = get_bpp_and_psnr(gop_id, backward_encoding) if encode metrics forward are better than encode metrics backward:, P coding techniques for a hybrid bidirectional-and-open GOP scheme. This hybrid approach codes the odd GOPs through the already described bidirectional coding, but allows the even GOPs to use a bidirectional approach while accessing information from the closest coded odd GOPs. An example coding may be described using the three GOPs in FIG.5.

[0063] Here, GOP 1 and GOP 3 (the odd numbered GOPs) are encoded first as closed- GOPs using the bidirectional coding configuration, that is both forward and reverse mode and selecting the best one. Next, GOP 2 will be coded as an open-GOP, using information from GOP 3 for the forward pass and information from GOP 1 for the backward pass. For example, for the forward pass, frame8, coded as a B picture, may use as one reference frame9 from GOP3. Similarly, in the reverse pass, frame5, coded as a B picture, may use as a reference frame4 from GOP 1. After applying the discriminating function to these two encodings, the best option will be selected and transmitted to the decoder.

[0064] FIG.7 depicts an example process flow for the bidirectional-open GOP scheme. Given a GOP identifier (ID) (705), first, one determines whether it is an odd or even GOP. If it is an odd-numbered GOP, it is processed as a closed GOP in both the forward and reverse modes (710), and the best mode is selected according to a mode-selection criterion. The features of the first and last frames of the GOP are also stored, say in an open GOP buffer (730). Otherwise, if it is an even-numbered GOP, it is processed as an open GOP (720). In the forward mode, if available, it may consider for coding the first frame of the next GOP, and in the reverse mode, if available, it may consider for coding the last frame of the previous GOP. Then (725), the best mode is selected according to the mode-selection criterion.

[0065] Using pseudocode, this example process may also be described as: Bidirectional Open-GOP:frame_coding_order ← input # Coding order within a GOP gop coding order ← input # Coding order of GOPsreferences for the last frame in an open-GOP, thus if the first frame of gop_id+1 (or the last frame of gop_id-1) is not an I or a P frame, one may skip encoding in open-GOP mode or one may select the next suitable I or P frame.

[0067] Persons skilled in the art will appreciate that while the examples presented herein relate mostly to simple IBP GOPs, the proposed techniques are applicable to any type of I, P, and B hierarchical coding, with arbitrary GOP size, such as IBBP, IBBPBBP, IBBBP, and the like.Training considerations

[0068] Training of the proposed hierarchical B-frame framework proceeds by processing mini batches of training sequences, one GOP at a time. As B-pictures required reordering, all the frames in one GOP are read and buffered before training / encoding. Within a GOP, the processing happens in the conventional B-frame hierarchical coding order as illustrated earlier. A GOP size, say, 4, 8, or higher, may be selected based on the memory constraints of the machine used for training. The decoded picture buffer (dpb) stores the multiple reference frames and features similar to a conventional VVC encoder. In the loss function, λ is weighted by the weight factor, / ^^ncorresponding to the temporal id (tid) or the level in the hierarchy of the current frame. Note that the loss function in deep learning-based videocoding typically weights the distortion with lambda in the form of ^ = ^ ^ ^^. So thelambda weighting scheme in deep learning video coding would be inversely proportional to the equivalent QP derived lambda of a conventional codec.

[0069] Table 1 depicts example values for sample DLVC lambda weights / ^^nfor random access B-frame coding for GOP size=8, using an initial QP = 32 and the VVC lambda scaling of λ= 0.57 x 2QP / 3 .Table 1. Example of DLVC lambda weighting for B frames and GOP = 8 Frame POC QP VVC λ DLVC λ

[0070] Similar weighting mechanisms can also be derived for other GOP sizes based on the equivalent hierarchical QP values used in conventional codecs.End-to-end B-Model using a sequence-based method

[0071] As described in Ref. [2] and Ref.

[0011] , the entropy model in FIG.1A, which is used to encode the quantized latent codes ^^^, includes two paths: a traditional arithmetic encoder (AE) path and a hyper prior model that includes a hyper prior encoder (HPE) and a hyper prior decoder (HPD) to enhance coding efficiency. Details of such an implementation are depicted in FIG.1B.

[0072] As depicted in FIG.1B, given input images (102), there are two main sub- networks: an autoencoder, which learns a quantized latent representation of the images (Contextual encoder (105) and Contextual decoder (135) blocks), and a sub-network which is responsible for learning a probabilistic model over quantized latents (^^) (107) used for entropy coding. The subnetwork combines the Temporal Feature Model (125), with the hyper-network (Hyper Encoder and Hyper Decoder blocks 110 and 115), which learns to represent information useful for correcting the context-based predictions. The data from these two sources is combined by an entropy modeling network (130), which generates entropy parameters (e.g., the mean and variance).

[0073] In FIG.1B, the arithmetic encoding (AE) blocks yield the compressed representation of the latent symbols (^^) (107) coming from the quantizer (Q), which is stored in a file. Therefore, at decoding time, any information that depends on the quantized latents may be used by the decoder once it has been decoded. For the context model 125 to work, it can only access previously decoded latents.

[0074] Real-valued latent representations are quantized (Q) to create quantized integer- valued latents (^^) (107) and quantized hyperlatents ( p)̂ (119), which are compressed into a bitstream using an arithmetic encoder (AE) and are decompressed by an arithmetic decoder (AD). Blocks with a cross-hatch background correspond to the components that are executed by the receiver to reconstruct an image (137) from a compressed bitstream.

[0075] As discussed in Ref.

[0011] , a hierarchical prior (or hyperprior) z (112) is used to improve the entropy model of the latents by capturing their spatial dependencies. Such a model allows for end-to-end training, which includes joint optimization of a quantized representation of the hyperprior, the conditional entropy model, and the base autoencoder. Under this model, the compressed hyperpriors may be added to the generated bitstream as side information, which allows the decoder to use the conditional entropy model. In this way, a separate entropy model (120) of the hyperpriors allows for a richer and more accuratemodel. After a picture is decoded, its representation is stored in the decoded picture buffer (DPB). Data in the DPB may be used to encode the next input frame.

[0076] As depicted in FIG.3, in another embodiment, the P-end-to-end model of FIG.1B is modified to support coding of B pictures using a vision sequence model (305). Depending on high-level signaling (HLS), to be discussed later, one can implement the vision sequence model using either a Transformer-based architecture, or a Mamba / state-space model-based coding architecture. While proposed details refer to the Mamba architecture, the proposed embodiments can be easily extended to a Transformer-based architecture.

[0077] Mamba is a state-space model architecture that operates over sequences similarly to Transformers but without the need for quadratic attention (Ref. [9]). This approach allows one to store and process multi-reference features and fuse them in the proposed B-frame coding architecture with minimal overhead.

[0078] As depicted in FIG.3, compared to FIG.1B, in system (300), new components include two new decoded picture buffers for reference frames (e.g., DPB (ref1) and DPB (ref2)) and the vision sequence model (305) to be explained next. The fused reference buffer picture buffer (t) (310) is the same as a P-frame picture buffer in model (150), except it now represents a fusion of the reference frames used to code a B picture, as generated by the vision sequence model.

[0079] To generalize, given RGB inputs, the input to this model is a sequence of videoframes packed into an input tensor of size ^1, rsh_juvw, 3, c, d^. Where rsh_juvw is theinput GOP size of the model (e.g., 8), 3 denotes the number of color channels, and H and W are the height and width respectively of pictures in the video sequence. For example, using the example discussed earlier, consider coding order I0, P0, B4, B2, B1, B3, B6, B5, B7.

[0080] First, the initial frame in the GOP, e.g., frame I0, is encoded via a pre-trained intra-frame model without information of future or previous frames. Next, as in the examples discussed earlier, one needs to code frame P0, the last frame in the GOP, with a pre-trained P- frame encoder, e.g., one based on a DCVC-DC architecture (Ref. [5]) (see FIG.1A), or system (150). This P-frame model takes as input from the I-frame model its decoded I0 frame to be used as the reference frame for P0. This information, generated from the contextual decoder (135), is packed into a DPB buffer, which includes:1. The reconstructed reference I0 frame (^^^ ^1,3, c, d^.2. The reference frame feature (%^^ ^1, ^, c, d^, where D (the size of a dimension in thesensor) is generally 48. An example of such a tensor could be (1,48,256,256). This would denote a 4D-tensor with 48256x2562D planes.3. The reference motion vector Feature (%^$^ ) ^1,2, c, d^.4. The reference ^^ ^1, ^, c / / 4, d / / 4^ where D is generally 128, and “ / / ” denotes integerdivision, e.g., a / / b = int(a / b).5. The reference ^^^$ ^1, ^, c / 4, d / 4^, where D is generally 64.These contents will then be used to help encode frame B4, which will have as input the reference frame from frame I0 and the reference frame from frame P0, as well as the contents of the whole DPB for frame P0.

[0081] In an embodiment, a vision sequence model (305) is applied to fuse together these two reference frames. FIG.4 depicts an example workflow for the vision sequence model according to an embodiment. As depicted in FIG.4, given two (or more) reference frames (420), in step 425, their tensor-packed representations are combined to create a new tensor ofshape ^2, ^, c, d^, where 2 is the temporal dimension (T). Next (430), one applies apatchify algorithm to generate non-overlapping, spatio-temporal patches like those used inRef.

[0010] . This allows one to have an output sequential tensor of the shape ^3^, 3z, 3{, |^,where d is the dimensionality of each token as decided by the patchify algorithm, and for atubelet of dimension t x h x w, 3 ^ ] I^ = }^~ , 3z = }z~ , 3{ = }{~ are the number of tubelets ineach input dimension, ^, c, and d. Nexta vision transformer model (say, a Transformer, or a mamba vision model) to learn the relative importance of each spatio- temporal token position to each other one. The output of either of these models will be asequential tensor of shape ^3^, 3z, 3{, |^ which will then (step 440) be de-patchified into atensor of size ^1,3, c, d^ which will be used as the fused reference frame for encoding frameB4 (step 445).

[0082] Every other frame within the GOP will be encoded as a B-frame model in the pattern suggested by traditional Hierarchical B-frame models, except, one will simply append each previously encoded reference frame to the previous packed tensor of shape^1, ^, c, d, 3^ rather than simply using two reference frames. This way, the sequential basedmodel learns how to best use the entire GOP history, up to the frame to be encoded, to encode the next B-frame.

[0083] Reference frame features are also fused via the same method, except the initialpacked tensor will be of shape ^1, ^, c, d, ^^ where ^ is the number of channels for thereference feature. Summary of selective nomenclature • x : input frames • ^^ : reconstructed frames, reference frames • %: frame feature, use to code latents (y) and generate context C • %^^$: motion vector feature, used to encode motion vector (v) • C : context used to encode latents • ^^: decoded latent of the input frame. Used with C to generate reconstructed frames ^^^^ and frame features (%) • ^^^$: decoded latent of a motion vector. Used with C to generate decoded motion vectors ( ^) and motion vector features (%^^$) • ): beta map Training considerations for the sequence-based model

[0084] As in the B-fusion model (250) described earlier, parameters of the sequence- based model can be trained either independently of the I / P-model or in combination with the I / P-model using the sequential training modes described earlier.

[0085] The usage of separate uni-feature adaptor and bi-feature fusion adaptors based on the availability of reference frame features, as depicted in FIG.2G, is also applicable for sequence-based Mamba fusion models. Example High Level syntax

[0086] An example HLS syntax using the invention for inter decoding process is described in Table 2. The HLS can be communicated to a decoder at a variety of levels in the bitstream syntax, such as at the stream level, the picture level, the slice level, and the like, or via metadata, such as supplemental enhancement information (SEI) messages.Table 2. An example of HLS for inter coding in Picture Header (PH) picture_header_structure( ) { Descriptor …access point) picture. ph_irap_pic_flag equal to 0 specifies that the current picture is not IRAP picture. ph_uni_ref_pred_flag equal to 1 specifies that the current picture uses only one reference index and one motion vector for conditional context generation. ph_uni_ref_pred_flag equal to 0 specifies that the current picture may use more than one reference index and / or more than one motion vector for conditional context generation. ph_model_type_idc specifies a model type index for the current picture. Note: in an embodiment, as an example, the following ph_model_type_idc values may be defined: • 0: Simple interpolation of two reference frames • 1: DCVC B-Model with blending bias • 2: Sequence-based model using a Transformer • 3: Sequence-based model using mamba ph_model_id indicates model identifications. For example, in embodiment, for any model type, one id may define a high-complexity model (e.g., HOP (High Operation Point)), and another id may define a lower-complexity model (e.g., LOP (Low Operation Point)).References Each one of the references listed herein is incorporated by reference in its entirety. The terms JVT, JCT-VC, and JVET refer to Joint Video Experts Teams of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29. [1] Guo Lu, et al. "DVC: An end-to-end deep video compression framework." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [2] Jiahao Li, Bin Li, and Yan Lu, “Deep contextual video compression,” Advances in Neural Information Processing Systems,” arXiv:2019.15047v1, 30 Sept.2021. [3] Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu, “Temporal Context Mining for Learned Video Compression,” IEEE Transactions on Multimedia, 2022, arXiv:2111.13850v1, 27 Nov.2021. [4] Jiahao Li, Bin Li, and Yan Lu, “Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression,” ACM MM 2022, arXiv:2207.05894v1, 13 Jul.2022. [5] Jiahao Li, Bin Li, and Yan Lu, “Neural Video Compression with Diverse Contexts,” CVPR 2023, arXiv:2302.14402v1, 28 Feb.2023. [6] Jiahao Li, Bin Li, and Yan Lu, “Neural Video Compression with Feature Modulation,” CVPR 2024, arXiv:2402.17414v1, 27 Feb.2024. [7] H. Schwarz, D. Marpe, and T. Wiegand, “Hierarchical B Pictures,” JVT-P014, Joint video Team of MPEG, Poznan, Poland, July, 2005. [8] K. Andersson, P. Wennersten, J. Samuelsson, J. Ström, P. Hermansson, M. Pettersson, “AHG 3 Recommended settings for HM,” JCTVC-X0038, JCT-VC Team of MPEG, Geneva, Switzerland, May, 2016. [9] A. Gu, T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” arXiv:2312.00752, Dec.2023.

[0010] A. Arnab, et al., “ViViT: A Video Vision Transformer,” arXiv:2103.15691v2, 1 Nov. 2021.

[0011] A. Arora, et al., “Threshold models for skipping entropy coding in end-to-end image compression using neural networks,” PCT Application PCT / US2024 / 31854, filed on 31 May 2024, (Dolby Ref. D23055).

[0012] A. Vaswani, et al., “Attention is all you need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. EXAMPLE COMPUTER SYSTEM IMPLEMENTATION

[0087] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or apparatus that includes one or more of such systems, devices or components. The computer and / or IC may perform, control, or execute instructions relating to applying video coding techniques using deep learning, such as those described herein. The computer and / or IC may compute any of a variety of parameters or values that relate to applying video coding techniques using deep learning as described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0088] Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder, or the like may implement methods related to applying video coding techniques using deep learning as described above by executing software instructions in a program memory accessible to the processors. Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non-transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0089] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.

[0090] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs): EEE 1. A method to code video sequences using a neural-network architecture, the method comprising: receiving a sequence of input pictures to be coded using intra (I) pictures, predictive (P) pictures, and bi-predictive (B) pictures; encoding and decoding a first input picture using a deep learning video coding (DLVC) network trained to encode I or P- only pictures and generating a first output picture representing a decoded approximation of the first input picture; encoding and decoding a second input picture using the DLVC network and generating a second output picture representing a decoded approximation of the second input picture; combining the first output picture and the second output picture based on a third input picture to be coded as a B-picture to generate a fused picture; and encoding the fused picture using the DLVC network to generate a coded representation of the third input picture with references to the first input picture and the second input picture. EEE 2. The method of EEE 1, wherein generating the fused picture further comprises: for a coding block in the third input picture: measuring a first coding cost to code the coding block using a first reference block from the first output picture; measuring a second coding cost to code the coding block using a second reference block from the second output picture; measuring a third coding cost to code the coding block using a third reference block representing a weighted average of the first reference block and the second reference block; computing a minimum coding cost among the first, second, and third coding costs; and selecting among the first reference block, the second reference block, and the third reference block the one which yields the minimum coding cost and apply it to encode the coding block in the third input picture. EEE 3. The method of EEE 2, wherein for a computed bit rate R when coding an input picture, and distortion D, a coding cost can be computed as J = R+ λ*D, where λ denotes aLagrangian multiplier and D represents an error metric between the input picture and a decoded representation of the input picture. EEE 4. The method of any one of EEEs 1 to 3, wherein generating the fused picture comprises inputting the first output picture and the second output picture into a neural network (NN) fusion model to generate the fused picture. EEE 5. The method of EEE 4, wherein the fusion model comprises a feature extraction block and a fusion block, and generating the fused picture further comprises: generating a first frame feature representation of the first output picture using the feature extraction block; generating a second frame feature representation of the second output picture using the feature extraction block; concatenating the first frame feature and the second frame feature representations to generate a combined frame feature representation; and applying the combined frame feature representation to the fusion block to generate the fused picture. EEE 6. The method of EEE 4 or EEE 5, wherein the fusion model is trained separately from the DLVC network or in combination with the DLVC network. EEE 7. The method of any one of EEEs 4 to 6, wherein the fusion model comprises a uni- feature adaptor and a bi-feature fusion adaptor and generates an output fused picture based on availability of reference features in the first output picture and the second output picture. EEE 8. The method of EEE 7, wherein if neither the first output picture and the second output picture have reference features, the fusion model generates a “none” output, else, if only the first output picture has reference features, it generates an output fused picture based on the reference features of the first output picture and the uni-feature adaptor, else, if only the second output picture has reference features, it generates an output fused picture based on the reference features of the second output picture and the uni-feature adaptor, else, it generates an output fused picture based on the reference features of the first output picture, the second output picture and the bi-feature adaptor.EEE 9. The method of any one of EEEs 4 to 8, wherein the fusion model comprises a video sequence model (305) comprising: a patchify model block, a tubulet model block, a vision transformer model block, and a depatchify model block, and generating the fused picture further comprises: concatenating (425) tensor representations of the first output picture and the second output picture to generate a combined tensor representation; applying (430) the combined tensor representation to the patchify model block to generate a tubelet representation; applying (435) a vision transformer model to the tubelet model block to generate a vision transformer output; and applying (440) the vision transformer output to the depatchify model block to generate the fused picture. EEE 10. The method of EEE 9, wherein the vision transformer model block comprises a Transformer block or a Mamba block. EEE 11. The method of EEE 9 or EEE 10, wherein a tensor representation of an output picture comprises: a pixel representation of the output picture (^^); a feature representation of the output picture (F); a motion vector feature representation of the output picture (%^$^; a latent representation of the output picture (^^); and a motion latent representation^^^^$^for the output picture. EEE 12. The method of any preceding EEE, further comprising generating metadata for the coded representation of an input picture, the metadata comprising one or more of: a flag indicating whether the coded picture is an I picture or not; a flag indicating whether the coded picture is a P picture or a B picture; a parameter indicating the DLVC network being used in encoding the input picture; and for a coded B picture: a model-type index parameter indicating a fusion model among two or more fusion models used to generate a fused picture for the coded B picture. EEE 13. A method to code video sequences using a neural-network architecture, the method comprising: receiving a sequence of input pictures to be coded using intra (I) pictures,predictive (P) pictures, and bi-predictive (B) pictures; for generating an I- or P-coded picture, encoding an input image using a first deep learning video coding (DLVC) network comprising a P-model motion coder (^^^^^^^^ block, a P-model temporal context ( ^^^^^^^^^) block, and a P-model frame-coding ^^^^^^^) block; and for generating a B-coded picture using two or more reference frames, accessing a weighted map; and encoding the input image using a second DLVC network comprising a B-model motion coder block (215), a B-model temporal context block (205), and a B-model frame-coding block (210), wherein the weighted map indicates a relative importance of the reference frames being used to generate the B-coded picture. EEE 14. The method of EEE 13, wherein the B-model temporal context block (205) comprises: a first P-model context block using as input a first reference frame to generate a first reference context (&"^; a second P-model context block as input a second reference frame to generate a second reference context (&'^; and generating a fused reference context (C) based on the first reference context, the second reference context, and the weighted map. EEE 15. The method of EEE 14, wherein generating the fused reference context (220) comprises: upsampling the weighted map to generate an upsampled weighted map (),-); andgenerating C as: C =2*324563456^ ),- ∗ &" , 81 − ),-9 ∗ &').EEE 16. TheB-model motion coder block (215) comprises: a first P-model motion estimation block to generate a first motion vector for a first reference picture; a second P-model motion estimation block to generate a second motion vector for a second reference picture; a concatenation block to concatenate the first motion vector and the second motion vector to generate a concatenated motion vector; and a conditional autoencoder (^^$) (225) to generate a quantized first motion vector for the first reference picture, a quantized second motion vector for the second reference picture, and anoutput fused motion feature (%^$) based on motion features from the first reference picture, the second reference picture, and the upsampled weighted map (),-). EEE 17. The method of EEE 16, wherein given a first motion feature (%"^$^ for the first reference picture and a second motion feature (%'^$^ for the second reference picture, theoutput fused motion feature (%^$) is generated as: %^$ =2*324563456^ )^$ ^$,- ∗ %", 81 − )^$ ^$,- 9 ∗ %' ).of EEE 4 or EEE 5 further comprising adding a modulation tensor M to the fused picture F and replacing the fused picture with the generated sum. EEE 19. The method of EEE 18, wherein the generating the modulation tensor M comprises: ì'GH ! KLMNO^PQQQQ^sin F I J 6 R S , if i %2^0 and iXL / 21b<∈ `0, a . 1bwhere,= ∈ `0, c . 1b ,> ∈ `0, d .1b% denotes the modulo operator, and L, H, and W represent the number of layers, height, and width, respectively, of the fused picture. EEE 20. The method of any preceding EEE, wherein the sequence of input pictures is organized into two or more groups pictures (GOPs), and encoding a GOP further comprises encoding the GOP in either a closed-GOP mode, where no P or B picture within the GOP iscoded using reference pictures outside of the GOP, or in an open-GOP mode, wherein a P or B picture within the GOP may use as a reference picture a picture from a neighbor GOP. EEE 21. The method of EEE 20, wherein encoding a GOP further comprises: encoding the GOP in a forward coding mode, from the GOP’s first picture in display order to the GOP’s last picture in display order; encoding the GOP in a reverse coding mode, from the GOP’s last picture in display order towards the GOP’s first picture in display order; and selecting a best coding mode between the forward and reverse coding modes according to a selection criterion optimizing a coding metric measured in the forward and reverse coding modes. EEE 22. The method of EEE 21, wherein the coding metric is based on bits per pixel (BPP) and peak signal to noise ratio (PSNR) measured in each coding mode. EEE 23. The method of EEE 22, wherein the selection criterion comprises: if hj1^l> w ghhlselect as the best coding mode the reverse coding mode; else select as the best coding mode the forward coding mode, wherein w is a positive constant, hj1^l=hj1^k . hj1^i , ghhl = ghhk . ghhi, hj1^i and ghhi denote PSNR and BPPmeasured in forward mode, and hj1^kand ghhkdenote PSNR and BPP measured in reverse coding mode. EEE 24 The method of any one of EEEs 21 to 23, further comprising: reading a GOP’s identification number (gop_id); if the gop_id is odd: coding the GOP in a closed-GOP mode using the best coding mode selected between a forward and a reverse coding mode of the GOP; else coding the GOP in an open-GOP forward coding mode, considering also using reference pictures from GOP with (gop_id+1); coding the GOP in an open-GOP reverse coding mode, considering also using reference pictures from GOP with (gop_id-1); and selecting the best coding mode between the open-GOP forward and reverse coding modes.EEE 25. A tangible computer-readable storage medium having stored thereon computer- executable instructions for executing with one or more processors a method in accordance with any one of the methods recited in EEEs 1-24. EEE 26. An apparatus comprising a processor and configured to perform any one of the methods recited in EEEs 1-24. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0091] Example embodiments that relate to applying video coding techniques using deep learning are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and what is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. ________________________________________________

Claims

CLAIMS What is claimed is:

1. A method to code video sequences using a neural-network architecture, the method comprising: receiving a sequence of input pictures to be coded using intra (I) pictures, predictive (P) pictures, and bi-predictive (B) pictures; encoding and decoding a first input picture using a deep learning video coding (DLVC) network trained to encode I or P-only pictures and generating a first output picture representing a decoded approximation of the first input picture; encoding and decoding a second input picture using the DLVC network and generating a second output picture representing a decoded approximation of the second input picture; combining the first output picture and the second output picture based on a third input picture to be coded as a B-picture to generate a fused picture; and encoding the fused picture using the DLVC network to generate a coded representation of the third input picture with references to the first input picture and the second input picture.

2. The method of claim 1, wherein generating the fused picture further comprises: for a coding block in the third input picture: measuring a first coding cost to code the coding block using a first reference block from the first output picture; measuring a second coding cost to code the coding block using a second reference block from the second output picture;measuring a third coding cost to code the coding block using a third reference block representing a weighted average of the first reference block and the second reference block; computing a minimum coding cost among the first, second, and third coding costs; and selecting among the first reference block, the second reference block, and the third reference block the one which yields the minimum coding cost and apply it to encode the coding block in the third input picture.

3. The method of claim 2, wherein for a computed bit rate R when coding an input picture, and distortion D, a coding cost can be computed as J = R+ λ*D, where λ denotes a Lagrangian multiplier and D represents an error metric between the input picture and a decoded representation of the input picture.

4. The method of any one of claims 1 to 3, wherein generating the fused picture comprises inputting the first output picture and the second output picture into a neural network (NN) fusion model to generate the fused picture.

5. The method of claim 4, wherein the fusion model comprises a feature extraction block and a fusion block, and generating the fused picture further comprises: generating a first frame feature representation of the first output picture using the feature extraction block; generating a second frame feature representation of the second output picture using the feature extraction block; concatenating the first frame feature and the second frame feature representations to generate a combined frame feature representation; and applying the combined frame feature representation to the fusion block to generate the fused picture.

6. The method of claim 4 or claim 5, wherein the fusion model is trained separately from the DLVC network or in combination with the DLVC network.

7. The method of any one of claims 4 to 6, wherein the fusion model comprises a uni- feature adaptor and a bi-feature fusion adaptor and generates an output fused picture based on availability of reference features in the first output picture and the second output picture.

8. The method of claim 7, wherein if neither the first output picture and the second output picture have reference features, the fusion model generates a “none” output, else, if only the first output picture has reference features, it generates an output fused picture based on the reference features of the first output picture and the uni-feature adaptor, else, if only the second output picture has reference features, it generates an output fused picture based on the reference features of the second output picture and the uni-feature adaptor, else, it generates an output fused picture based on the reference features of the first output picture, the second output picture and the bi-feature adaptor.

9. The method of any one of claims 4 to 8, wherein the fusion model comprises a video sequence model (305) comprising: a patchify model block, a tubulet model block, a vision transformer model block, and a depatchify model block, and generating the fused picture further comprises: concatenating (425) tensor representations of the first output picture and the second output picture to generate a combined tensor representation; applying (430) the combined tensor representation to the patchify model block to generate a tubelet representation; applying (435) a vision transformer model to the tubelet model block to generate a vision transformer output; and applying (440) the vision transformer output to the depatchify model block to generate the fused picture.

10. The method of claim 9, wherein the vision transformer model block comprises a Transformer block or a Mamba block.

11. The method of claim 9 or claim 10, wherein a tensor representation of an output picture comprises: a pixel representation of the output picture (^^)a feature representation of the output picture (F); a motion vector feature representation of the output picture (%^$^; a latent representation of the output picture (^^); and a motion latent representation ^^^^$^ for the output picture.

12. The method of any preceding claim, further comprising generating metadata for the coded representation of an input picture, the metadata comprising one or more of: a flag indicating whether the coded picture is an I picture or not; a flag indicating whether the coded picture is a P picture or a B picture; a parameter indicating the DLVC network being used in encoding the input picture; and for a coded B picture: a model-type index parameter indicating a fusion model among two or more fusion models used to generate a fused picture for the coded B picture.

13. A method to code video sequences using a neural-network architecture, the method comprising: receiving a sequence of input pictures to be coded using intra (I) pictures, predictive (P) pictures, and bi-predictive (B) pictures; for generating an I- or P-coded picture, encoding an input image using a first deep learning video coding (DLVC) network comprising a P-model motion coder (^^^^^^^^ block, a P-model temporal context ( ^^^^^^^^^) block, and a P-model frame-coding ^^^^^^^) block; and for generating a B-coded picture using two or more reference frames, accessing a weighted map; and encoding the input image using a second DLVC network comprising a B- model motion coder block (215), a B-model temporal context block (205), and aB-model frame-coding block (210), wherein the weighted map indicates a relative importance of the reference frames being used to generate the B-coded picture.

14. The method of claim 13, wherein the B-model temporal context block (205) comprises: a first P-model context block using as input a first reference frame to generate a first reference context (&"^; a second P-model context block as input a second reference frame to generate a second reference context (&'^; and generating a fused reference context (C) based on the first reference context, the second reference context, and the weighted map.

15. The method of claim 14, wherein generating the fused reference context (220) comprises: upsampling the weighted map to generate an upsampled weighted map (),-); and generating C as: C=2*324563456^ ),- ∗ &" , 81. ),-9 ∗ &').

16. Thethe B-model motion coder block (215) comprises: a first P-model motion estimation block to generate a first motion vector for a first reference picture; a second P-model motion estimation block to generate a second motion vector for a second reference picture; a concatenation block to concatenate the first motion vector and the second motion vector to generate a concatenated motion vector; anda conditional autoencoder (^^$) (225) to generate a quantized first motion vector for the first reference picture, a quantized second motion vector for the second reference picture, and an output fused motion feature (%^$) based on motion features from the first reference picture, the second reference picture, and the upsampled weighted map (),-).

17. The method of claim 16, wherein given a first motion feature (%"^$^ for the first reference picture and a second motion feature (%'^$^ for the second reference picture, the output fused motion feature (%^$) is generated as: %^$ =2*324563456^ ),^-$ ∗ %^$ ^ ^$" , 81 . ),-$9 ∗ %' ).

18. Thea modulation tensor M to the fused picture F and replacing the fused picture with the generated sum.

19. The method of claim 18, wherein the generating the modulation tensor M comprises: KLMNO^PQQ ^ìsin F'GHI J 6! QQR S , if i %2^0 and iXL / 21b<∈ `0, a . 1bwhere,= ∈ `0, c . 1b ,> ∈ `0, d .1b% denotes the modulo operator, and L, H, and W represent the number of layers, height, and width, respectively, of the fused picture.

20. The method of any preceding claim , wherein the sequence of input pictures is organized into two or more groups of pictures (GOPs), and encoding a GOP further comprises encoding the GOP in either a closed-GOP mode, where no P or B picture within the GOP is coded using reference pictures outside of the GOP, or in an open- GOP mode, wherein a P or B picture within the GOP may use as a reference picture a picture from a neighbor GOP.

21. The method of claim 20, wherein encoding a GOP further comprises: encoding the GOP in a forward coding mode, from the GOP’s first picture in display order to the GOP’s last picture in display order; encoding the GOP in a reverse coding mode, from the GOP’s last picture in display order towards the GOP’s first picture in display order; and selecting a best coding mode between the forward and reverse coding modes according to a selection criterion optimizing a coding metric measured in the forward and reverse coding modes.

22. The method of claim 21, wherein the coding metric is based on bits per pixel (BPP) and peak signal to noise ratio (PSNR) measured in each coding mode.

23. The method of claim 22, wherein the selection criterion comprises: if hj1^l> w ghhlselect as the best coding mode the reverse coding mode; else select as the best coding mode the forward coding mode, wherein w is a positive constant, hj1^l= hj1^k. hj1^i, ghhl= ghhk. ghhi, hj1^iand ghhidenote PSNR and BPP measured in forward mode, and hj1^kand ghhkdenote PSNR and BPP measured in reverse coding mode.

24. The method of any one of claims 21 to 23, further comprising: reading a GOP’s identification number (gop_id);if the gop_id is odd: coding the GOP in a closed-GOP mode using the best coding mode selected between a forward and a reverse coding mode of the GOP; else coding the GOP in an open-GOP forward coding mode, considering also using reference pictures from GOP with (gop_id+1); coding the GOP in an open-GOP reverse coding mode, considering also using reference pictures from GOP with (gop_id-1); and selecting the best coding mode between the open-GOP forward and reverse coding modes.

25. A tangible computer-readable storage medium having stored thereon computer- executable instructions for executing with one or more processors a method in accordance with any one of the methods recited in claims 1-24.

26. An apparatus comprising a processor and configured to perform any one of the methods recited in claims 1-24.

Images