Tunable hybrid neural video representation

Abstract

To provide systems and methods including a tunable neural network-based video encoder.SOLUTION: A method by a tunable neural network-based video encoder comprises: receiving a video sequence including a plurality of video frames; generating a frame-specific embedding of a first video frame of the plurality of video frames; identifying one or more group-of-pictures (GOP) features of a subset of the plurality of video frames, the subset including the first video frame; combining the frame-specific embedding of the first video frame and the one or more GOP features of the first plurality of the plurality of video frames to provide a latent feature corresponding to a compressed version of the first video frame; and optionally outputting the latent feature of the first video frame to a decoder.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Related Applications This application claims the benefit of and priority to pending U.S. Provisional Patent Application No. 63 / 599,972, entitled "Tunably Variable Hybrid Neural Video Representation," filed November 16, 2023, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Video compression is a long-standing, challenging problem that has inspired much research. The primary goal of video compression is to represent digital video (typically a sequence of frames, each represented by a two-dimensional (2-D) array of pixels, RGB or YUV colors) using minimal storage while simultaneously minimizing loss of quality. While many advances to traditional video codecs have been made in recent decades, the emergence of deep learning has inspired many new approaches that surpass traditional video codecs.

[0003] For example, implicit neural representations (INRs) have attracted significant research interest and have been applied to various domains, including video compression. In addition to exhibiting desirable properties such as fast decoding and the ability for temporal interpolation, INR-based approaches can match or surpass traditional standard video codecs, such as Advanced Video Encoding (AVC, also known as H.264) and High Efficiency Video Encoding (HEVC), in compression performance. However, these existing approaches for utilizing INRs for video compression only perform well in limited, sometimes highly constrained settings, such as being restricted to specific model sizes, fixed aspect ratios, and relatively static video sequences.

[0004] As background, early video INRs employed pixel-wise representations that mapped pixel indices to RGB colors but suffered from limited performance and poor decoding speed. However, a frame-wise representation, called the neural representation for video (NeRV), has also been proposed. In NeRV, a multilayer perceptron (MLP) generates temporal features from position-coded frame indices that are reshaped and upsampled. Inspired by research on generative adversarial networks (GANs), NeRV enables real-time decoding speeds and significantly better reconstruction than previous video INRs, Expedite NeRV (E-NeRV) was developed. In this NeRV, the featurizer used in traditional NeRV is decomposed into temporal and spatial contexts that are fused by a transformer network, and temporal information is further injected into the decoder block via an adaptive instance normalization (AdaIN) layer. Summary of the Invention [Problem to be solved by the invention]

[0005] Flow-guided frame-wise NeRV (FFNeRV) improves performance on dynamic sequences by enforcing temporal consistency with a decoder that warps adjacent independent frames and predicts both the independent frames and a set of optical flow maps, which are used to combine them to provide the final output frame. A hybrid representation, HNeRV, resembles an autoencoder during training and employs an encoder that extracts content-specific embeddings. Once training is complete, the video is represented by the decoder's parameters and frame-by-frame embeddings. While existing variations of NeRV identified above, namely E-NeRV, FFNeRV, and HNeRV, represent improvements over the original NeRV, their usefulness is often limited to specific settings, such as calming high-motion content followed by static sequences, or vice versa. While traditional hybrid approaches show promise in low-bitrate regimes, they fail to scale to higher bitrates and are currently limited to videos with unusual 2:1 aspect ratios. Therefore, there is a need in the art for neural network-based video compression solutions that can efficiently encode video frames across a range of use cases and parameters. [Means for solving the problem]

[0006] The following description includes specific information regarding the implementation of the present invention. Those skilled in the art will recognize that the present invention may be implemented in ways different from those specifically discussed herein. The drawings in this application and their accompanying detailed description are directed to exemplary implementations only. Unless otherwise specified, the same or corresponding elements among the figures are designated by the same or corresponding reference numerals. Furthermore, the drawings and figures in this application are generally not to scale and are not intended to correspond to actual relative dimensions.

[0007] The present application improves upon existing techniques and advances the state of the art by using a tunable hybrid implicit neural representation (INR) for video (hereinafter "T-NeRV") solution based on the insight that conventional codecs, such as Advanced Video Encoding (AVC) also known as H.264 and High Efficiency Encoding (HEVC), require both local and non-local information to efficiently encode frames. Furthermore, the tunable hybrid neural video representation solution disclosed herein may advantageously be implemented as an automated or substantially automated system and method. As used herein, the terms "automation," "automated," and "automating" refer to systems and processes that do not require the participation of a human user, such as a human system operator or administrator. In some implementations, performance of the systems and methods disclosed herein may be monitored by a human system operator, although human involvement is optional. Thus, the methods described herein may be performed under the control of hardware processing components of the disclosed systems. [Effects of the Invention]

[0008] The novel and inventive T-NeRV solution disclosed in this application implements a video compression system that includes a tunable neural network-based video encoder that combines frame-specific embeddings with group-of-picture (GOP-specific) features and upsamples them in conjunction with an optical flow-based decoder. This novel and ingenious combination of frame-specific and GOP-specific features provides levers for content-specific fine-tuning. For example, larger frame embeddings, i.e., emphasizing frame-specific features, can extract more high-frequency information and improve performance for slow-motion sequences. Conversely, more prominent GOP-specific features enable the T-NeRV model to extract more temporal context that the decoder can utilize for frame reconstruction in dynamic sequences. For compression, the T-NeRV solution disclosed in this application jointly minimizes rate and distortion to similarly capture embeddings via quantization-aware training and entropy-constrained training. By employing a single entropy model for all embeddings, our T-NeRV solution can exploit their redundancies to a greater extent than previous approaches. End-to-end training forces our T-NeRV network to fine-tune itself to the target video content by spending more bits on frame-specific embeddings or GOP-specific features and automatically adjusting the tuning levers.

[0009] The current T-NeRV solution contributes at least the following features to the current state-of-the-art: (i) a tunable hybrid video INR that combines frame-specific embeddings and GOP-specific features, thereby providing levers for content-specific fine-tuning, and (ii) an extension of the information-theoretic INR compression framework to include embeddings, thereby exploiting significant redundancies within them. While system users can assert some control by defining the maximum size of the feature maps used, it should be noted that model tuning and balancing the combination of frame-specific embeddings with GOP-specific features are performed automatically during the training and optimization of the T-NeRV model. Furthermore, when evaluated on the UVG dataset, the T-NeRV solution disclosed herein outperforms all previous video INRs in both video representation and video compression tasks. [Brief explanation of the drawings]

[0010] [Figure 1A] 1 shows a diagram illustrating an example architecture for a tunable hybrid implicit neural video representation (T-NeRV) encoder of a video compression system, according to one implementation. [Figure 1B] 1A shows a diagram illustrating an example architecture for an optical flow-based T-NeRV decoder suitable for use with the example T-NeRV encoder shown in FIG. 1A, according to one implementation. [Figure 2] 1A and 1B schematically illustrate a series of T-NeRV blocks suitable for use in the T-NeRV decoder shown in FIG. 1B, according to one implementation. [Figure 3] 1 is a flowchart outlining a method for implementing tunable hybrid neural video representation, according to one implementation. DETAILED DESCRIPTION OF THE INVENTION

[0011] JPEG0007789159000001.jpg88160

[0012] 1A, the T-NeRV encoder 110 includes a large encoder 120, a position encoder 124, a multi-layer perceptron 126, a multi-resolution feature grid 112, a fusion block 127, and in some implementations may also include an MLP 132. Also shown in FIG. 1A are learned weights 106, spatial frame embedding 122, one or more temporal features 128 (hereinafter referred to as “temporal features 128”), one or more GOP features (hereinafter referred to as “GOP features 114”), frame-specific embedding 116, and optional temporal feature output 130.

[0013] During training, the T-NeRV network, including the T-NeRV encoder 110 and the T-NeRV decoder 140, performs forward and backward passes, as known in the art, to optimize network parameters. Once training is complete, only the GOP features 114, the frame-specific embeddings 116 (i.e., feature maps or tensor representations of the video frames), the latent features 118, and the optional temporal feature outputs 130 are retained as part of the video state, while the rest of the features shown in FIG. 1A are discarded. Thus, decoding an input video frame 102 involves obtaining the GOP features 114 from the multi-resolution feature grid 112 and fusing them with the frame-specific embeddings 116 using any suitable combination technique, such as concatenation or fusion, to obtain the latent features 118 (Zt) of the input video frame, which are then passed through the T-NeRV decoder 140.

[0014] Frame-specific embedding JPEG0007789159000002.jpg823116 can serve as one of two components of the latent feature 118. The frame-specific embedding 116 can be generated in two steps. First, the large encoder 120 generates the spatial frame embedding 122. JPEG0007789159000003.jpg78 can be extracted from the input video frame 102 and then augmented with additional temporal features 128. This augmentation is a mathematical operation that combines spatial frame embeddings 122 with temporal features 128 by concatenating them or, alternatively, by combining them using any other mathematical technique, neural network, or transformer, for example. Because the frame-specific embeddings themselves are transmitted to the decoder, all network blocks involved in their generation can be discarded after encoding. As a result, the large encoder 120 can be approximately 100 times larger, i.e., contain 100 times more weights, than conventional encoders used in HNeRV networks, because the encoder weights themselves are not transmitted to the decoder.

[0015] JPEG0007789159000004.jpg148163

[0016] The T-NeRV encoder 110 encodes the GOP features 114 The multi-resolution feature grid 112 is utilized to obtain JPEG0007789159000005.jpg723. During the forward pass, the frame index 104, t, is used to index into the grid and select two features in the formation of a learned number or tensor, where those t features are present in all levels of the multi-resolution feature grid 112. Bilinear interpolation can be performed between these two features, and the results from all levels can be concatenated, fused, or otherwise combined to obtain the GOP's features 114. In contrast to FFNeRV, T-NeRV allows features at different levels of the multi-resolution feature grid 112 to be combined in different ways, while maintaining their spatial dimensions (h, w), and the number of channels. Note that we allow the size to vary with respect to JPEG0007789159000006.jpg53. This modification allows T-NeRV to advantageously adjust the amount of information captured for different GOP sizes. We note that this adjustment by T-NeRV occurs automatically during optimization during the training process.

[0017] The latent features 118 obtained from the fusion of the GOP features 114 and the frame-specific embeddings 116 are used to generate the temporal feature output 130 generated by the MLP 132. 1B, the T-NeRV decoder 140 may include three parts: a pre-block convolutional layer 142, a series of T-NeRV blocks 150 (e.g., five T-NeRV blocks), and a flow-based warping block 144. Note that the pre-block convolutional layer 142 includes a convolutional layer with kernels that perform either channel widening or channel narrowing, i.e., tensor reshaping operations known in the art, depending on the size of the latent features 118 and the dimensions predicted by the first of the T-NeRV blocks 150. Also shown in FIG. 1B are head layers, i.e., output layers 164a and 164b, optical flow map 143, independent frame buffer 146, aggregated frame 148, independent frame 166, aggregated block 168, and predicted frame 170 corresponding to input video frame 102 of FIG. 1A.

[0018] The T-NeRV decoder 140 can utilize a series of T-NeRV blocks 150 to predict the independent frames 166 and the optical flow map 143. Referring to FIG. 2, FIG. 2 schematically illustrates a series of T-NeRV blocks 250 suitable for use in the T-NeRV decoder 140 of FIG. 1B , according to one implementation. As shown in FIG. 2, a linear layer 252 extracts per-channel statistics, such as mean and variance, from the T-NeRV encoder 110, which are used to shift the distribution of the input tensor via an adaptive instance normalization (AdaIN) layer 254, thereby injecting temporal information into the series of T-NeRV blocks. Note that the temporal feature output 230 and the series of T-NeRV blocks 250 generally correspond to the temporal feature output 130 and the series of T-NeRV blocks 150, respectively, shown variously in FIGS. 1A and 1B. As a result, a series of temporal feature outputs 130 and T-NeRV blocks 150 can share any of the properties attributed to the respective temporal feature outputs 230 and series of T-NeRV blocks 250, and vice versa, according to the present invention.

[0019] As further shown in FIG. 2, in addition to the linear layer 252 and the AdaIN layer 254, the first two T-NeRV blocks in the sequence of T-NeRV blocks 150 / 250 may include a convolutional layer 256 with channel expansion (e.g., a 5×5 convolutional layer) followed by a pixel shuffle layer 258, which may perform spatial upsampling and may be followed by an activation function 260a in the form of a Gaussian Error Linear Unit (GELU) activation function. In some implementations, the T-NeRV decoder 140 may utilize a low channel reduction factor, e.g., r = 1.2, and increase the kernel size. This approach allows the T-NeRV decoder 140 to allocate more parameters to later stages, aiding in the reconstruction of high-frequency details. In particular, this strategy is advantageous for maximizing the third T-NeRV block in the sequence of T-NeRV blocks 150 / 250, given its dual use in upsampling and optical flow prediction.

[0020] HNeRV is a kernel with a size of k i Note that while we teach σ = min{2i-1,5}, 5 × 5 convolutions have been found to be computationally expensive and parameter-inefficient. In further contrast to HNeRV, a series of three to five T-NeRV blocks 150 / 250 uses two consecutive convolutional layers 262 (e.g., 3 × 3 convolutional layers) with the same total number of parameters and an additional activation 260b between the convolutional layers 262, resulting in the block architecture shown in panel (b) of Figure 2. The higher parameter efficiency and additional nonlinearity further assist the T-NeRV decoder 140 in reconstructing high-frequency information.

[0021] JPEG0007789159000008.jpg151162

[0022] JPEG0007789159000009.jpg29162

[0023] With respect to training a T-NeRV network including the T-NeRV encoder 110 and the T-NeRV decoder 140, note that INR compression can be modeled as a rate-distortion problem L = D + λR, where D denotes some distortion loss, R represents the entropy of the INR parameter θ, and λ establishes a trade-off relationship between the two. Training the INR with respect to the loss L has the desirable result of both minimizing the rate and distortion during training and enabling compression to be achieved by entropy encoding the parameter θ after training is completed. Thus, the T-NeRV network including the T-NeRV encoder 110 and the T-NeRV decoder 140 can be trained using an optimized loss function including a distortion loss and an entropy loss, where the distortion loss and the entropy loss can be jointly optimized.

[0024] Furthermore, the training process described above can be implemented using, for example, integer Because it requires a discrete set of symbols, such as the set JPEG0007789159000010.jpg53, quantization recognition training is performed using a straight-through estimator (STE), as known in the art, to ensure differentiability. Each layer is independently quantized using scale reparameterization with two trainable parameters. The entropy of each layer can then be estimated independently by fitting a small neural network to the weight distribution.

[0025] Next, the same framework is extended to handle parametric encoding, such as the multi-resolution feature grid 112 of the T-NeRV encoder 110, as well as embedding. Each of the multiple feature grids contained within the multi-resolution feature grid 112, e.g., three feature grids, is treated as its own layer with its own quantization parameters. During the forward pass, each feature grid is quantized and dequantized using the same scale reparameterization scheme before bilinearly interpolating two of its features. Note that employing one entropy model per feature grid allows the T-NeRV network to determine how much the features at each time frequency should be compressed.

[0026] In contrast to the approach described above with reference to the multi-resolution feature grid 112, a single entropy model is employed for all frame-specific embeddings 116, thereby encouraging the T-NeRV network to exploit inter-frame redundancy. This counter balances the injection of temporal features 128 into the frame-specific embeddings 116 described above, restricting the T-NeRV network to use such temporal information only when the gain in video quality outweighs the increase in entropy. Similar to the distortion loss, the entropy loss is back-propagated through the T-NeRV network, encouraging the T-NeRV network to learn features in its feature grid that exhibit low entropy.

[0027] The functionality of the T-NeRV encoder 110 shown in Figure 1A is further explained by reference to Figure 3. Figure 3 shows a flowchart 380 outlining an exemplary method for implementing tunably variable hybrid neural video representation, according to one implementation. Note that certain details and features of the method outlined in Figure 3 have been omitted from the flowchart 380 so as not to obscure the discussion of the inventive features in this application.

[0028] 3 in combination with FIG. 1A, the method outlined by flowchart 380 includes receiving (operation 381) by T-NeRV encoder 110 a video sequence including a plurality of video frames, such as input video frame 102. As shown in FIG. 1A, input video frame 102, as well as other video frames included in the video sequence received in operation 381, can be received by T-NeRV encoder 110 using a large encoder 120 of T-NeRV encoder 110. As mentioned above, large encoder 120 can be approximately 100 times larger, i.e., can include 100 times more weights, than conventional encoders used in HNeRV networks.

[0029] 3 and 1A in combination, the method outlined by flowchart 380 further includes generating a frame-specific embedding 116 for a first video frame (hereinafter "input video frame 102") of the video frames included in the video sequence received in operation 381 (operation 382). As noted above, the frame-specific embedding 116 can be generated in one or two steps. First, large encoder 120 extracts spatial frame embedding 122 from input video frame 102, which can then serve as frame-specific embedding 116 in some use cases or can be augmented with additional temporal features 128 in other use cases.

[0030] Spatial Frame Embedding 122 In an implementation performing JPEG0007789159000011.jpg66, the output by the large encoder 120 is further augmented with temporal features 128 to provide frame-specific embeddings 116, and a PE 124 followed by an MLP 126 may be utilized to extract temporal information from the frame index 104. The temporal information may be extracted from a subset of video frames included in the video sequence received in operation 381, where the subset of video frames of the video sequence includes the input video frame 102. The temporal information output by the MLP 126 is then augmented with temporal features 128 that match the dimensions of the spatial frame embeddings 122. JPEG0007789159000012.jpg723. The temporal features 128 can then be fused with the spatial frame embedding 122 to obtain the frame-specific embedding 116. Thus, the frame-specific embedding 116 can be generated in operation 382 by the T-NeRV encoder 110 using the large encoder 120 to provide the spatial frame embedding 122, which in some use cases combines the spatial frame embedding 122 and the temporal features 128.

[0031] 3 and 1A in combination, the method outlined by flowchart 380 further includes identifying one or more GOP characteristics of a subset of video frames included in the video sequence received in operation 381, the subset of video frames including input video frame 102 (operation 383). Such GOP characteristics may include, for example, I-frames (intra-coded pictures), P-frames (predicted pictures), and B-frames (bidirectional pictures), as well as integers M and N, where M specifies the number of frames between two anchor frames, i.e., the distance between an I-frame and a P-frame or the distance between two P-frames, and N specifies the number of frames between two I-frames. It should be noted that the subset of video frames from which features of one or more GOPs are identified in operation 383 may be the same subset of video frames from which temporal features 128 are optionally extracted as part of operation 382, ​​or another subset of video frames of the video sequence received in operation 381 that also includes input video frames 102.

[0032] Operation 383 may involve the T-NeRV encoder 110 obtaining features for one or more GOPs using the multi-resolution feature grid 112. As described above, during the forward pass, the frame index 104, t, is used to index into the grid and select two features for which the frame index 104 falls within each level of the multi-resolution feature grid 112. Bilinear interpolation may be performed between these two features, and results from all levels may be concatenated, fused, or otherwise combined to obtain the features 114 for the GOP. As further described above, in contrast to FFNeRV, T-NeRV allows features at different levels of the multi-resolution feature grid 112 to be distributed across channels C while maintaining their spatial dimensions (h, w). i This modification allows T-NeRV to advantageously adjust the amount of information captured for different GOP sizes.

[0033] 3 and 1A in combination, the method outlined by flowchart 380 further includes combining the frame-specific embeddings 116 of the input video frames 102 with the GOP features 114 identified in act 383, e.g., by concatenating them or by using any other mathematical technique, neural network, or transformer, to provide latent features 118 corresponding to a compressed version of the input video frames 102 (act 384). As described above, the T-NeRV encoder 110 is a tunable neural network-based video encoder configured to combine the frame-specific embeddings 116 with groups of GOP features using a fusion block 127 and concatenate them, or alternatively, combine them using any other mathematical technique, neural network, or transformer, e.g., Combining the frame-specific embeddings 116 with the GOP features 114 provides a lever for content-specific fine-tuning, as the frame-specific embeddings 116 extract more high-frequency information from the video frames, thereby improving performance on low-motion sequences. Conversely, giving more salience to GOP features 114 allows the T-NeRV model to extract more temporal context that the decoder 140 can utilize to better reconstruct frames in dynamic sequences. As mentioned above, this levering, or balancing of the combination of frame-specific embeddings and GOP-specific features, is performed automatically during training and optimization of the T-NeRV model. However, system users can assert some control by defining the maximum size of the feature maps used.

[0034] Thus, the latent features 118 may be provided by the T-NeRV encoder 110 at operation 384 as a weighted or unweighted combination of the frame-specific embeddings 116 and the GOP features 114. Furthermore, in some implementations in which the latent features 118 are provided as a weighted combination of the frame-specific embeddings 116 and the GOP features 114, one or both of the first weights applied to the frame-specific embeddings 116 or the second weights applied to the GOP features in the weighted combination may be determined by the T-NeRV encoder 110 based, for example, on whether the content shown in the video sequence received at operation 381 is primarily static or primarily dynamic.

[0035] 1A, 1B, and 3 in combination, in some implementations, the method outlined by flowchart 380 may further include outputting (operation 385) the latent features 118 of the input video frames 102 to a decoder, such as T-NeRV decoder 140, or in other implementations, to yet another type of conventional NeRV-based decoder. Note that operation 385 is optional, and in some implementations, the method outlined by flowchart 380 may conclude with operation 384 described above. Optional operation 385, if performed, is performed by T-NeRV encoder 110 and may also include providing temporal feature output 130 to the decoder, as described above.

[0036] While operations 382, ​​383, and 384, or operations 382, ​​383, 384, and 385, are described above as being performed on one video frame at a time, e.g., input video frame 102, it should be noted that in some implementations, these operations may be performed on a segment (hereinafter "subsequence") of a video sequence. That is, in some use cases, input video frame 102 may correspond to a subsequence of the subset of video frames identified in operation 383, frame-specific embeddings 116 may include multiple frame-specific embeddings, each corresponding to a video frame in the identified subsequence of the subset of video frames, and latent features 118 may be multiple latent features, each corresponding to a compressed version of a video frame in the identified subsequence of the subset of video frames. In implementations in which latent features 118 correspond to multiple latent features, the multiple latent features may be output to a decoder in operation 385 or may be stored for later decoding.

[0037] In some implementations, the method outlined by flowchart 380 may culminate in operation 385 described above. However, in implementations in which the video compression system including the T-NeRV encoder 110 further includes a T-NeRV decoder 140, the method outlined by flowchart 380 may further include the operations of receiving, by the T-NeRV decoder 140, the latent features 118 from the T-NeRV encoder 110 and decoding, by the T-NeRV decoder 140, the latent features 118 to provide a predicted frame 170 as an uncompressed video frame corresponding to the input video frame 102. Note that the decoding of the latent features 118 to provide the predicted frame 170 may be performed by the T-NeRV decoder in the manner described in detail above with reference to FIGS. 1B and 2.

[0038] With respect to the method outlined by flowchart 380 and described above, it should be noted that operations 381, 382, ​​383 and 384, or operations 381, 382, ​​383, 384 and 385, may be performed in an automated process from which human participation may be omitted.

[0039] Accordingly, this application discloses a system and method for implementing a tunable hybrid neural video representation that advances the state of the art through at least the following contributions: (i) developing a T-NeRV encoder that combines frame-specific embeddings with GOP-specific features, thereby providing levers for content-specific fine-tuning; and (ii) extending information-theoretic INR compression frameworks to include embeddings, thereby advantageously exploiting the considerable redundancy therein. As noted above, when evaluated on the UVG dataset, the T-NeRV solution disclosed by this application outperforms all previous video INR solutions in both video representation and video compression tasks.

[0040] It is apparent from the foregoing description that various techniques may be used to implement the concepts described herein without departing from the scope of those concepts. Moreover, while the concepts have been specifically described with reference to particular implementations, those skilled in the art will recognize that changes can be made in form and detail without departing from the scope of those concepts. As such, the described implementations are to be considered in all respects as illustrative and not restrictive. Also, the present application is not limited to the particular implementations described herein, but it will be understood that many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.

Claims

1. 1. A system comprising a tunable neural network based video encoder, comprising: The tunable neural network based video encoder comprises: receiving a video sequence including a plurality of video frames; generating a frame-specific embedding for a first video frame of the plurality of video frames; identifying characteristics of one or more group of pictures (GOPs) of a first plurality of the video frames, the first plurality of the video frames including the first video frame; and combining the frame-specific embedding of the first video frame with features of the one or more GOPs of the first plurality of video frames to provide latent features corresponding to a compressed version of the first video frame.

2. 2. The system of claim 1, wherein features of the one or more GOPs of the first plurality of the video frames are identified using a multi-resolution feature grid of the tunable neural network-based video encoder.

3. The tunable neural network based video encoder comprises: configured to extract from the first plurality of the video frames one or more temporal features of the first plurality of the video frames; The system of claim 1 , wherein the frame-specific embedding is generated based on a combination of the one or more temporal features.

4. The system of claim 1 , wherein the latent features comprise a weighted combination of the frame-specific embedding and features of the one or more GOPs.

5. 5. The system of claim 4, wherein at least one of the first weights applied to the frame-specific embedding or the second weights applied to the features of the one or more GOPs is determined by the tunable neural network-based video encoder.

6. The system of claim 1 , wherein the tunable neural network-based video encoder is trained using an optimized loss function that includes a distortion loss and an entropy loss.

7. The system of claim 6 , wherein the distortion loss and the entropy loss are jointly optimized.

8. 2. The system of claim 1, wherein the first video frame comprises a subsequence of the first plurality of the video frames, the frame-specific embedding comprises a plurality of frame-specific embeddings respectively corresponding to each of the subsequences of the first plurality of the video frames, and the latent feature is one of a plurality of latent features respectively corresponding to a compressed version of each of the subsequences of the first plurality of the video frames.

9. 1. A system comprising a neural network based video decoder, The neural network based video decoder comprises: configured to receive latent features corresponding to a compressed version of a first video frame of a first plurality of video frames included in a video sequence, the latent features being a combination of a frame-specific embedding of the first video frame and features of one or more group of pictures (GOPs) of the first plurality of video frames; The system is configured to decode the latent features to provide an uncompressed video frame corresponding to the first video frame.

10. The system of claim 9 , wherein the frame-specific embedding is generated based on a combination of the first plurality of video frames with one or more temporal features extracted from the plurality of video frames.

11. The system of claim 9 , wherein the latent features include a weighted combination of the frame-specific embedding and features of the one or more GOPs.

12. 10. The system of claim 9, wherein the first video frame comprises a sub-sequence of the first plurality of the video frames, the frame-specific embedding comprises a plurality of frame-specific embeddings respectively corresponding to each of the sub-sequences of the first plurality of the video frames, and the latent feature is one of a plurality of latent features respectively corresponding to a compressed version of each of the sub-sequences of the first plurality of the video frames.

13. 1. A method for implementation by a tunable neural network based video encoder, the method comprising: receiving a video sequence comprising a plurality of video frames; generating a frame-specific embedding for a first video frame of the plurality of video frames; identifying characteristics of one or more group of pictures (GOPs) of a first plurality of said video frames, said first plurality of said video frames including said first video frame; combining the frame-specific embedding of the first video frame with features of the one or more GOPs of the first plurality of the video frames to provide latent features corresponding to a compressed version of the first video frame.

14. 14. The method of claim 13, wherein features of the one or more GOPs of the first plurality of the video frames are identified using a multi-resolution feature grid of the tunable neural network-based video encoder.

15. extracting one or more temporal features of the first plurality of video frames from the first plurality of video frames; The method of claim 13 , wherein the frame-specific embedding is generated based on a combination of the one or more temporal features.

16. The method of claim 13 , wherein the latent features comprise a weighted combination of the frame-specific embedding and features of the one or more GOPs.

17. 17. The method of claim 16, further comprising determining at least one of a first weight to be applied to the frame-specific embedding or a second weight to be applied to the one or more GOP features.

18. The method of claim 13 , wherein the tunable neural network-based video encoder is trained using an optimized loss function comprising a distortion loss and an entropy loss.

19. The method of claim 18 , wherein the distortion loss and the entropy loss are jointly optimized.

20. 14. The method of claim 13, wherein the first video frame comprises a sub-sequence of the first plurality of the video frames, the frame-specific embedding comprises a plurality of frame-specific embeddings respectively corresponding to each of the sub-sequences of the first plurality of the video frames, and the latent feature is one of a plurality of latent features respectively corresponding to a compressed version of each of the sub-sequences of the first plurality of the video frames.

Images