Fusion of multilevel latents in neural networks for image and video coding

Abstract

Methods, systems, and bitstream syntaxes are described for latent feature fusion in multi-level end-to-end neural networks used in image and video compression. The fused architectures can be static or dynamic based on image characteristics (e.g., natural vs. screen content images) or other coding parameters such as bitrate constraints or rate-distortion optimization. A variety of multi-level fusion architectures are discussed.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority to the following applications: Indian Provisional Patent Application No. 202141038587 filed on August 25, 2021; US Provisional Patent Application No. 63 / 257,388 filed on October 19, 2021; European Patent Application No. 21209479.1 filed on November 22, 2021; and Indian Provisional Patent Application No. 202141058191 filed on December 14, 2021. Each of these is incorporated herein by reference in its entirety.

[0002] Technique This document generally relates to images. More specifically, certain embodiments of the present invention relate to the multi - level latent fusion in neural networks used for image and video encoding.

Background Art

[0003] In 2020, the MPEG group of the International Organization for Standardization (ISO), in cooperation with the International Telecommunication Union (ITU), released the first version of the Versatile Video Coding standard (VVC), also known as H.2,66. More recently, the same joint group (JVET) and experts in still - image compression (JPEG) have begun working on the development of a next - generation coding standard that provides improved coding performance over existing image and video coding techniques. As part of this research, coding techniques based on artificial intelligence and deep learning are also being considered. As used herein, the term "deep learning" refers to a neural network having at least three layers, preferably more than three layers.

[0004] As understood by the inventors, improved techniques for encoding images and videos based on neural networks are described herein. The approaches described in this section are approaches that could have been pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise noted, none of the approaches described in this section should be assumed to qualify as prior art simply by being included in this section. Similarly, any problems identified with respect to one or more approaches should not be assumed, unless otherwise noted, to have been recognized in any prior art based on this section. Each of the references listed here is incorporated by reference within the whole. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] D. Minnen, J. Balle', and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression”, 32nd Conf. on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada, 2018 [Non-Patent Document 2] J. Balle', D. Minnen, S. Singh, SJ Hwang, and N. Johnston, “Variational image compression with a scale hyperprior”, International Conference on Learning Representations (ICLR), 2018, also arXiv:1802.01436v2 (2018) [Non-Patent Document 3] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. “Learned image compression with discretized Gaussian mixture likelihoods and attention modules”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), p.7939-7948, 2020, also arXiv e-prints (2020): arXiv-2001.01568v3, 30 March 2020 [Non-Patent Document 4] T-Y Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection”, Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21,17-2125, 2017 [Non-Patent Document 5] S. Liu, X. Xu, S. Lei, and K. Jou, “Overview of HEVC extensions on screen content coding”, APSIPA Transactions on Signal and Information Processing, vol. 4, p. e10, 2015 [Non-Patent Document 6] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, "DVC: An end-to-end deep video compression framework", 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10998-11007). IEEE Computer Society, 2019, also arXiv:1812.00101v3, 7 April 2019 [Non-Patent Document 7] J. Duda, "Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding", arXiv preprint arXiv:1311.2540v2, 6 Jan. 2014 [Non-Patent Document 8] M. Zhu, K. Han, C. Yu, and Y. Wang, "Dynamic Feature Pyramid Networks for Object Detection", arXiv preprint arXiv:2012.00779 (2020) [Non-Patent Document 9] C. Guo, B. Fan, Q. Zhang, S. Xiang, and C. Pan, "Augfpn: Improving multi-scale feature learning for object detection", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 12595-12604, 2020 [Non-Patent Document 10] Y. Han et al., “Dynamic Neural Networks: a survey”, arXiv preprint arXiv:2102.04906 (2021) [Non-Patent Document 11] Tensorflow compression toolbox https: / / github.com / tensorflow / compression [Brief explanation of the drawing]

[0006] Embodiments of the present invention are shown in the accompanying drawings as examples, not as limitations. In the drawings, similar reference numerals refer to similar elements.

[0007] [Figure 1] This document presents an exemplary neural network model for end-to-end image and video coding.

[0008] [Figure 2] An example of an exemplary neural network processing model for image and video coding using multilevel latent (MLL) fusion according to one embodiment of the present invention is shown.

[0009] [Figure 3] Sections 3A and 3B detail exemplary examples of MLL-fused neural networks according to embodiments of the present invention.

[0010] [Figure 4] An exemplary fusion gate for an MLL fusion architecture with dynamic fusion, according to one embodiment of the present invention, is shown.

[0011] [Figure 5] An example of a spatially weighted MLL fusion architecture according to one embodiment of the present invention is shown.

[0012] [Figure 6A]An example of an MLL fusion architecture with potential multi-layer transmission is shown. [Figure 6B] An example of an MLL fusion architecture with potential multi-layer transmission is shown.

[0013] [Figure 6C] Exemplary attention blocks using single-layer convolution and sigmoid blocks are shown.

[0014] [Figure 7] An exemplary framework for using a neural network in video coding is shown. [

Embodiments for Carrying Out the Invention

[0015] Exemplary embodiments regarding multi-level potential fusion in neural networks used in image and video coding [encoding] are described herein. In the following description, for the purpose of explanation, numerous individual details are set forth in order to provide a thorough understanding of the various embodiments of the present invention. However, it will be apparent that the various embodiments of the present invention may be practiced without these individual details. On the other hand, well-known structures and devices are not described in exhaustive detail in order to avoid obscuring, obfuscating, or rendering unreadable the embodiments of the present invention.

[0016] overview The exemplary embodiments described herein relate to image and video coding using a neural network. In one embodiment, a processor receives an input image of an input spatial resolution to be compressed using latent features. Next, the processor processing the input image using a plurality of consecutive levels of a convolutional network to generate a potential fusion output, wherein for each network level of the plurality of convolutional networks, its output has a lower spatial resolution than its input; A step of quantizing the latent fusion output to generate a quantized fusion latent; The steps include applying arithmetic coding to the quantized fused latent to generate a coded fused latent, To generate potential fusion output, further: Select a latent output from two or more convolutional networks among the aforementioned multiple consecutive levels of convolutional networks; This includes fusing the selected latent outputs to generate the latent fused output.

[0017] In another embodiment, the processor receives an input image of input spatial resolution which is to be compressed using latent features. The processor then A step of processing the input image using a plurality of successive levels of convolutional networks to generate one or more potential fused outputs, wherein for each level of the plurality of convolutional networks, the output has a lower spatial resolution than its input; A step of selecting a potential level L1 having an L1 output and a potential level L2 having an L2 output, wherein level L2 is after level L1; The step of generating an upscaled level L2 output by upscaling the L2 output to match the spatial resolution of the L1 output; The steps include: combining the upscaled L2 output and the L1 output to generate a fused L1 output; The steps include: quantizing and encoding the fused L1 output to generate a fused encoded L1 output; The process involves quantizing and encoding the L2 output to generate the encoded L2 output.

[0018] Exemplary end-to-end video coding model Deep learning-based image and video compression techniques are becoming increasingly popular and are an active area of ​​research. Most of the popular techniques are based on variational autoencoders that employ convolutional neural networks (CNNs) trained end-to-end on training datasets. Figure 1 shows an example of a process pipeline (100) of such a technique (Non-Patent Documents 1, 2, 6) that uses a four-layer architecture for encoding and decoding latent features.

[0019] As used herein, the terms “latent feature” or “latent variable” refer to features or variables that are not directly observable, but rather inferred from other observable features or variables, for example, by processing directly observable variables. In image and video coding, the term “latent space” may refer to a representation of compressed data where similar data points are closer to each other. In video coding, examples of latent features include representations of transformation coefficients, residuals, motion representations, syntax elements, and model information. In the context of neural networks, latent space is useful for learning data features and for finding simpler representations of image data for analysis.

[0020] As shown in Figure 1, given an input image x (102) with an input resolution of h × w, in the encoder (100E), the input image is processed by a series of convolutional neural network blocks (also called convolutional networks or convolutional blocks), each followed by a nonlinear activation function (105, 110, 115, 120). In each of these layers (which may include multiple sublayers of convolutional networks and activation functions), its output is typically reduced (for example, by more than 2 times) (this factor is typically called the "stride," where stride=1 means no downsampling, stride=2 means 2x downsampling in each direction, etc.). For example, using stride=2, the output of the convolutional network (105) is h / 2 × w / 2. The final layer (e.g., 120) generates the output latent coefficient y(122), which is quantized (Q) and further entropy encoded (e.g., by an arithmetic encoder AE) before being sent to the decoder (100D). A hyper-prior network and a spatial context model network (not shown) are also used to generate a probabilistic model of the latent (y).

[0021] In the decoder (100D), the process is reversed. After arithmetic decoding (AD), given the decoded latent ^y (^y; hereafter the same) (124), a series of deconvolution layers (125, 130, 135, 140), each combining a deconvolution neural network block and a nonlinear activation function, are used to generate an output ^x (142) that approximates the input (102). In the decoder, the output resolution of each deconvolution layer is typically increased (e.g., by more than 2x) to match the downsampling factor at the corresponding convolution level in encoder 100E. This ensures that the input and output images have the same resolution.

[0022] In such architectures, the receptive field area of ​​latents increases based on the downsampling ratio and kernel size used in each layer, as shown in Figure 1. Since main latents are encoded at the last layer depth (e.g., 120), they typically have a high receptive field size based on the number of layers (typically 4-6), the downsampling ratio (typically 2), and the kernel size used in each layer (typically 3x3-5x5). Such fixed-depth-based neural networks have been primarily developed for encoding natural images and may not be optimal for encoding latent features of varying spatial sizes and texture, edge, and smooth area characteristics in a highly diverse set of image / video sources. Diverse image / video sources include, for example, screen content, natural content, user-generated content, computer-generated image (CGI) based game and animation content, and mixed content for various operating bitrates. For example, screen content images (SCI) differ significantly from natural images in their color structure and statistics. SCIs tend to have more frequent abrupt color changes and uniform color regions compared to natural images. Furthermore, SCIs often contain text that can be of a wide range of sizes, from very small to very large. Therefore, good reproduction of SCIs requires the ability to handle intensity variations across a wide range of spatial scales. The proposed embodiment enables deep learning-based image and video codecs to adaptively fuse and encode latents at different layer depths, which allows for a multilevel receptive field-based latent coding architecture for optimal rate-distortion coding performance for diverse content sets. While the multilevel receptive field coding architecture may broadly relate to variable block-size coding in conventional block-based video codecs, it does not require explicit coding of specific sizes and structures of transform or predictive blocks.

[0023] Deep learning architectures for image and video encoding are approaching competition with conventional methods in the case of natural images. The proposed adaptive multilevel latent fusion method can improve upon this current technique for natural images and videos, but they are particularly beneficial for screen content compression. For SCI, current deep learning schemes lag significantly behind conventional methods, and the proposed method yields significantly better results.

[0024] The proposed architecture is motivated by the feature pyramid network (FPN) introduced in the object detection and object classification literature, with the goal of improving object detection and classification across scale (Non-Patent Literature 4). Conventional image coding standards such as HEVC and VVC employ several specialized techniques to efficiently compress SCI images. For example, the HEVC standard employs specialized tools such as intra-block copying, palette coding, adaptive color conversion, and adaptive motion resolution to handle SC video (Non-Patent Literature 5). Further gains may be possible by incorporating some of these ideas more directly into CNN-based codecs, but at present, this is a topic of ongoing research.

[0025] Multi-level latent fusion Current CNN-based architectures used for image compression (Non-Patent Documents 1, 2) have a feature hierarchy structure organized into several levels (or layers), with output downsampling from one level to the next. This leads to a gradual decrease in the spatial resolution of the feature map at each subsequent level, while simultaneously increasing the receptive field of the convolutional filter. Features from higher levels tend to be semantically stronger and capable of greater representation and compression, but are not spatially as well localized due to the reduced spatial resolution. As shown in Figure 1, in current compression methods, latent features to be transmitted (e.g., 122) are taken from the highest level of the network (e.g., 120). Due to the limited spatial resolution and large receptive field of latent features, current architectures may not be particularly well-suited for reconstructing small image structures such as short edges and text characters, which are common in SCI. During compression and decompression, certain fine details of text characters may be lost, sometimes reducing their readability.

[0026] On the other hand, features from lower levels are subsampled fewer times, resulting in more precise localization for higher spatial resolution. Fusing features from multiple levels using lateral connections (also called skip connections) can leverage the strengths of different levels in the feature hierarchy. While this idea has been previously demonstrated in the literature to help improve the detection and classification of objects of varying sizes (Non-Patent Literature 4), to our knowledge it is novel in the field of image and video compression and has demonstrable advantages in compressing screen content images.

[0027] Figure 2 shows an exemplary embodiment of the proposed multi-level latents fusion architecture, simply called the MLL fusion architecture. Compared to Figure 1 (100), the encoder (200E) has a new latent fusion network (205) added. In the decoder (200D), the decoder CNN (210) uses a suitable number of deconvolution / nonlinear activation layers, as shown in decoder 100D. As before, the output 212 represents a decoded approximation of the input 102. The latent fusion network (205) can take various forms, and exemplary embodiments are considered below.

[0028] Figures 3A and 3B show examples of MLL fusion architectures, where, for simplicity, the quantizer (Q), arithmetic coding (AE), and arithmetic decoding (AD) blocks are not shown. Figure 3A shows a single fusion ladder that fuses the output of convolutional network 4 with the output of convolutional layer 3 to produce a latent output 305. The upsampling operation is a 2x upsampling if convolutional layer 4 has half the resolution of convolutional layer 3. GDN stands for generalized divisive normalization, and IGDN stands for inverse GDN, a type of transformation that performs local divisive normalization and has been shown to be quite effective in image compression (Non-Patent Literature 1, 2). For example, GDN / IGDN functions are available as part of the TensorFlow compression toolbox (Non-Patent Literature 11). The "Conv" block represents a convolutional network (e.g., m×n×C×K,S, where m×n represents the convolutional kernel, C represents the number of channels being processed, K is the number of convolutional kernels, and S represents the stride). As an example, a filter bank of size 3×3×1×2, S=1 consists of two convolutional kernels, each operating on one channel, having a size of 3 pixels × 3 pixels and a stride of 1. The "Deconv" block represents a deconvolutional block. A 1×1 convolutional layer is required to match the number of channels between the feature levels being fused. As shown in Figure 3A, the decoder network has Deconv and IGDN blocks, which are modified by removing the first incoming deconvolutional layer (e.g., Decon-4) to produce output 322, which is a decoded approximation of input 302. Alternatively, the extra deconvolution layer (e.g., Deconv-4) can be retained, but since the latent (305) is encoded in layer 3 (e.g., at twice the resolution of layer 4), the stride of Deconv-4 should be set to 1 to avoid further upsampling.

[0029] Figure 3B is similar to the architecture in Figure 3A, but merges features from three levels, namely levels 2, 3, and 4, to generate a latent (315) at the second level. Here again, in the decoder, two top levels are removed (Deconv-4 and Deconv-3), and given the decoded latent 318, the decoder generates an output 325 that approximates the input 302.

[0030] These architectures can be easily extended to architectures with more than four levels and can merge more than three levels.

[0031] Dynamic MLL Fusion Architecture In Figures 3A and 3B, the fused network uses a 1x1 convolutional network. In another embodiment, the receptive field size for each feature level can be individually controlled adaptively to the image by switching the convolutional kernel width within this convolutional block based on the features of the input image. Figure 4 shows an exemplary architecture of what is called a “dynamic gate,” conceived by the Dynamic Feature Network (FPN) of Non-Patent Literature 8, used for efficient feature extraction. As shown in Figure 4, gating logic (405) based on separate neural networks is used to adaptively select processing convolutional kernels (e.g., Conv1, Conv2, Conv3, etc., which can be of sizes such as 1x1, 3x3, and 5x5, respectively). In one embodiment, even a combination of such convolutional networks (e.g., Conv1(1x1) + Conv2(3x3) + Conv3(5x5)) can be used. Dynamic gates allow the encoder an additional level of adaptability to the image being compressed, potentially giving it higher compression efficiency. For example, an encoder may use a kernel based on whether the input consists of natural images or screen content images, or according to some other criterion for which the neural network (NN) 405 has been trained. Gating logic (405) consists of convolutional layers, global mean pooling layers, and two or more fully connected layers (FC1, FC2, ..., FC n ) and terminate with the Softmax function. Convolutional layers and the last one (e.g., FC) n Each fully coupled layer, excluding the one shown above, is followed by a rectifier-linear unit (ReLU) activation function layer (indicated as "+ReLU" in the diagram).

[0032] The experimental results indicate that in image and video coding, certain bitrate constraints or requirements may necessitate transmitting a different number of layers. For example, returning to Figure 2, a low bitrate requirement might require adding a new convolutional level (e.g., convolution 5, not shown) and fusing it with convolutional layer 4 or another layer. This can help reduce the number of latents to be encoded (thus reducing the number of bits per pixel) and, at the same time, improve image quality. Thus, in some embodiments, a fixed latent fusing network (205) can be replaced by a learning switch network or gate network that can be used to dynamically select which layers should be fusing. This is considered analogous to the concept of block size adaptation used in traditional codecs. In addition, networks can, for example, use dynamic neural networks (Non-Patent Literature 10) to adapt their layer architecture to the input image / video and the required bitrate / quality during inference. Thus, the MLL architecture can be directly applied to dynamic neural network architectures. In such scenarios, the encoder may need to send additional MLL fusion metadata to inform the decoder how to adjust the decoded convolutional layer according to the encoder's selected fusion structure (see examples in the decoder in Figures 3A and 3B). Examples of such metadata are given later (see Tables 1-4).

[0033] Spatially weighted MLL fusion architecture In another embodiment, instead of using simple addition to merge layers, a spatially weighted fusion can be applied (Non-Patent Document 9). Figure 5 shows an example of such an embodiment. As shown in Figure 5, we begin by concatenating the feature levels of interest using a concatenated network (505). Note that before the layers are concatenated, the layers must be appropriately upsampled (520) so that all feature layers have the same spatial resolution (e.g., w × h). Although Figure 5 depicts all four layers being concatenated, one may choose to concatenate only two or three layers (such as in Figure 3A or Figure 3B).

[0034] A separate attention-like network (515) can be used to generate the weight map. Specifically, network 515 takes upsampled features as input and generates one spatial weight map for each feature. For example, when N layers, each with C channels, are concatenated, the input to the concatenator is N(C×h×w) and its output is NC×h×w. In block 515, after the Conv1 convolutional layer, there are C×h×w outputs. Since N layers of weighted aggregation are needed, one or more subsequent convolutional layers (Conv2, Conv3, etc.) reduce the output to N×h×w, and the choice of a 3×3 kernel size for the convolutional layers gives flexibility in smaller spatial receptive fields to give a better localized weight map. Each convolutional layer except the last one includes ReLU as a nonlinear activation function (indicated as "+ReLU" in the figure). The Softmax block generates the final weights.

[0035] In block 510, feature levels are merged using a weighted sum with spatially varying weights. This has the additional advantage that the encoder can spatially adapt to the image being compressed. Thus, the encoder may be able to handle smooth and rapidly changing image regions in different ways using feature levels with suitable receptive field sizes. The decoder architecture is maintained as in Figure 2, depending on how many levels are merged together (see, for example, Figures 3A and 3B).

[0036] MLL with multilevel transmission Figure 6A shows another example of an MLL architecture according to one embodiment. In such a scenario, the latent is transmitted at two or more levels, or at a level lower than the highest available level (of the lowest spatial resolution). As shown in Figure 6A, in this example, the latent is transmitted at both level 4 and level 3, though not limited to this. Alternatively, only the latent at layer 3 may be transmitted, as indicated by the dotted line showing an arbitrary path. The entropy model is modified to model subsequent latent levels conditioned at levels already transmitted in order to reduce the number of bits that need to be transmitted. A fusion encoder network block (605) for fusing two levels may be an append, concatenate, or any other way of combining latents.

[0037] In the decoder, the corresponding fused decoder network (620) may precede deconvolution to reverse the operation of the fused decoder network before the subsequent deconvolution layer (e.g., 130). Specifically, the fused decoder network is used to merge features from two neighboring inputs (e.g., from AD-3 and Deconv-4 (125)), which can be implemented as a simple concatenated layer or a predictive and residual fused block. In this example, features from the higher-level deconvolution block (125) are used for prediction and combined with residual features received from the current-level arithmetic decoder (AD-3). Subsequent processing in the decoder is the same as before.

[0038] Figure 6B shows a variant of Figure 6A as an example, where latents are transmitted at levels 4 and 3, and the level 3 latent is predicted using the upscaled level 4 latent by a two-layer deep deconvolution-based predictor network (630), and the two deconvolution networks are separated by an inverse generalized divisive normalization (IGDN) network. Level 3 predicted residuals

number

[0039] In Figure 6B, the decoding path uses a similar architecture, where the predicted value based on the level 4 latent is added to the decoded level 3 latent before being concatenated with the level 4 latent.

[0040] As described in Non-Patent Document 1, the training goal is to minimize the expected length of the bitstream and the expected distortion of the reconstructed image relative to the original image, which is a rate-distortion (R / D) optimization problem: R+λD (1) This causes the following: Here, λ is the Lagrange multiplier that determines the desired rate-distortion (R / D) tradeoff. In one embodiment, during training of this network, the training error function (e.g., D) is modified to have an additional term with a variable-scale factor for the distortion of the level 3 latent prediction, which is typically the mean squared error of the level 3 latent predictor, e.g.,

number

[0041] Note: This particular architecture requires decoder modification because the latents are explicitly encoded at different levels and merged at the decoder side. The previous architecture, on the other hand, has only one level of encoded latents in the bitstream and does not require merging at the decoder side.

[0042] Considerations on video coding While the exemplary embodiments described so far have focused on image compression, the same tools are applicable to video compression as well. Figure 7 shows an exemplary framework that uses neural networks at various stages of a video coding pipeline, where each such network replaces one or more existing conventional coding tools. The proposed architecture can be used in the compression of intra-frames, residual (or intercoded) frames, or to encode motion vector information. In the case of residual frames, further compression efficiency may be possible by fitting the entropy model to better match the residual image statistics. For motion vector (MV) encoder and decoder networks, it has been shown that increasing the number of convolutional levels (e.g., to 8 levels) can yield significant gains at high bitrates, but relatively lower gains at lower bitrates. Therefore, MLL fusion architectures such as those proposed herein can either further improve gains or reduce complexity at lower bitrates. For example, it is often not necessary to have more than four layers.

[0043] Given an MLL fused network, experimental results suggest that to optimize performance for different bitrates or quality requirements, it may be necessary to apply NNs with different total numbers of layers or different fusion models. One way to select the optimal neural network architecture is to exhaustively explore all options based on rate-distortion optimization (RDO) in the encoder, and then select the neural network architecture with the best RDO. To further benefit from multiple MLL fused networks, an image (or input picture) can be divided into patches (e.g., 128x128 or 256x256). For each patch, RDO can then be applied to select the best network. For each patch, the best combination of neural network fusion parameters can then be signaled, either as part of high-level syntax (HLS) or as supplementary enhancement information (SEI) messaging. This patch-based inference can be beneficial for parallelization, especially for high-resolution images such as 4k, 8k, or higher. For video encoding, a patch-based RDO framework can also be applied for intra / inter / skip decisions. Combined with MLL fusion networks, it should be possible to build networks that support multimode and multiresolution adaptation at the patch granularity.

[0044] The following table shows, without limitation, various examples of such high-level syntax for MLL fusion adaptation according to various embodiments. This high-level syntax may be part of a bitstream at multiple levels of a hierarchical structure (e.g., video stream level, picture level, slice level, tile level, etc.) or as separate SEI messaging. The syntax provides the following information: a) whether the splitting into patches is uniform or non-uniform (see, for example, Table 1); b) MLL fusion adaptation information for each patch. Note: For part a), alternatively, a syntax similar to that used to indicate uniform or non-uniform tiling in HEVC or VVC may be applied.

[0045] In the first example (Table 2), general information about the MLL fusion adaptation data is signaled first, followed by an enable flag being sent for each patch to enable or disable MLL fusion for the current patch. In the second example (Table 3), more detailed MLL fusion adaptation syntax is signaled for each patch. The first example requires fewer bits than the second example, but the second example is more flexible. [Table 1] A MLL_adaptation_enabled_flag equal to 1 specifies that MLL adaptation is enabled for decrypted pictures. A MLL_adaptation_enabled_flag equal to 0 specifies that MLL adaptation is not enabled for decrypted pictures. A uniform_patch_flag equal to 1 specifies that patch column boundaries and patch row boundaries are uniformly distributed across the picture. A uniform_patch_flag equal to 1 specifies that patch column boundaries and patch row boundaries are explicitly signaled. `patch_width_in_luma_samples` specifies the width of the decoded picture in units of luma samples. `patch_width_in_luma_samples` is never equal to 0 and is an integer multiple of 64. `num_patch_columns_minus1` can be derived based on `pic_width_in_luma_samples`. `patch_height_in_luma_samples` specifies the height of the decoded picture in units of luma samples. `patch_height_in_luma_samples` is never equal to 0 and is an integer multiple of 64. `num_patch_rows_minus1` can be derived based on `pic_height_in_luma_samples`. Adding 1 to num_patch_columns_minus1 specifies the current number of patch columns for the picture. If it does not exist, the value is inferred as above if uniform_patch_flag is equal to 1. Otherwise, the value is inferred to be 0. Adding 1 to num_patch_rows_minus1 specifies the current number of patch rows in the picture. If it does not exist, and uniform_patch_flag is equal to 1, the value is inferred as above. Otherwise, the value is inferred to be 0. Adding 1 to patch_column_width_minus1[i] specifies the width of the i-th patch column. Adding 1 to patch_row_height_minus1[i] specifies the height of the i-th patch row. [Table 2]

[0046] Table 2 first signals intra and interMLL fusion-related information. (Note: If MLL_adaptation_enabled_flag is equal to 1, then intra_MLL_adaptation_enabled_flag || inter_MLL_adaptation_enabled_flag is assumed to be equal to 1). Next, for each patch, it signals whether or not MLL is enabled for that patch. A value of 1 for intra_MLL_adaptation_enabled_flag indicates that MLL adaptation is enabled for intra-encoding of the decoded picture. A value of 0 for intra_MLL_adaptation_enabled_flag indicates that intra-MLL adaptation is not enabled for intra-encoding of the decoded picture. intra_fusion_idc specifies the fusion method used for intra-MLL. Note: Examples of fusion IDC values ​​may be 0 for MLL fusion architectures as shown in Figure 3 A / B, 1 for dynamic MLL fusion architectures as shown in Figure 4, 2 for spatially weighted MLL fusion architectures as shown in Figure 5, etc. Adding 1 to intra_num_layers_minus1 specifies the number of layers used for the intra MLL. inter_mv_fusion_idc specifies the fusion method used for inter_mv_MLL. Adding 1 to inter_mv_num_layers_minus1 specifies the number of layers used for the inter-motion vector MLL fusion network. inter_residue_fusion_idc specifies the fusion method used for interresidual MLL networks. Adding 1 to inter_residue_num_layers_minus1 specifies the number of layers used for the inter-residual MLL. A patch_MLL_adaptation_enabled_flag[j][i] equal to 1 specifies that MLL adaptation is enabled for the j-th patch row and i-th patch column. A patch_MLL_adaptation_enabled_flag[j][i] equal to 0 specifies that MLL adaptation is not enabled for the j-th patch row and i-th patch column. patch_intra_MLL_adaptation_enabled_flag[j][i] is set to equal to (patch_MLL_adaptation_enabled_flag[j][i]&intra_MLL_adaptation_enabled_flag). patch_inter_MLL_adaptation_enabled_flag[j][i] is set to equal to (patch_MLL_adaptation_enabled_flag[j][i]&inter_MLL_adaptation_enabled_flag). In another embodiment, patch_MLL_adaptation_enabled_flag[j][i] is not signaled, and instead, patch_intra_MLL_adaptation_enabled_flag[j][i] and patch_inter_MLL_adaptation_enabled_flag[j][i] are directly signaled.

[0047] Note: For each patch, it is assumed that all intra_MLL and inter_MLL are acceptable. If only one case is acceptable, that case simply needs to be signaled. The same assumption holds for the following example.

[0048] In another example, to allow for greater flexibility, all MLL fusion adaptation-related information for each patch is signaled, as shown in Table 3. For example, the syntax in Table 3 allows some patches to be encoded as intra-encoded patches and some patches to be encoded as inter-encoded patches. [Table 3]

[0049] For simplicity, the syntax above supports fusion of only the top two levels. If more than two layers need to be fused, new syntax elements (e.g., xxx_num_fusion_layers_minus2 and xxx_fusion_layer_number[i], where "xxx" can be "inter", "intra", etc.) may be added to identify which levels are fused and how. Table 4, for example, gives an example of such syntax for intra coding using MLL fusion adaptation with more than two layers.

[0050] [Table 4] Adding 2 to intra_num_fusion_layers_minus2 specifies the number of layers to be fused for the intra MLL. intra_fusion_layer_number[i] specifies the layer number of the i-th layer to be fused.

[0051] Similar syntax can be applied to other neural networks used in video encoding. Note that patch-based algorithms can produce boundary artifacts at patch boundaries. Blocking rejection filters or NN-based in-loop filters may be added to address such issues.

[0052] Potential scalability The experimental results showed that latent energy (for example, calculated in one embodiment as the mean square of the quantized latent) is concentrated in a small subset of output latent channels. This is particularly true for MLL-based architectures. 〓 shows an example of the collected data, where the neural network was trained on natural or screen images to match a test image. Term q1 represents the low-bitrate case with 192 output latent channels, while q7 represents the high-bitrate case with 320 output latent channels. For example, for the MLL network in Figure 3A, in the case of q1, for natural images, 20 of the 192 channels contain over 99% of the total energy, and for screen content images, 28 of the 192 channels contain the majority of the latent energy. For the corresponding case of q7, the number is 83 out of 320 channels for natural images and 93 out of 320 channels for screen content images. This data demonstrates that NN-based image codecs can be adapted to enable complexity scalability and / or quality scalability. [Table 5]

[0053] Complexity scalability allows the decoder to operate entropy decoding and reconstruction based on available resources limited by the hardware or software capabilities of the device. To support complexity scalability, in some embodiments, latent channels can be sorted based on their energy concentration. For example, in some embodiments, the most dominant latent channels may be stored in a base layer, and then refinement layers for less dominant latent channels may be stored in an incremental way that can reduce the complexity of decoding. Since the sorting can be predefined, no overhead needs to be passed to the decoder. As an example, 192 channels may be used, and the channels may be numbered 0, 1, ..., 191. Then, the order of the encoded channels, e.g., 0, 3, 20, ..., can be explicitly specified. In the decoder, the decoder can simply decode the channels based on its available resources. In other embodiments, it may be permitted to signal the channel order individually or to group channels to save bitrate overhead.

[0054] Potential quality scalability requires consideration of bandwidth adaptation. Bitstreams can be packaged so that either the user or the network can drop potential channels based on bandwidth requirements. To enable this capability, some high-level syntax (HLS) is required, for example, a syntax similar to that used in scalable HEVC for quality scalability (see, for example, Annex F of the HEVC / H.265 specification). More specifically for NN codecs, in one embodiment, the bitstream may first signal how many quality levels it supports. Then, for each network abstraction layer (NAL) unit, only the bitstream relevant to the relative quality level is included. In another example, the channels can be reordered first, and then HLS can be used to signal how many channels there are at each quality level. This allows the user or network to remove non-essential channels from the bitstream based on bandwidth requirements. Note that the complexity scalability and quality scalability described herein are not limited to MLL-based architectures and are applicable to other NN-based codecs as well.

[0055] Example implementation of a computer system Embodiments of the present invention may be implemented in computer systems, systems configured in electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, field-programmable gate arrays (FPGAs), or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific integrated circuits (ASICs), and / or apparatus comprising one or more such systems, devices, or components. Computers and / or ICs may implement, control, or perform instructions relating to multilevel latent fusion in neural networks for image and video coding, such as those described herein. Computers and / or ICs may compute any of the various parameters or values ​​relating to multilevel latent fusion in neural networks for image and video coding as described herein. Embodiments of image and video may be implemented in hardware, software, firmware, and various combinations thereof.

[0056] Certain implementations of the present invention include a computer processor that executes software instructions causing the processor to perform the method of the present invention. For example, one or more processors, such as a display, encoder, set-top box, or transcoder, may implement the method relating to multilevel latent fusion in neural networks for image and video coding as described above by executing software instructions in program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. The program product may include any non-temporary tangible medium that, when executed by a data processor, carries a set of computer-readable signals including instructions causing the data processor to perform the method of the present invention. The program product according to the present invention may be any of a wide variety of non-temporary and tangible forms. The program product may include physical media such as magnetic data storage media including floppy disks and hard disk drives, optical data storage media including CD-ROMs and DVDs, and electronic data storage media including ROMs and flash RAM. The computer-readable signals on the program product may optionally be compressed or encrypted. Where components (e.g., software modules, processors, assemblies, devices, circuits, etc.) are referred to above, unless otherwise indicated, references to such components (including references to “means”) should be interpreted as including any components that perform the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structure performing the function of the illustrated exemplary embodiment of the Invention.

[0057] Equivalents, extensions, substitutes, and others An exemplary embodiment relating to multilevel latent fusion in neural networks for image and video coding is described herein. In the above specification, embodiments of the invention are described with reference to a number of individual details that may differ from implementation to implementation. Thus, the sole and exclusive indicator of what the invention is and what the applicant intends to be the invention is the set of claims issued to this application, including any subsequent amendments, in the specific form in which such claims are permitted. If there are definitions expressly provided herein for terms included in such claims, those definitions shall govern the meaning of such terms as used in the claims. Thus, no limitations, elements, characteristics, features, advantages, or attributes not expressly provided in the claims should in any way limit the scope of such claims. Thus, this specification and the drawings should be considered illustrative and not restrictive. Several aspects are described below. [Aspect 1] A method for compressing and decompressing images using a neural network, wherein the method is: The first step is to receive an input image with an input spatial resolution that should be compressed using latent features; A step of processing the input image using a plurality of successive levels of convolutional networks to generate a potential fused output, wherein for each network level of the plurality of convolutional networks, the output has a lower spatial resolution than its input; A step of quantizing the latent fusion output to generate a quantized fusion latent; The step includes applying arithmetic coding to the quantized fused latent to generate a coded fused latent, To generate potential fusion output, further: Select a latent output from two or more convolutional networks among the aforementioned multiple consecutive levels of convolutional networks; This includes fusing selected latent outputs to generate the latent fused output, method. [Aspect 2] The step of receiving the encoded fused latent; The steps include: decoding the encoded fused latent to generate a decoded fused latent; A step of processing the decoded fused latent using a plurality of successive levels of deconvolutional networks to generate an approximation of the input image at the input spatial resolution, wherein for each network level of the plurality of deconvolutional networks, its output has a higher spatial resolution than its input, The method described in Embodiment 1. [Aspect 3] Given a selected latent level L1 with output L1 and a selected latent level L2 with output L2, where level L2 is after level L1, and the selected latent outputs are merged: An upscaled level L2 output is generated by upscaling the L2 output to match the spatial resolution of the L1 output; The further includes generating the latent fused output based on the upscaled L2 output and the L1 output, The method according to embodiment 1 or 2. [Aspect 4] To generate the aforementioned latent fusion output: The method further includes processing the L1 output with a 1x1 convolutional network and adding the output of the 1x1 convolutional network to the upscaled L2 output to generate the latent fused output. The method according to embodiment 3. [Aspect 5] To generate the aforementioned latent fusion output: The process further includes processing the L1 output using a k×k convolutional network, adding the output of the k×k convolutional network to the upscaled L2 output to generate the latent fused output, where k≧1 is an odd number selected based on one or more characteristics of the input image. The method according to embodiment 3. [Aspect 6] The method according to aspect 5, wherein the characteristics of the input image include one or more of a natural image, a screen content image, an HDR image, a virtual reality image, a computer-generated image, or any abstract feature map extracted from an image. [Aspect 7] To generate the aforementioned fused output potential: The method further includes processing the L1 output with two or more separate convolutional networks, and combining the outputs of the two or more separate convolutional networks with the upscaled L2 output to generate the fused output latent. The method according to embodiment 3. [Aspect 8] The steps include: using a connected network to connect the upscaled L2 output and the L1 output; For each latent feature to be merged, there is a step to generate a weighted map; The further step includes applying the weighted map to the output of the connected network to generate the fused output potential, The method according to any one of embodiments 3 to 7. [Aspect 9] The method according to any one of embodiments 1 to 8, wherein selecting the latent output from two or more convolutional networks includes selecting the latent output from two or more convolutional networks among a plurality of consecutive levels of convolutional networks. [Aspect 10] A method for compressing and decompressing an image using a neural network for generating and processing latent features, the method being: The first step is to receive an input image with an input spatial resolution that should be compressed using latent features; A step of processing the input image using a plurality of successive levels of convolutional networks to generate one or more potential fused outputs, wherein for each level of the plurality of convolutional networks, the output has a lower spatial resolution than its input; A step in which a potential of level L1 having an L1 output and a potential of level L2 having an L2 output are selected, where level L2 is after level L1; The step of generating an upscaled level L2 output by upscaling the L2 output to match the spatial resolution of the L1 output; The steps include: combining the upscaled L2 output and the L1 output to generate a fused L1 output; The steps include: quantizing and encoding the fused L1 output to generate a fused encoded L1 output; The process includes the steps of quantizing and encoding the L2 output to generate an encoded L2 output. method. [Aspect 11] The steps include receiving and decoding the encoded L2 output and the fused encoded L1 output to generate a decoded L2 input and a decoded fused L1 input; A step of extracting an unfused L1 input based on the decoded L2 input and the decoded fused L1 input; The step of applying the unfused L1 input to one or more subsequent deconvolutional networks to generate an approximation of the input image at the input spatial resolution, wherein for each network level of the one or more deconvolutional networks, its output has a higher spatial resolution than its input, further comprising: The method according to embodiment 10. [Aspect 12] The method according to any one of embodiments 3 to 10, wherein the selection of the L1-level latent and the L2-level latent is performed dynamically based on optimizing one or more coding parameters. [Aspect 13] The method according to embodiment 12, wherein one or more encoding parameters include one or more of the following: target encoding bitrate, rate distortion optimization, decoder complexity, or image / video characteristics. [Aspect 14] The process further includes a step of generating metadata related to fusing selected levels of potential, wherein the metadata is: A first flag indicating whether adaptive fusion at each level is enabled; The patch width and patch height values ​​in lumar sample units used for the selected level; An index parameter indicating the selected fusion format from among multiple fusion formats; A first variable indicating the total number of fusion levels used in the selected fusion format; and One or more syntax elements indicating whether fusion is enabled for inter-coding, intra-coding, or motion vector coding. Including one or more of the following: The method according to embodiment 12 or 13. [Aspect 15] A method for compressing and decompressing an image using a neural network for generating and processing latent features, the method being: The first step is to receive an input image with an input spatial resolution that should be compressed using latent features; A step of processing the input image using a plurality of successive levels of convolutional networks to generate one or more potential outputs, wherein for each level of the plurality of convolutional networks, the output has a lower spatial resolution than its input; A step in which a potential of level L1 having an L1 output and a potential of level L2 having an L2 output are selected, where level L2 is after level L1; The steps include: generating a quantized L2 output based on the aforementioned level L2 output; A step of generating a predictive level L1 output based on a predictive neural network (630) and the quantized L2 output; The steps include: subtracting the predicted L1 output from the L1 output to generate the residual L1 latent; The steps include: quantizing and encoding the residual L1 latent to generate an encoded residual L1 output; The process includes the step of encoding the quantized L2 output to generate the encoded L2 output. method. [Aspect 16] The method according to embodiment 15, wherein the predictive neural network has two layers: a spatial upscaler and a deconvolutional network. [Aspect 17] The steps include receiving and decoding the encoded L2 output and generating the decoded L2 output; The steps include receiving and decoding the encoded residual L1 output and generating the decoded residual L1 output; A step of generating a predicted level L1 output by the decoder based on the decoder prediction neural network and the decoded L2 output; The steps include: adding the level L1 output predicted by the decoder to the decoded residual L1 output to generate the decoded L1 output; A step of generating a concatenated L1 output based on the decoded L1 output and the decoded L2 output; The step of applying the concatenated L1 input to one or more subsequent deconvolutional networks to generate an approximation of the input image at the input spatial resolution, wherein for each network level of the one or more deconvolutional networks, its output has a higher spatial resolution than its input, The method described in aspect 15. [Aspect 18] The method according to embodiment 1, further comprising generating the latent fused output, generating and transmitting latent scalability metadata associated with the output latent. [Aspect 19] The method according to embodiment 18, wherein the latent scalability metadata includes complexity scalability parameters and / or quality scalability parameters. [Aspect 20] The method according to embodiment 19, wherein the latent scalability metadata includes information relating to the quantized latent energy content. [Aspect 21] The method according to aspect 20, wherein, under quality scalability, the potential channels transmit from higher energy to lower energy according to their energy levels. [Aspect 22] The method according to embodiment 2, further comprising the step of receiving potential scalability metadata for the encoded fused latent, wherein decoding of the encoded fused latent is performed based on the potential scalability metadata and scalability criteria. [Aspect 23] The method according to aspect 21, wherein the scalability criterion includes complexity scalability, and the encoded fused latent channels are decoded from higher energy to lower energy according to their energy levels. [Aspect 24] A non-temporary computer-readable storage medium storing computer-executable instructions for performing the method described in any one of embodiments 1 to 23 using one or more processors. [Aspect 25] An apparatus comprising a processor and configured to perform the method described in any one of embodiments 1 to 23.

Claims

1. A method for compressing and decompressing images using a neural network, wherein the method is: The first step is to receive an input image with input spatial resolution that should be compressed using latent features; A step of processing the input image using multiple convolutional networks of consecutive levels to generate a latent fused output, wherein for each network level of the multiple convolutional networks, the output has a lower spatial resolution than its input; The steps include: quantizing the latent fusion output to generate a quantized fusion latent; The step includes applying arithmetic coding to the quantized fused latent to generate a coded fused latent, To generate potential fusion output, further: Select a latent output from two or more convolutional networks among the multiple convolutional networks of consecutive levels; This includes fusing the selected latent outputs to generate the latent fused output, The method in question is: The step of receiving the encoded fused latent; The steps include: decoding the encoded fused latent to generate a decoded fused latent; A step of processing the decoded fused latent using a plurality of deconvolutional networks of successive levels to generate an approximation of the input image at the input spatial resolution, wherein for each network level of the plurality of deconvolutional networks, its output has a higher spatial resolution than its input, method.

2. Given a selected latent level L1 with output L1 and a selected latent level L2 with output L2, where level L2 is after level L1, and the selected latent outputs are merged: An upscaled level L2 output is generated by upscaling the L2 output to match the spatial resolution of the L1 output; The further includes generating the latent fused output based on the upscaled L2 output and the L1 output, The method according to claim 1.

3. To generate the aforementioned latent fusion output: The method further includes processing the L1 output with a 1x1 convolutional network and adding the output of the 1x1 convolutional network to the upscaled L2 output to generate the latent fused output. The method according to claim 2.

4. To generate the aforementioned latent fusion output: The process further includes processing the L1 output using a k×k convolutional network, adding the output of the k×k convolutional network to the upscaled L2 output to generate the latent fused output, where k≧1 is an odd number selected based on one or more characteristics of the input image. The method according to claim 2.

5. The method according to claim 4, wherein the characteristics of the input image include one or more of a natural image, a screen content image, an HDR image, a virtual reality image, a computer-generated image, or any abstract feature map extracted from an image.

6. To generate the latent fusion output: The method further includes processing the L1 output with two or more separate convolutional networks, and combining the outputs of the two or more separate convolutional networks with the upscaled L2 output to generate the latent fused output. The method according to claim 2.

7. The steps include: using a connected network to connect the upscaled L2 output and the L1 output; For each latent feature to be merged, there is a step to generate a weighted map; The further step includes applying the weighted map to the output of the connected network to generate the potential fused output, The method according to claim 2.

8. The method according to claim 1, wherein selecting the latent output from two or more convolutional networks includes selecting the latent output from two or more convolutional networks among a plurality of convolutional networks of consecutive levels.

9. A method for compressing and decompressing an image using a neural network for generating and processing latent features, the method being: The first step is to receive an input image with input spatial resolution that should be compressed using latent features; A step of processing the input image using a plurality of convolutional networks of successive levels to generate one or more potential fused outputs, wherein for each level of the plurality of convolutional networks, the output has a lower spatial resolution than its input; The stage of selecting between a potential level L1 having an L1 output and a potential level L2 having an L2 output, where level L2 is after level L1; The steps include: generating an upscaled level L2 output by upscaling the L2 output to match the spatial resolution of the L1 output; The steps include: combining the upscaled L2 output and the L1 output to generate a fused L1 output; The steps include: quantizing and encoding the fused L1 output to generate a fused encoded L1 output; The process includes the steps of quantizing and encoding the L2 output to generate an encoded L2 output. method.

10. The steps include receiving and decoding the encoded L2 output and the fused encoded L1 output to generate a decoded L2 input and a decoded fused L1 input; A step of extracting an unfused L1 input based on the decoded L2 input and the decoded fused L1 input; The step of applying the unfused L1 input to one or more subsequent deconvolutional networks to generate an approximation of the input image at the input spatial resolution, wherein for each network level of the one or more deconvolutional networks, its output has a higher spatial resolution than its input, further comprising: The method according to claim 9.

11. The method according to claim 2, wherein the selection of the latent level L1 and the latent level L2 is performed dynamically based on optimizing one or more coding parameters.

12. The method according to claim 11, wherein one or more encoding parameters include one or more of the following: target encoding bitrate, rate distortion optimization, decoder complexity, or image / video characteristics.

13. The process further includes a step of generating metadata related to fusing selected levels of potential, wherein the metadata is: A first flag indicating whether adaptive fusion at different levels is enabled; The patch width and patch height values ​​in lumar sample units used for the selected level; An index parameter indicating the selected fusion format from among multiple fusion formats; A first variable indicating the total number of fusion levels used in the selected fusion format; and One or more syntax elements indicating whether fusion is enabled for inter-coding, intra-coding, or motion vector coding. Including one or more of the following: The method according to claim 11.

14. A method for compressing and decompressing an image using a neural network for generating and processing latent features, the method being: The first step is to receive an input image with input spatial resolution that should be compressed using latent features; A step of processing the input image using a plurality of convolutional networks of successive levels to generate one or more potential outputs, wherein for each level of the plurality of convolutional networks, the output has a lower spatial resolution than its input; The stage of selecting between a potential level L1 having an L1 output and a potential level L2 having an L2 output, where level L2 is after level L1; The steps include: generating a quantized L2 output based on the aforementioned L2 output; A step of generating a predicted L1 output based on a predictive neural network (630) and the quantized L2 output; The steps include: subtracting the predicted L1 output from the L1 output to generate the residual L1 latent; The steps include: quantizing and encoding the residual L1 latent to generate an encoded residual L1 output; The process includes the step of encoding the quantized L2 output to generate the encoded L2 output. method.

15. The method according to claim 14, wherein the predictive neural network has two layers: a spatial upscaler and a deconvolutional network.

16. The steps include receiving and decoding the encoded L2 output and generating the decoded L2 output; The steps include receiving and decoding the encoded residual L1 output and generating the decoded residual L1 output; The steps include: generating a predicted level L1 output by the decoder based on the decoder prediction neural network and the decoded L2 output; The steps include: adding the level L1 output predicted by the decoder to the decoded residual L1 output to generate a decoded L1 output; A step of generating a concatenated L1 output based on the decoded L1 output and the decoded L2 output; The step of applying the concatenated L1 input to one or more subsequent deconvolutional networks to generate an approximation of the input image at the input spatial resolution, wherein for each network level of the one or more deconvolutional networks, its output has a higher spatial resolution than its input, The method according to claim 14.

17. The method according to claim 1, further comprising generating the latent fusion output, generating and transmitting latent scalability metadata associated with the latent output.

18. The method according to claim 17, wherein the latent scalability metadata includes complexity scalability parameters and / or quality scalability parameters.

19. The method according to claim 18, wherein the latent scalability metadata includes information relating to the quantized latent energy content.

20. The method according to claim 19, wherein, under quality scalability, the latent channels transmit from higher energy to lower energy according to their energy levels.

21. The method according to claim 1, further comprising the step of receiving potential scalability metadata for the encoded fused latent, wherein decoding of the encoded fused latent is performed based on the potential scalability metadata and scalability criteria.

22. The method according to claim 21, wherein the scalability criterion includes complexity scalability, and the encoded fused latent channels are decoded from higher energy to lower energy according to their energy levels.

23. A non-temporary computer-readable storage medium storing computer-executable instructions for performing the method according to any one of claims 1 to 22 using one or more processors.

24. An apparatus comprising a processor and configured to perform the method according to any one of claims 1 to 22.

25. The method according to claim 1, wherein the selection of latent outputs from two or more convolutional networks is performed dynamically based on optimizing one or more coding parameters.

Images