Method for learned image compression and related autoencoder

Abstract

It is described a method for learned image compression implemented in an autoencoder (1') comprising an image reconstruction encoder (fa) and an image reconstruction decoder (fs), said method comprising the steps of: a) extracting from an image (x) a main latent representation (y) by means of said image reconstruction encoder (fa); b) quantizing said main latent representation (y) by means of a quantizer (2) to obtain a discrete main latent representation (ŷ); c) entropy coding said discrete main latent representation (ŷ) by means of an entropy encoder (4) to obtain a bitstream (5); d) entropy decoding said bitstream (5) by means of an entropy decoder (6) to obtain an entropy decoded bitstream; e) feeding said entropy decoded bitstream to said image reconstruction decoder (fs); f) recovering a reconstructed image (x ̂) by means of said image reconstruction decoder (fs); g) training said autoencoder (3) via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing said discrete main latent representation (ŷ) by means of a soft-quantized main latent representation (ỹ), wherein said quantizer (2) is implemented as a parameterized activation function (I) wherein L is a number of sought quantization levels, β is an inverse temperature parameter regulating a slope of said quantization levels, and w and b are parameters respectively determining reconstruction levels and width of quantization intervals of said soft-quantized main latent representation (ỹ).

Description

[0001] METHOD FOR LEARNED IMAGE COMPRESSION AND RELATED

[0002] AUTOENCODER

[0003] DESCRIPTION

[0004] The present invention relates to a method for learned image compression according to the preamble of claim 1.

[0005] The present invention also relates to an autoencoder implementing the method for learned image compression.

[0006] Learned image compression (LIC) has seen much interest since it has achieved compression efficiency comparable to standardized codecs (see Ma, Siwei et al., "Image and video compression with neural networks: A review", IEEE Transactions on Circuits and Systems for Video Technology, 2019, reference [1]).

[0007] At the transmitter side, the image is first projected into a lower-dimensional latent representation via a convolutional encoder. Next, the latent representation is quantized and entropy-coded, yielding a compressed representation of the picture in the form of a binary bitstream. At the receiver side, this representation is reversed and projected back to the pixel domain by a decoder, obtaining a lossy representation of the original image.

[0008] Such encoder-decoder (autoencoder) models are trained end-to-end via back- propagation of the error gradient to minimize some rate-distortion (RD) cost function in the form

[0009] £ = R + - D (1) where l is a hyper-parameter that regulates the trade-off between rate R of the latent representation and distortion D of the reconstructed image. For example, a larger A places more importance on reconstruction quality at the expense of the rate of the latent space.

[0010] The quantization of the latent representation is a crucial aspect: since quantization represents a non-differentiable function, it cannot be easily incorporated into error gradient back-propagation.

[0011] A common approach in LIC is to replace quantization with additive uniform noise, ensuring resilience to quantization errors.

[0012] Most existing approaches to learned image compression all share the same shortcomings, i.e. a separate encoder-decoder model with millions of parameters must be trained for each different bitrate, with two important implications. First, training a different model from scratch for each rate incurs high costs in terms of both energy and time. Second, the need to store a different model for each rate is a significant drawback, especially for resource-constrained devices.

[0013] These issues jeopardize the feasibility of learned image compression in real-world scenarios, where controlling the rate is of paramount importance.

[0014] While solutions have been proposed recently, the issue of variable bitrate (VBR) image compression is far from being solved.

[0015] Learned image compression fundamentals

[0016] Learned Image Compression models have shown the potential to match or even outperform standardized codecs such as the recent H.266 / VVC (see Bross B. et al., "Versatile video coding", JVET, 2020, reference [2]) in RD terms.

[0017] Early seminal works such as Balle J., Laparra V. and Simoncelli E., "End-to-end optimized image compression", Proceedings of the International Conference on Learning Representations (ICLR), 2017, (reference [4]), and Theis L., Shi W., Cunningham A. and Huszar F., "Lossy image compression with compressive autoencoders", International Conference on Learning Representations (ICLR), 2017, (reference [5]), exploited a simple convolutional autoencoder structure with a unique latent representation modeled with a fully factorized distribution among channels modeled either analytically (see Presta A. et al., "A differential entropy model for learned image compression", International Conference on Learning Representations (ICLR), 2017, reference [6]) or through an auxiliary neural network, and exploiting Generalized Divisive Normalization (GDN) activation functions (see Balle J., Laparra V. and Simoncelli. E.P., "Density modeling of images using a generalized normalization transformation", International Conference on Learning Representations (ICLR), 2016, reference [7]). The scheme was improved by introducing a pyramid-based architecture composed of two nested variational autoencoders (see Balle J., Minnen D., Singh S., Hwang S.J. and Johnston N., "Variational image compression with a scale hyperprior", Inti. Conf, on Learning Representations (ICLR), 2018, reference [8]). Here, the first is called the hyperprior and captures spatial correlation within the image, while the second one models the latent representation, which is supposed to follow a zero-mean Gaussian distribution.

[0018] Minnen D., Balle J. and Toderici, G.D., "Joint autoregressive and hierarchical priors for learned image compression", Advances in neural information processing systems, 2018, reference [9], Lee J. et al., "Context- Adaptive Entropy Model for End- to-end optimized Image Compression", Inti. Conf, on Learning Representations (ICLR), 2019, reference

[0010] , and Minnen D. and Saurabh S, "Channel-wise autoregressive entropy models for learned image compression", IEEE International Conference on Image Processing (ICIP), 2020, reference

[0011] , combined previous architecture with a context-based auto-regressive entropy model to capture more local spatial correlation by exploiting already decoded parts of images.

[0019] Similarly, Li M. et al., "Efficient and effective context-based convolutional entropy modeling for image compression", IEEE Transactions on Image Processing, 29, 2020, pp. 5900-5911, reference

[0012] , improved context modeling through a 3D zigzag scanning order, and improved parallelism entropy decoding with a 3D code dividing technique by partitioning the latent representation in multiple independent groups.

[0020] In Li, Mu et al., "Learning context-based nonlocal entropy modeling for image compression", IEEE Transactions on Neural Networks and Learning Systems, 2021, [reference 13], a special non-local operation is proposed to consider global similarity within the context, by introducing U-net-like blocks, while T. Chen et al., "End-to- End Learnt Image Compression via Non-Local Attention Optimization and Improved Context Modeling", IEEE Transactions on Image Processing, 2021, [reference 14], introduced non-local network operations as non-linear transforms in both latent representations. In more recent works other techniques have been exploited to improve RD performance.

[0021] Cheng Z. et al., "Learned image compression with discretized gaussian mixture likelihoods and attention modules", Conference on Computer Vision and Pattern Recognition, 2020, [reference 15], replaced simple Gaussian distribution with a mixture of Gaussians and introduced some attention module to enhance entropy estimation, while Fu H. et al., "Learned Image Compression with Gaussian- Laplacian-Logistic Mixture Model and Concatenated Residual Modules", IEEE Transactions on Image Processing, 2023, [reference 16], introduced a more flexible discretized Gaussian-Laplacian-Logistic mixture model for the latent representation.

[0022] In Zou, Renjie, Chunfeng Song and Zhaoxiang Zhang, "The devil is in the details: Window-based attention for image compression", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022, [reference 17], local attention is exploited to combine the local-aware attention with the global-related feature learning and to the present date is among the best-performing architectures. Z. Tang H. et al., "Joint Graph Attention and Asymmetric Convolutional Neural Network for Deep Image Compression," IEEE Transactions on Circuits and Systems for Video Technology, 2023, [reference 18], proposed a self-attention mechanism based on graphs to improve entropy estimation, and other threads of works tried to enhance some aspects of previous models.

[0023] Xie Y., Cheng K. L. and Chen Q., "Enhanced invertible encoding for learned image compression", Proceedings of the 29th ACM international conference on multimedia, 2021, [reference 19], for example extracted better transformation between image and latent features space by exploiting invertible networks, while Jacobsen J.-EL, Smeulders A. and Oyallon E., "i-revnet: Deep invertible networks", International Conference on Learning Representations (ICLR), 2018, [reference 20], and Zhu Y. et al., "Transformer-based transform coding", International Conference on Learning Representations (ICLR), 2022, [reference 21], replaced convolutional modules with the Swin transformer to achieve better compression efficiency with fewer parameters.

[0024] On the other hand, Y. Wu et al., "Learned Block-Based Hybrid Image Compression", IEEE Transactions on Circuits and Systems for Video Technology, 2022, [reference 22], focused on the optimization of the image decoding through a learned block-based framework. Wang D. et al., "Neural data-dependent transform for learned image compression", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, [reference 23], tried to adapt the entire structure to single images with an additional model stream to generate the transform parameters at the decoder side, and He, Dailan et al., "Checkerboard context model for efficient learned image compression", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, [reference 24], introduced a checkerboard context model to improve efficiency during the decoding stage.

[0025] In Duan Z. et al., "Lossy Image Compression with Quantized Hierarchical VAEs", Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2023, [reference 25], hierarchical VAE (Variational Auto Encoder) architecture, originally designed for generative image modeling, is exploited for lossy image compression, redefining their probabilistic model to allow easy quantization and practical entropy coding.

[0026] In addition to the aforementioned variational autoencoder-based approaches, other approaches have been explored for compressing images using neural networks. Agustsson, E., et al., "Generative adversarial networks for extreme learned image compression", Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, [reference 26], and Mentzer, F. et al., "High-fidelity generative image compression", Advances in Neural Information Processing Systems 33, 2020, [reference 27], exploited GAN-based architecture to reduce image compression artifacts, especially at extremely low bitrates.

[0027] The common feature shared by all the mentioned works is that a single model targets a single rate, meaning that multiple models must be trained and stored to cover the range of rates required in practical image or video compression scenarios. This issue is critical since each model includes tens of millions of learnable parameters. The issue with variable rate image compression can be also related to the way quantization takes place, as discussed in the following.

[0028] Quantized latent representations for end-to-end learning

[0029] In end-to-end image compression, latent space quantization is critical because of its non-differentiability: the gradient of the quantization function is zero everywhere apart from the boundaries among quantization levels, where it is undefined. The main approaches to this problem are now reviewed, such approaches revolving around replacing quantization with an approximated function.

[0030] 1) Straight-through estimator: utilized in reference [5], this method involves substituting the derivative of the quantization with a smooth approximation during the backward step while retaining the original function during the forward step. In particular, such method exploits the linear function as a derivative, since it is easy to implement as it brings no modification to the gradient. This method lacks elasticity since it does not allow for gradual relaxation during training, thus enforcing quantization on integers even during training.

[0031] 2) Additive Uniform noise: introduced in reference [4], it consists in replacing the actual quantizer with additive uniform noise during training.

[0032] The benefit of this method is that the density function associated with the noisy latent space represents a continuous relaxation of the discrete density mass found in the quantized space: moreover, independent uniform noise is commonly employed as a representation of quantization error due to its ability to approximate the marginal moments of the error (see Gray, Robert M. and David L. Neuhoff, "Quantization", IEEE transactions on information theory, 1998, [reference 28]).

[0033] In addition to these two aspects, adding uniform noise allows to reframe the training as a variational optimization problem, see reference [4], resulting in more effective learning of flexible latent space.

[0034] All these benefits have made this method one of the most widely used for approximating quantization during learning, see references [4], [8], [9],

[0015] , thus becoming a de-facto standard. However, this method has one significant drawback: it allows quantization only on integers, without the possibility of adapting quantization intervals. This happens because there is no way to control the latent space since it is not parameterized, which forces training a different model for each RD trade-off.

[0035] 3) Soft to Hard annealing: these methods are based on annealing a parameter, called temperature, to approximate quantization.

[0036] The main concept is to decrease this hyper-parameter during training to gradually constrain the latent representation, moving towards a hard quantization shape to ultimately freeze it.

[0037] For example, using the softmax function over a partition of the latent space in Voronoi tessellation over centers enables soft quantization in Agustsson E. et al., "Soft-to-hard vector quantization for end-to-end learning compressible representations", Advances in neural information processing systems, 2017, [reference 29], and in Guo, Zongyu et al., "Soft then hard: Rethinking the quantization in neural image compression", International Conference on Machine Learning, PMLR, 2021, [reference 30].

[0038] Agustsson E. and Theis L., "Universally quantized neural compression", Advances in neural information processing systems, 2020, Vol. 33, [reference 31], combine additive uniform noise and a new variant of softmax quantization to bridge the gap between quantized and continuous latent space, achieving robustness to quantization errors. Despite efficacy, this method suffers from several limitations; first, it is not agnostic to the entropy estimation since it cannot handle Gaussian distribution as prior, second, it introduces a further term in the loss function, and third it is not focused on variable rate adaptation.

[0039] The approach in Shlezinger, Nir and Eldar, Yomina C, "Deep task-based quantization", Entropy, 2021, reference

[0032] , deals with multiple-input multipleoutput (MIMO) communications and relies on a finite summation of hyperbolic tangents to overcome the non-differentiability of the quantization step. While in the present application the same conceptual framework is shared, the present application deals with the specific and different challenges of image compression. Also, reference

[0032] requires annealing a specific temperature for each hyperbolic tangent, requiring tuning as many parameters as the quantization steps.

[0040] Almost all the above approaches to quantization entail the same drawback, i.e., a different encoder-decoder model must be trained for each rate. Several proposals have been made towards variable rate learned image compression and some of them are reviewed in the following.

[0041] In reference [5], the latent representation of a single autoencoder is scaled before quantization by adding a learnable scaling parameter, one for each channel; this allows adaptation of the latent space based on the required bitrate. However, using a single value for each channel to adapt the quality level may lead to a reduction in R-D performance.

[0042] Choi, Y., Mostafa E. and Jungwon L., "Variable rate deep image compression with a conditional autoencoder", Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, [reference 33], proposed an autoencoder that is conditioned on the Lagrangian multiplier X, that is not treated as a regular hyperparameter as usual but is instead used as input to the network to produce specific latent representations. Additionally, the network was trained using mixed quantization bin sizes, enabling it to adapt the rate by adjusting the bin size of the quantization applied to the latent representation. This model increases the complexity of the optimization since it introduces both X and the bin size as inputs to determine the target bitrates.

[0043] Another approach is to deploy a model that considers different resolutions at the same time in the same training phase, making it adaptable to different RD tradeoffs.

[0044] Yang F. et al., "Variable rate deep image compression with modulated autoencoder", IEEE Signal Processing Letters, 2020, [reference 34], presented a problem of optimizing variable RD, which involves adding a modulated framework to the deep image compression structure. This framework enables the structure to adapt to various levels of compression. In C. Cai, L. Chen, X. Zhang and Z. Gao, "Efficient Variable Rate Image Compression With Multi-Scale Decomposition Network", IEEE Transactions on Circuits and Systems for Video Technology, 2019, reference

[0035] , the autoencoder is trained to break down the input image into multiple levels of representations, aiming to optimize the rate-distortion performance across all scales.

[0045] However, both reference

[0034] and reference

[0035] made the training phase more complicated and unstable.

[0046] In Cui, Ze et al., "Asymmetric gained deep image compression with continuous rate adaptation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, reference

[0036] , a pair of parametric gain units are inserted before and after the quantization step to achieve discrete rate adaptation with one single model; by using exponential interpolation, continuous rate adaptation is achieved without compromising performance. In particular, for each target level, there is a specific pair of gain units multiplying the latent representation element- wise before and after quantization. However, beyond gain units, reference

[0036] modifies the entropy model, passing from symmetric to asymmetric Gaussian distribution, which makes it complex to assess the benefits of gain units in isolation. Following a different approach, Jooyoung L., Seyoon J., and Munchurl K., "Selective compression learning of latent representations for variable-rate image compression", Thirty-Sixth Conference on Neural Information Processing Systems, 2022, reference

[0037] , introduced a 3D importance map to achieve essential representations for compression at various quality levels, encoding thus only a partial version of the latent representation based on the desired quality level. Furthermore, they exploited quality adapter quantization by multiplying the latent representation by a quantization vector, similarly to reference

[0036] . Despite effectiveness, reference

[0037] does require learning the importance map, which involves training thousands of parameters.

[0047] Similarly to reference

[0036] , Chen, Tong, and Zhan Ma, "Variable bitrate image compression with quality scaling factors", International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, reference

[0038] , embeds a set of quality scaling factors (SFs) into a model, by which they can encode images across an entire bitrate range with a single model; however, this approach shares the same limitations of reference

[0036] .

[0048] In Gao, Chenjian, et al. "Flexible neural image compression via code editing", Advances in Neural Information Processing Systems, 35, 2022, reference

[0039] , a "per image" optimal representation is obtained by applying SGD to the latent space and determining the quantization step using grid-search. This approach is limited by the need for optimization when coding every single image, making it unfeasible in most use cases that impose constraints on computational costs or real-time capacity. Guo-Hua Wang, Li Jiahao, Li Bin and Lu Yan, "EVC: Towards Real-Time Neural Image Compression with Mask Decay", Eleventh International Conference on Learning Representations, 2023, reference

[0040] , exploited mask parameter decay, adjustable quantization step, and knowledge distillation to train a smaller model from a teacher model; variable bit-rate is obtained using adjustable quantization steps. A large model is pruned using learnable mask decay layers. However, this method relies on a multi-stage training phase that is very complex, and it is not agnostic with respect to the architecture, jeopardizing the possibility to add this technique in a general image compression model.

[0049] In general, the problem of adapting a single model to different rates has received less attention as more efforts have been focused on improving compression performance. Nonetheless, the ability to efficiently switch among different rates is a fundamental requirement for practical image coding.

[0050] The present invention aims at solving these and other problems by providing a method for learned image compression, and a related autoencoder, that reduces the overall training complexity when training a model at different bitrates.

[0051] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that allows to adapt the same model to different bitrates.

[0052] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that are suitable for any learnable image compression scheme.

[0053] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that save energy and processing time when training a model at different bitrates.

[0054] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that is applicable to resource-constrained devices used in real- wo rid scenarios.

[0055] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that is agnostic to the prior.

[0056] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that can be easily plugged into different types of models.

[0057] A further scope of the present invention is to provide a method for learned image compression, and a related autoencoder, that does not add additional terms in the loss function to minimize the quantization error.

[0058] In a nutshell, in order to build a variable rate model, the present invention exploits an activation function called STanH, a differentiable quantization layer that can be plugged into a pre-trained codec to achieve variable bitrates.

[0059] The activation function STanH is designed around a finite summation of hyperbolic tangents that relaxes the quantization function during training.

[0060] The relaxation is controlled by a single temperature that is annealed during training to approach the desired quantization levels.

[0061] The activation function STanH can be implemented as a differentiable quantization layer with just a few hundred learnable parameters, allowing standard training via end-to-end error back-propagation.

[0062] With respect to comparable methods, the activation function STanH directly manipulates the latent representation, determining uniquely the quantization levels. This allows moving from fine to coarse-grained quantization simply refining the parameters of the quantization layer, without retraining the other model parameters. The method according to the present invention allows switching encoding rate by simply switching the quantization layer, reducing the memory requirements for storing models, avoiding long training times, and reducing energy requirements. The method according to the present invention is experimented with three different state-of-the-art image compression architectures and it is shown that it allows variable rate coding with negligible impact on encoding efficiency.

[0063] To the best of the Applicants' knowledge, the present application is the first to show that a trained LIC model can be turned into a variable rate one simply by plugging a learnable quantization layer.

[0064] The method according to the present invention requires tuning a single value to control the annealing of a specific temperature.

[0065] According to a first embodiment of the method according to the invention, it is described a method for learned image compression implemented in an autoencoder comprising an image reconstruction encoder and an image reconstruction decoder, said method comprising the steps of: a) extracting from an image a main latent representation by means of the image reconstruction encoder; b) quantizing the main latent representation by means of a quantizer to obtain a discrete main latent representation; c) entropy coding the discrete main latent representation by means of an entropy encoder to obtain a bitstream; d) entropy decoding the bitstream by means of an entropy decoder to obtain an entropy decoded bitstream; e) feeding the entropy decoded bitstream to the image reconstruction decoder; f) recovering a reconstructed image by means of the image reconstruction decoder; g) training the autoencoder via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing the discrete main latent representation by means of a soft-quantized main latent representation, wherein the quantizer is implemented as a parameterized activation function wherein L is a number of sought quantization levels, [3 is an inverse temperature parameter regulating a slope of the quantization levels, and w and b are parameters respectively determining reconstruction levels and width of quantization intervals of the soft-quantized main latent representation.

[0066] According to an aspect of the method according to the invention, the parameter w for determining the reconstruction levels and the parameter b for determining the width of the quantization intervals are learned to minimize the rate distortion cost function L.

[0067] According to a further aspect of the method according to the present invention, the rate distortion cost function L is in the form J = R + A ■ D where A is a hyperparameter that regulates the trade-off between the bitrate R of the main latent representation and the distortion D of the reconstructed image.

[0068] According to a further aspect of the method according to the present invention, the inverse temperature parameter [3 is gradually increased during training with an annealing procedure.

[0069] According to a further aspect of the method according to the present invention, the annealing procedure relies on a function that considers both the number of training iterations and the difference between the soft-quantized main latent representation and the discrete main representation.

[0070] According to a further aspect of the method according to the present invention, the inverse temperature parameter [3 is computed as:

[0071] / A - 1 wherein t is t-th step of the training, wherein A are respectively quantization errors of the soft-quantized main latent representation and of the discrete quantized latent representation, wherein is the error between the quantization errors at training step t, wherein ptrepresents the value used at the t-th step, and wherein K is a factor that regulates the velocity with which the soft-quantized main latent representation is frozen.

[0072] According to a second embodiment of the method according to the present invention, it is described a method for learned image compression implemented in an autoencoder comprising an image reconstruction encoder and an image reconstruction decoder, the method comprising the steps of: a) extracting from an image a main latent representation by means of the image reconstruction encoder and a hyperprior latent representation by means of a hyperprior analysis transform block; b) quantizing the main latent representation by means of a first quantizer to obtain a discrete main latent representation and the hyperprior latent representation by means of a second quantizer to obtain a discrete hyperprior latent representation; c) entropy coding the discrete main latent representation by means of a first entropy encoder to obtain a first bitstream and the discrete hyperprior latent representation by means of a second entropy encoder to obtain a second bitstream; d) entropy decoding the first bitstream by means of a first entropy decoder to obtain a first entropy decoded bitstream and the second bitstream by means of a second entropy decoder to obtain a second entropy decoded bitstream; e) feeding the first and second entropy decoded bitstreams to the image reconstruction decoder; f) recovering a reconstructed image by means of the image reconstruction decoder; g) training the autoencoder via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing the discrete main latent representation by means of a soft-quantized main latent representation and the discrete hyperprior latent representation by means of a soft- quantized hyperprior latent representation, wherein the quantizers are implemented as respective parameterized activation functions wherein wherein f indicates the f-th training step, L is a number of sought quantization levels, p is an inverse temperature parameter regulating a slope of the quantization levels, w and b are parameters respectively determining reconstruction levels and width of quantization intervals of the soft-quantized main latent representations, and the subscripts {h,m} refer to the two different latent representations, namely the hyperprior and the main one, respectively.

[0073] According to a further aspect of the method according to the present invention, the rate distortion cost function L is wherein d is a distortion metric, X is a hyper-parameter that regulates the trade-off between bitrate R of the main latent representation and distortion D of the reconstructed image, represents a rate of the hyperprior latent representation, and represents a rate of the main latent representation from which the image is reconstructed.

[0074] According to a further aspect of the method according to the present invention, the inverse temperature parameters P ,t and pm,t are gradually increased during training with an annealing procedure.

[0075] According to a further aspect of the method according to the present invention, the annealing procedure relies on a function that considers both the number of training iterations and a difference between the soft-quantized main latent representation and the discrete main representation and a difference between the soft-quantized hyperprior latent representation and the discrete hyperprior latent representation.

[0076] According to a further aspect of the method according to the present invention,-the inverse temperature parameter pm,tis computed as: wherein t is t-th step of the training, wherein are respectively quantization errors of the soft-quantized main latent representation (y) and of the discrete quantized latent representation (y), wherein is the error between the quantization errors at training step t, and wherein K is a factor that regulates the velocity with which the self-quantized latent representation is frozen; and the inverse temperature parameter h,t is computed as: wherein t is t-th step of the training, wherein et= ||z — z||2and et= ||z — z||2are respectively quantization errors of the soft-quantized hyperprior latent representation (z) and of the discrete quantized hyperprior representation (z), Et= |et— et] is the error between the quantization errors at training step t, and wherein K is a factor that regulates the velocity with which the soft-quantized hyperprior latent representation (z) is frozen.

[0077] According to a further embodiment of the present invention, it is disclosed an autoencoder for learned image compression comprising:

[0078] - an image reconstruction encoder configured to extract from an image a main latent representation;

[0079] - a quantizer configured to quantize the main latent representation to obtain a discrete main latent representation;

[0080] - an entropy encoder configured to entropy coding the discrete main latent representation to obtain a bitstream;

[0081] - an entropy decoder configured to entropy decoding the bitstream to obtain an entropy decoded bitstream;

[0082] - an image reconstruction decoder configured to receive the entropy decoded bitstream;

[0083] - an image reconstruction decoder configured to recover a reconstructed image;

[0084] - means for training the autoencoder via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing the discrete main latent representation by means of a soft- quantized main latent representation, wherein said quantizer is implemented as a parameterized activation function wherein L is a number of sought quantization levels, [3 is an inverse temperature parameter regulating a slope of the quantization levels, and w and b are parameters respectively determining reconstruction levels and width of quantization intervals of the soft-quantized main latent representation.

[0085] According to a further embodiment of the present invention, it is described an autoencoder for learned image compression comprising:

[0086] - an image reconstruction encoder configured to extract from an image a main latent representation and a hyperprior analysis transform block configured to extract from said image a hyperprior latent representation;

[0087] - a first quantizer configured to quantize the main latent representation to obtain a discrete main latent representation and a second quantizer configured to quantize the hyperprior latent representation to obtain a discrete hyperprior latent representation;

[0088] - a first entropy encoder configured to entropy coding the discrete main latent representation to obtain a first bitstream and a second entropy encoder configured to entropy coding the discrete hyperprior latent representation to obtain a second bitstream;

[0089] - a first entropy decoder configured to entropy decoding the first bitstream and a second entropy decoder configured to entropy decoding the second bitstream to obtain a second entropy decoded bitstream;

[0090] - an image reconstruction decoder configured to receive the first and second entropy decoded bitstreams;

[0091] - an image reconstruction decoder configured to recover a reconstructed image;

[0092] - means for training the autoencoder via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing the discrete main latent representation by means of a soft- quantized main latent representation and the discrete hyperprior latent representation by means of a soft-quantized hyperprior latent representation, wherein the first and second quantizers are implemented as respective parameterized activation functions wherein f indicates the f-th training step, L is a number of sought quantization levels, |3 is an inverse temperature parameter regulating a slope of the quantization levels, w and b are parameters respectively determining reconstruction levels and width of quantization intervals of the soft-quantized main latent representations, and the subscripts {h,m} refer to the two different latent representations, namely the hyperprior and the main one, respectively.

[0093] According to a further embodiment of the method according to the present invention, it is a described a method for variable rate image compression implemented in a device, the method comprising:

[0094] - training one or more anchor models from scratch by selecting appropriate X values for placement across a target rate-distortion (RD) curve;

[0095] - generating one or more derivations from each of the trained anchor models by cloning the trained anchor model and setting a lower X to target a lower rate;

[0096] - resuming the training procedure for each derivation, wherein only the quantization layers related to the parameterized activation function according to the invention are refined, and all other parameters of the derivation are kept unchanged from the reference anchor model;

[0097] - storing only the refined quantization layers after the refinement process is completed, wherein all layers of the derivation, except for the refined quantization layers, are identical to the anchor model; and

[0098] - enabling variable rate image compression in said device by replacing a current quantization layer with that of a refined anchor to switch rates without the necessity of training a separate model for each rate.

[0099] The invention will be described in detail hereinafter through non-limiting embodiments with reference to the attached Figures, wherein:

[0100] Figure 1 schematically represents a prior art reference learned image compression architecture comprising two quantization layers according to the invention for quantizing a main latent space and a hyperprior latent space;

[0101] Figure la schematically represents a channel-wise autoregressive entropy model block of Figure 1;

[0102] Figure 2 represents a plot of an activation function according to the invention;

[0103] Figure 3 represents a table showing parameters used to define anchors for prior art reference models, i.e. models where all layers are trained end-to-end;

[0104] Figure 4 shows a table representing values used for training derivations, i.e. models where only the quantization layers according to the invention are refined;

[0105] Figures 5(a)-5(f) represent plots showing rate-distortion performance of the prior art reference architecture of Figure 1 on Kodak dataset using different number of anchors, from six to one;

[0106] Figure 6 shows a table representing BD-Rate and BD-PSNR with respect to the prior art reference architecture of Figure 1 on the Kodak test set for different numbers of anchors, wherein savings are reported in terms of trainable parameters;

[0107] Figures 7(a)-7(i) represent plots showing Rate-PSNR for the method according to the present invention and relative prior art reference for Kodak, Clic, and Tecnik datasets and for three anchors and three derivations; Figures 8(a)-8(c) represent MS-SSIM plots for the method according to the present invention and relative prior art reference for Kodak dataset and for three anchors and three derivations;

[0108] Figure 9 shows a table representing BD-Rate and BD-PSNR on the Kodak dataset for three anchors, wherein savings are reported in terms of trainable parameters;

[0109] Figure 10 shows a table representing costs of the method according to the invention in terms of various parameters;

[0110] Figure 11 represents a RD-plot for the model according to the present invention and corresponding prior art references for Kodak data set;

[0111] Figure 12 shows a table representing some parameters of the method according to the invention with respect to the prior art over a H266 / WC reference;

[0112] Figure 13 shows a RD-plot for a model according to the invention for Kodak data set with respect to a prior art architecture;

[0113] Figure 14 shows a radar plot comparing the prior art reference and the method according to the invention in terms of various parameters;

[0114] Figure 15 shows a prior art reference learned image compression architecture comprising a quantization layer according to the invention for quantizing a main latent space.

[0115] Figure 1 schematically represents an autoencoder 1 for learned image compression, i.e. an architecture based on reference

[0017] and using a Convolutional Neural Network, CNN, wherein a first quantizer 2 and a second quantizer 3 according to the present invention are integrated.

[0116] An image reconstruction encoder faprojects an input image x onto a low dimensional, main latent representation y of dimension M, representing the number of channels, which is then quantized by the first quantizer 2, thereby obtaining a discrete main latent representation y : y is referred to as main latent representation, since the image is directly reconstructed from it.

[0117] Moreover, the main latent representation y is further projected, by means of a hyperprior analysis transform block ha, into a hyperprior latent representation z = ha(y) of dimension N which is then quantized, via a second quantizer 3, to a discrete hyperprior latent representation z.

[0118] To allow the transmission of the discrete hyperprior latent representation z from the encoding side to the decoding side of the autoencoder 1, a first entropy encoder 7, in particular an arithmetic encoder, is configured to process the discrete hyperprior latent representation z to provide a first compressed latent space 8 in the form of a bitstream which is then fed to a first entropy decoder 9, in particular an arithmetic decoder, for providing a first decompressed latent space corresponding to the discrete hyperprior latent representation z.

[0119] The discrete hyperprior latent representation z is exploited to find the spatial correlation for entropy estimation in a channel-wise autoregressive entropy model block 10, see reference

[0011] , where channel-conditioning and latent residual prediction have been introduced, for enhancing rate approximation and reducing quantization error, respectively.

[0120] To allow the transmission of the discrete main latent representation y from the encoding side to the decoding side of the autoencoder 1, a second entropy encoder 4, in particular an arithmetic encoder, is configured to process the discrete main latent representation y to provide a second compressed latent space 5 in the form of a bitstream which is then fed to a second entropy decoder 6, in particular an arithmetic decoder, for providing a second decompressed latent space corresponding to the discrete main latent representation y-

[0121] With reference to Figure la, the main latent representation y is divided into slices 13,14 and input to the block 10 and each slide is encoded in order using those previously encoded.

[0122] The channel-wise autoregressive entropy model block 10 comprises slice encoders 15,16. In addition, the slice encoder blocks 15,16 exploit the output of the hyperprior encoder / decoder 17.

[0123] Residual vectors r are the outputs of blocks 18 called LRPi and LRP2, wherein LRP stands for Layer-wise Relevance Propagation. Towards this end, the discrete main latent representation y, or feature map, is divided into a predetermined number of slices, and then the spatial context of a specific slice is extracted by integrating information, via an adder block 12 (see Figure 1), from both the hyperprior and a channel context model that receives previously decoded slices as input.

[0124] In addition, also a residual vector r obtained during the quantization step is estimated, which is then added to the latent representation, always via the adder block 12, to reduce quantization error.

[0125] Again with reference to Figure 1, an output 11 of the channel-wise autoregressive entropy model block 10 is represented in terms of means / / and standard deviation o representing spatial correlation for each element of the discrete main latent representation y since the latter is modeled as a Gaussian distribution.

[0126] The standard deviation o of the output 11 is fed both to the second arithmetic encoder 4 and to the second arithmetic decoder 6, while the mean / / of the output 11 is used to translate the discrete main latent representation y so that it has zero mean when compressed.

[0127] Finally, the discrete main latent representation y is fed to an image reconstruction decoder fs(synthesis transform), thereby obtaining a reconstructed image = gs(^). According to the invention, the first quantizer 2 and the second quantizer 3 are implemented as a new parameterized activation function called STanH.

[0128] The first quantizer 2 and the second quantizer 3 are respectively implemented for quantizing the main latent representation y, or main latent space, and the hyperprior latent representation z, or hyperprior latent space.

[0129] Sum of Hyperbolic tangents for differential quantization

[0130] It is recalled that the present invention aims at designing a scalar quantization function that is differentiable and allows back-propagating the gradient of the error function at training time when plugged into an autoencoder 1 like that of Figure 1. Let y be a tensor representing a real-valued latent space, where C, H, and W represent the channels, the height, and the width dimension respectively.

[0131] In this context, the goal is exemplified as relaxing the discrete main latent representation y through a continuous proxy soft-quantized main latent representation y at training time.

[0132] Toward this goal, the desired quantization function must satisfy the following requirements: i) the slope of the quantization steps shall be controllable at training time; ii) the width of the quantization interval and the corresponding reconstruction level shall be parametric and learnable at training time.

[0133] Requisite i) is instrumental in making the quantizer arbitrarily close to an actual scalar quantizer, i.e. a ladder-like function.

[0134] Requisite ii) allows the model according to the present invention to advantageously adapt the quantization intervals during training, thus replacing the conventional technique of rounding to the integer. This aspect is fundamental for the method according to the present invention since it allows to adapt the same model to different rates, as it will experimentally be shown later.

[0135] In order to meet the two requirements, a parameterized activation function STanH according to the invention is illustrated in Figure 2.

[0136] In particular, Figure 2 represents a plot 30 of the STanH activation function according to the invention with L = 5 quantization levels and for increasing values of inverse temperature [3.

[0137] More in particular: curve 20 represents the STanH activation function with (3 = 1; curve 21 represents the STanH activation function with (3 = 2; curve 22 represents the STanH activation function with (3 = 5; curve 23 represents the STanH activation function with [3 = 10; curve 24 represents the STanH activation function with [3 = 20; curve 25 represents the STanH activation function with (3 -> +oo.

[0138] Let L be the number of desired quantization levels, then the STanH quantization function is defined as the summation of L - 1 translated and weighted hyperbolic tangents and is applied to the main latent representation y element-wise as follows:

[0139] About requisite i), i.e. the slope of the quantization steps shall be controllable at training time, [3, which is referred to as inverse temperature, regulates the slope of the quantization steps, i.e. the relaxation of the discrete main latent representation y: the higher [3, the closer the slope to that of the step-wise function. For this reason, the value of [3 is gradually increased during the training with an annealing procedure which is detailed later on.

[0140] About requisite ii), i.e. the width of the quantization interval and the corresponding reconstruction level shall be parametric and learnable at training time, the parameters w = (w1;w2, ... , wL-i) and b = (hp b2, . . . , bL-1) determine the reconstruction levels for the soft-quantized main latent representation y and width of the quantization intervals, respectively.

[0141] In fact, assuming / ? = oo in formula (2), the first reconstruction level is equal to where the subscript y < bi represents the application of STanH to a general value smaller than bi.

[0142] The i-th reconstruction level li is then obtained by adding to the previous level h-i as follows:

[0143] Consequently, at inference time, with formula (2) is replaced by the actual scalar quantizer with reconstruction levels as in formula (4) and quantization intervals determined by parameter b, obtaining thus the soft-quantized main latent representation y .

[0144] During training, with [3 < +co, transitions between reconstruction levels are exponentially smoothed as in Figure 2, that exemplifies STanH with L = 5: for [3 = 1, the shape of STanH is close to the hyperbolic tangent; as [3 increases, the shape progressively approaches the step-wise quantization function.

[0145] Since STanH has a derivative all over its domain, parameters w and b can be learned to minimize the rate distortion cost function when the STanH quantization function is plugged into the back-propagation procedure as described below.

[0146] It is now explained how the inverse temperature [3 is annealed during training to achieve a final configuration that is consistent with the (not differentiable) function that performs quantization directly at the quantization levels.

[0147] To achieve this goal, the procedure for annealing the inverse temperature [3 is crucial towards convergence: annealing too fast could lead the model to settle on a local minimum, whereas annealing too slowly could prevent being robust against quantization errors.

[0148] Following Sohl-Dickstein J. et al., "Deep unsupervised learning using nonequilibrium thermodynamics", International Conference on Machine Learning, 2015, reference

[0041] , the issue is tackled by incrementally increasing [3, relying on a function that considers both the number of training iterations and the difference between the relaxed latent space y and the quantized latent space y. In this way, it is possible to progressively approach the discrete, or quantized, latent configuration during training.

[0149] Taking inspiration from reference

[0029] , it is proposed a strategy where [3 is increased in a semi-deterministic way.

[0150] In a nutshell, the possible values of the latent representation are progressively driven towards the discrete quantization levels, yet without overly constraining the configuration of the latter during training.

[0151] Let f indicate the f-th step of the training procedure based on the back-propagation of the error gradient described in the following.

[0152] For a given batch of training samples, the quantization errors of the soft-quantized main latent representation y and of the discrete quantized latent representation y are computed as:

[0153] By taking the absolute difference between formula (5) and formula (6), it is obtained the error between the discrete and continuous representation at training step f: A A, . 7, y (7)

[0154] The smaller this error, the closer the latent space y to the actual discrete space to be encoded y. After computing Et, the values of the temperature are updated: where Pt represents the value used at the f-th step, and K is a factor that regulates the velocity with which the latent representation is frozen.

[0155] It is easy to understand that, as the training steps progress, the latent representation will become on average increasingly frozen towards the final true configuration, making the model robust to the quantization error.

[0156] End-to end learning with STanH

[0157] It is now shown how the differentiable quantizer according to the invention is implemented as a layer that can be plugged into a generic image compression model.

[0158] Next, it is formulated the rate-distortion cost function to be minimized at training time and it is detailed the end-to-end training procedure.

[0159] While the training procedure for the autoencoder 1 of Figure 1 is exemplified, it can be generalized to any autoencoder architecture.

[0160] The activation function STanEt can be implemented as a parametric layer that can be plugged at any arbitrary position in a neural model adapted to carry out learned image compression to implement quantization.

[0161] In the case of the architecture 1 of Figure 1, both the hyperprior latent representation z and the main latent representation y require to be quantized, therefore two independent instances of STanH layer are required.

[0162] Every instance operates independently and undergoes separate training to acquire its own distinct quantization functions.

[0163] In fact, z and y have different semantic meanings: the former represents the hyperprior latent representation, while the latter represents the main latent representation from which the image is reconstructed. Therefore, it is likely that the latent representations z and y have distinct distributions with distinct optimal quantization levels. Having two separate STanH layers advantageously involves only a few hundred additional parameters to train, which is a negligible figure compared to training the entire model.

[0164] When plugging STanH, the following latent representations are obtained: and

[0165] M ™ SlAmW (M DO) where f indicates the f-th training step, while the subscripts {h,m} refer to the two different latent representation, namely the hyperprior and the main one, respectively.

[0166] It is highlighted that since STanH / ;and STanHmare implemented as two independent layers, Ph,t and pm,t are also modified through annealing independently one from the other.

[0167] Plugging the STanH layer in an image compression model preserves the nature of the standard rate-distortion (RD) optimization problem wherein the first term is some distortion metric d, the second term represents the latent space rate of the hyperprior which is calculated by means of a small neural network trained to calculate said rate, and the third term represents the rate of the main latent space from which the image is reconstructed and is calculated using the Gaussian probability mass function since y is distributed as a Gaussian latent space. The hyper-parameter A is the Lagrangian multiplier that controls the trade-off between rate and distortion. A bigger A puts more penalty on large distortions, whereas a smaller A puts more penalty on the rate of the latent representations z and y.

[0168] In particular, for the second term it is followed the standard approach proposed in reference [8] to use an ad-hoc neural network to directly estimate the rate: in this way, following the same configuration as prior art works, it is imposed a fully factorized distribution among channels as follows: where i j represents the learnable parameters.

[0169] The third term represents the rate of a Gaussian-like latent space, therefore there is no need for a further neural network to determine the rate. In fact, for a specific yt, it is possible to evaluate its cumulative distribution at training time as: where is the cumulative distribution function of the Gaussian distribution with mean and standard deviation oi, while r~ and r(+are the left and the right bounds of the quantization intervals related to yt, respectively.

[0170] Notice that, differently from the prior art that rely on uniform quantization over integers, where r~ and r are both equal to 0.5, in the method according to the present invention these two values change depending on the activation function STanH.

[0171] 3) Learning procedure: it is recalled that one of the goals of the present invention is achieving variable rate image coding without training a separate model for each target rate as in references

[0015] ,

[0017] and

[0019] .

[0172] Let us assume that at least one model has been preliminary trained for some target rate that is called anchor model.

[0173] Since in the method according to the present invention, the quantizer is implemented as a learnable layer, different rate-distortion tradeoffs can be achieved by plugging a different STanH layer in the anchor and refining the STanH parameters for different X values.

[0174] Practically speaking, the STanH layer is plugged in the anchor, all the anchor layers are frozen but STanH and this latter layer is refined for a few epochs.

[0175] The procedure is repeated for each different target rate refining a separate STanH layer for a different <\ value for each target rate. In detail, the STanH layer is refined for increasingly lower X values in formula

[0011] (i.e., the target rate is gradually reduced).

[0176] The result of this procedure is a set of refined models that share the same learnable parameters as the anchor except for the STanH layer: these models are called derivations.

[0177] Therefore, only one anchor model and one STanH layer from each derivation need to be stored, once the refinement process is over. Once the refinement procedure above is over, it is possible to switch encoding rates simply by plugging into the anchor model the STanH layer from the correct derivative. Notice that in principle, it is possible to consider more anchors trained for different rates, and in this case, it is desirable that the anchors are evenly spaced across the target rate range. In the following, different numbers of anchors and target rates with different LIC models are experimented.

[0178] From fine-grained to continuous rate adaptation In practical image coding applications, it is of paramount importance to control the rate at a fine granularity, e.g. by tuning the QP in standardized codecs (see reference [2]).

[0179] The activation function STanH allows rate adaptation in two ways. The first, straightforward, way is refining a new STanH layer for the appropriate <\, as described above, achieving fine-grained rate control. When finetuning a novel derivation is not possible (e.g., lack of training resources or samples), it is possible to strike continuous rate control by interpolating two existing derivations as follows.

[0180] Let us suppose there are two derivations with corresponding layers STanHi and STanH2, achieving two different RD point (r2, q2) and (n, qi), with qi > q2 and n > r2. The goal here is to produce a third derivation STanHi at the RD point (ri, q i) with ri > ri > r2 and qi > q i > q2.

[0181] Let (wi, bi) and (wi, bi) be the learnable parameters related to STanHi and STanH2, respectively. Instead of refining the new STanHi, a new set of parameters (ws, bs) can be interpolated as follows; from now on these type of model are called interpolations. where p E (0, 1) is tuned to match the desired target rate n > ri > r2 (with correspondent qualities qi > qi > q2).

[0182] The results of the experiments carried out by the Applicants are now presented.

[0183] The activation function STanH is experimented over three recent learned image compression architectures. Namely, all experiments rely on the codebase of the CompressAI project (see Begaint Jean, Racape Fabien, Feltman Simon and Pushparaja Akshay, "CompressAI: a PyTorch library and evaluation platform for end-to-end compression research", arXiv preprint arXiv:2011.03029, 2020, [reference 43]), for the codebase of the architectures Zou22, see reference

[0017] , Xie21 (see reference

[0019] ) and Cheng20 (see reference

[0015] ). The impact of the activation function STanH is measured both in terms of RD performance and complexity terms on the above models, showing that the activation function STanH enables variable rate image coding with robust savings in terms of training and deployment costs.

[0184] While in a practical scenario one would simply refine a pre-trained anchor following the procedure above, in these experiments the anchors are retrained from scratch with the STanH module for two reasons; first, it is wanted to measure the cost of training from scratch and to set an accurate baseline model for benchmarking RD efficiency. Second, it is also wanted to evaluate STanH module in a fixed rate context scenario in which only anchor models without derivations were considered.

[0185] Now it is described in detail how the three above architectures are trained with the activation function STanH as a quantizer.

[0186] From now on, it is referred to z-th anchor as Ai, where Ai represents the point with the highest rate and quality in the RD plane (i.e., the top-right point on the curve).

[0187] It is referred to the / -th derivation from the z-th anchor as Dij, where a larger value of / indicates a greater deviation on the RD plot from the reference anchor (i.e., towards the bottom-left corner of the RD plot).

[0188] Coherently with the existing literature, the MSE (Mean Squared Error) as distortion metric d in equation (11) is optimized, and six different X values, i.e. points, on the RD plot are considered.

[0189] At inference time, arithmetic coding to encode latent representations is exploited, using the torchac library (see Mentzer F. et al., "Practical Full Resolution Learned Lossless Image Compression", IEEE Conference on Computer Vision and Pattern Recognition, 2019, [reference 44]).

[0190] For the architectures Cheng20 and Xie21, L=60 quantization levels are fixed for both the latent representations, and w and b are initialized in order to have an initial uniform quantization in the range [-30, 30] for a total of 240 parameters.

[0191] For Zou22, it is increased L to 120 for the main latent representation y (Gaussian- distributed), and L was reduced to 40 for the hyperprior latent representation; in this case, the number of parameters included in the two STanH modules increased to 320.

[0192] Factor K is empirically fixed to K = 15, which regulates the annealing velocity of the inverse temperature doubling it when the cost function reaches a plateau for the first time, with a patience of 50 epochs.

[0193] 24K random samples are used from the Openimages dataset (see Kuznetsova, A. et al., "The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection", IJCV, 2020, dataset available from https: / / github.com / openimages, [reference 45]), for training each anchor and, depending on the architecture, the model was trained for ~ 1-1.5M steps with a batch size of 16 images, using the Adam optimizer (see Kingma Diederik P. et al.," Adam: A method for stochastic optimization", Proceedings of the International Conference on Learning Representations, 2015, [reference 46]), with an initial learning rate of 10-4that is reduced by a 2 factor when a plateau is reached with 50 epochs patience.

[0194] The number of anchors to train is a hyper-parameter that drives a trade-off between RD performance and training costs in terms of time and storage: this trade-off will be later explored in detail.

[0195] Figure 3 lists different values for X used for training anchors for each reference model, considering the case of three anchors per model (actual values are from the reference papers). All models are trained on an NVIDIA A40 GPU.

[0196] Finally, the STanH layers are refined to target different rates. As each STanH layer is only a few hundred learnable parameters, it was found that only about 8000 samples from the training dataset are enough to refine the layer. Each derivation is refined for 2-3 K steps and the patience for a learning rate reduction is reduced from 50 to 10 epochs.

[0197] In Figure 4 for each architecture the values of <\ used for refining the derivations are listed, considering the case of three anchors (one derivation for each of them). It was experimentally observed that it is possible to refine a derivation starting from either a higher quality anchor or from the nearest anchor and moving in both directions with respect to the target bitrate, the latter approach yielding somewhat better RD efficiency.

[0198] All the above-trained models are evaluated on Kodak PhotoCD image dataset (see Eastman Kodak Company, Kodak Lossless True Color Image Suite, 1999, ics / kodak / , [reference 47]), the Clic Professional validation and test dataset (see Toderici G. et al., "Workshop and challenge on learned image compression", CVPR, 2021, [reference 48]), and the Tecnik dataset (see Asuni N. et al., "TESTIMAGES: a Large-scale Archive for Testing Visual Devices and Basic Image Processing Algorithms", Proceedings of the Conference on Smart Tools and Applications in Computer Graphics, 2014, [reference 49]).

[0199] The Kodak dataset comprises 24 uncompressed images with a resolution of 768x512, Clic dataset consists of 60 images of varying and higher resolutions, while Tecnik includes 100 images with a resolution of 1200x1200. The image quality (i.e., distortion) is evaluated as peak signal-to-noise ratio (PSNR) and secondarily as multiscale structural similarity (MS-SSIM), (see Wang Z. et al., "Multiscale structural similarity for image quality assessment", The Thirty-Seventh Asilomar Conference on Signals, Systems and Computers, 2003, Vol. 2., [reference 50]).

[0200] The rate of the compressed latent representations is measured in terms of bits per pixel (bpp) to account for the different image resolutions. Such metrics are plotted as rate-distortion curves and pairs of curves are compared in terms of Bjontegard metrics (see Bjontegaard, Gisle, "Calculation of average PSNR differences between RD-curves", VCEG-M33, 2001, [reference 51]).

[0201] It is recalled that a negative BD-Rate (fewer bpps required for the same PSNR) or / and positive BD-PSNR (higher PSNR for the same bpps) indicate better encoding efficiency.

[0202] Experimenting with the number of anchors

[0203] As a first experiment, it is explored the performance-complexity tradeoff as a function of the number of anchors and derivations.

[0204] It was experimented with decrementing the number of anchors from six (all models trained end-to-end, in a fixed-rate approach) to one (only one model trained end- to-end, other 5 are derived refining only the STanH layers). For the time being, it was measured the complexity as the number of learnable parameters, since both overall training time and storage cost directly depend on that. Zou22 architecture was taken as a reference, where the model is trained with different lambdas and without STanH layer.

[0205] Figure 5 shows the Rate-Distortion (RD) performance of the architecture of Zou22 on Kodak dataset as a function of different number of anchors, from six (a) to one (f), wherein crosses represent anchors and circles represent refined derivation(s).

[0206] Figure 6 shows the corresponding BD-Rate and BD-PSNR vs. the architecture Zou22 on the Kodak test set for different numbers of anchors, wherein savings are reported in terms of Trainable Parameters (TP) for the method according to the present invention.

[0207] The case with six anchors is reported as a sanity check to show that STanH entails no RD penalty with respect to the reference fixed-rate quantization.

[0208] As the anchor number decreases, the RD performance worsens slightly, yet the complexity drops much faster, as shown in Figure 6. For three anchors, the BD- Rate drop is within 1%, while the complexity is slashed by a factor of two.

[0209] Finally, Figure 5(f) reports the corner case of one anchor only and five derivatives for the purpose of stressing the potential of the method according to the present invention. As the derivations move away from the anchor, the RD performance of the derivations degrades proportionally to such a distance, yet in a graceful manner. Since the three anchors setup of Figure 5(d) enables a reasonable performancecomplexity trade-off (complexity reduced by ~ 50% with a BD-Rate penalty below 1%), it is referred to this setup in the following.

[0210] Rate-distortion performance

[0211] The RD analysis of the experiments made by the Applicants is now extended to the Xie21 and Cheng20 architectures while keeping the number of anchors equal to three.

[0212] Figure 7 show Rate-PSNR plots for the proposed STanH-based method according to the invention and relative reference for Kodak (top row), Clic (central row), and Tecnik (bottom row) datasets and for three anchors and three derivations. As an additional reference, a curve for the H.266 / WC reference encoder VTM-20.2, see reference [2], was added.

[0213] The method according to the present invention exhibits curves that overlap almost entirely with the reference curves, i.e. it does not affect the compression efficiency despite a 50% reduction in complexity. Similar results can be observed also in Figure 8 where image quality is measured in MS-SSIM terms (MS-SSIM is converted to -10 logio(l - MS-SSIM)) for the Kodak dataset.

[0214] Figure 9 compares the three-anchors model proposed according to the invention with references in terms of BD-Rate and BD-PSNR on the Kodak dataset, wherein memory savings are reported in terms of trainable parameters. For both Zou22 and Cheng20 architectures the BD-rate loss is below 1%, while for Xie21 it remains well below 2% despite a reduction in complexity below 50%, as it is discussed in detail later on.

[0215] Training cost

[0216] A largely overlooked aspect of learned image compression is the cost of training the models from scratch, where a single training can take up to 10 days, see reference

[0043] , for a stable solution.

[0217] Conversely, with the method according to the present invention, when refining an anchor into a derivation, only the parameters in the quantization layer(s) need to be updated, for less than 0.001% of the total model complexity in most cases.

[0218] The cost of training the models in terms of parameters to train or refine (TP), training time (TT), and energy consumption (TC) is now quantified.

[0219] For the reference models, the numbers in the original papers are reported; otherwise, the models to produce the required numbers are trained by the Applicants themselves.

[0220] For the method according to the present invention, the usual scheme is considered where three anchors are trained from scratch and three derivations are refined.

[0221] Figure 10 shows a table related to the approximate costs in terms of Trainable parameters (TP), Training Time (TT), Training Cost (TC), and Storage (SG). " Reference-total" and "Proposed-total" refer to the cost for training the six models covering the entire BD range considered. In parentheses it is reported the percentage gain of that particular field compared to the reference.

[0222] Figure 10 shows that the cost of refining a derivation is just one-tenth of training the reference model. In fact, refining an anchor amounts to training 240-360 M (depending on the model) parameters for each STanH layer only, rather than a deep convolutional model. As a result, the activation function STanH saves from 33% (with respect to the architecture Cheng20) to 45% (with respect to the architecture Zou22) of the energy required and 48% of the training time for architecture Xie21 for training three anchors and refining three derivations.

[0223] Further savings can be of course achieved by replacing further anchors with derivations at the price of somewhat lower RD performance (see Figure 5).

[0224] Figure 10 also shows that training an anchor with the activation function STanH requires slightly more time and energy than training the reference model.

[0225] It was observed that annealing the inverse temperature p to a stable configuration increases the training time.

[0226] Additionally, the bounds of the integral according to formula (13) vary at each iteration depending on the parameters of the activation function STanH, and calculating these parameters can lengthen the training. The method according to the invention enables consistent gains considering the total energy consumption and training time.

[0227] It is specified that these calculations are based on the training information given by the original papers, using the Applicants' computational resources to calculate the average power consumption and multiplying it by the hours required for the network training: the reported energy value is therefore an estimation of the real value; however, it shall allow appreciating a drastic improvement when refining the STanH layer only.

[0228] Storage cost

[0229] Another overlooked aspect of learned image compression is the requirements for storing the trained models on user devices, especially resource-constrained devices such as mobiles, settop-boxes, Systems-on-a-Chip (SoCs), etc., where storage is limited by design.

[0230] Figure 10 presents (fourth column "SC" for each architecture) the storage requirements for the pickle format, and it is recalled that the footprint of the models varies depending on the model and the size of the latent spaces.

[0231] The reference schemes need to store six models, whereas the method according to the present invention needs to store three (or fewer) anchors.

[0232] With the method according to the present invention, storing the derivations amounts to storing only the few hundred parameters of the refined STanH layers.

[0233] The storage cost reduction is about 50%: for architectures Cheng20 and Zou22 it is reported 49% and 53% savings, respectively.

[0234] The most significant improvement is observed with the architecture Xie21, where the footprint is reduced by ~ 57% . This is because the two lowest bitrate anchors are smaller in this case (with N = 128).

[0235] It is hypothesized that such numbers could be further reduced if the models were saved in some compressed format.

[0236] Comparison with variable rate models

[0237] The method according to the present invention is compared with Gain [reference 36], EVC [reference 40] and SCR [reference 37], i.e. three state-of-the-art VBR codecs previously introduced.

[0238] About EVC and SCR, numbers from the original papers are taken as reference, as both rely on ad-hoc architectures and training procedures. Regarding Gain, it achieves adaptive quantization plugging into a model, like in the present invention, yet it relays on an ad-hoc entropy model distribution of the latent representation. For a fair comparison, the Applicants themselves implemented gains units over the same Zou22 model which was taken as a reference for the method according to the present invention.

[0239] Training is performed according to the process described in the original Gain paper, using six different qualities. Furthermore, only one anchor was used (see A2 in Figure 3) to cover the entire range, since Gain is composed by only one model. Figure 11 represents a RD-plot on Kodak for the method according to the present invention versus Gain (reference

[0036] ), over the architecture Zou22, EVC (reference

[0040] ) and SCR (reference

[0037] ).

[0240] Figure 11 shows that the method according to the present invention outperforms the three references in proximity of the anchor A2 (0.25-0.50 bpp range).

[0241] Figure 12 represents BD-Rate, BD-PSNR, number of parameters for variable bitrate (#Pars for VBR), and storage cost (SC) in GB of the method according to the present invention with one anchor vs. Gain vs. EVC over a H.266 / VVC reference.

[0242] When moving away from the anchor, the RD efficiency of the method according to the present invention degrades gracefully, and Figure 12 shows that the method according to the present invention is still the best performer (minimal RD efficiency loss) over a H.266 / WC reference software VTM, see reference [2],

[0243] In terms of complexity, the method according to the present invention and Gain have similar training costs, as only one anchor needs to be trained from scratch. Also the storage costs are comparable since both methods need to store only one anchor plus the few thousands extra parameters that enable VBR coding. Regarding EVC, it introduced a dual prior encoder that extrapolate a point- wise gain unit in order to obtain bitstreams at different quality. Because of this, it necessitates more parameters to obtain VBR (V ariable Bit Rate), moreover this method is not agnostic with respect to model architecture (e.g., it is not possible to use as it is on channelwise model like Zou22).

[0244] Continuous rate adaptation

[0245] It is now evaluated the ability of the method according to the present invention to achieve both fine-grained and continuous rate control using the interpolation strategy previously described.

[0246] From the three initial anchors, a total of 13 derivations (4160 additional parameters) were tuned, which have been exploited to interpolate about 50 extra RD points, i.e. interpolations, to achieve continuous rate adaptation.

[0247] Figure 13 shows the resulting RD curves for the Zou22 architecture on the Kodak dataset, where stars represent the anchors, the circles the derivations and the empty boxes are the interpolations. This suggests that interpolating STanH layers by sweeping p in formula (14) affect minimally the RD performance, allowing to achieve continuous variable rate.

[0248] Figure 14 condenses in a single radar plot for activation function STanH and the reference schemes seven different metrics, namely average PSNR (Avg-PSNR) and average bitrate (Avg-Rate), storage cost in GB (SC), number of trainable parameters (TP), training cost in kWh (TC), training time in hours (TT), and rate granularity (RG), defined as the average rate distance between adjacent RD points. Apart from Avg-PSNR, for all these metrics the lower, the better, i.e. a narrower radar profile corresponds to better performance. The plot confirms that the activation function STanH enables almost identical RD performance yet for lower training and storage costs, improving all the considered aspects with respect to Zou22, including the benefit of continuous rate granularity.

[0249] Figure 15 represents a second autoencoder 1' as a possible variant of the autoencoder of Figure 1.

[0250] The autoencoder 1' is identical to the autoencoder 1 except for the hyperprior branch which is missing. Therefore the autoencoder 1' comprises an image reconstruction encoder fathat projects an input image x onto a main latent representation y of dimension M, representing the number of channels, which is then quantized by the quantizer 2, thereby obtaining a discrete main latent representation y .

[0251] To allow the transmission of the discrete main latent representation y from the encoding side to the decoding side of the autoencoder 1, an entropy encoder 4, in particular an arithmetic encoder, is configured to process the discrete main latent representation y to provide a second compressed latent space 5 in the form of a bitstream which is then fed to an entropy decoder 6, in particular an arithmetic decoder, for providing a decompressed latent space corresponding to the discrete main latent representation y .

[0252] Finally, the discrete main latent representation y is fed to an image reconstruction decoders, thereby obtaining a reconstructed image = gs( ).

[0253] Although the autoencoder is structurally simpler than the autoencoder 1, the quantizer 2 is implemented as the already described parameterization activation function STanH and presents the advantages previously explained.

[0254] To sum up, it is proposed a novel method to convert fix-rate LIC models to variable rates by exploiting STanH, a parametric module that approximates quantization.

[0255] By definition, STanH converges to the stepwise quantizer if its inverse temperature P is properly annealed at training time. Variable rate is achieved by training only a few anchor models end-to-end, and then refining the STanH layers only for other RD tradeoffs into different derivation models.

[0256] Once the anchors have been trained, refining additional derivations has negligible training and storage costs, practically enabling both fine-grained and continuous rate control, by computing weighted average of already existing STanH 's.

[0257] In summary, it has been shown that the method according to the present invention achieves comparable results with respect to both fix-rate and variable-rate LIC models; moreover, thanks to its simplicity it is totally agnostic to the reference architecture.

[0258] The present description has tackled some of the possible variants, but it will be apparent to the man skilled in the art that other embodiments may also be implemented, wherein some elements may be replaced with other technically equivalent elements. The present invention is not therefore limited to the explanatory examples described herein, but may be subject to many modifications, improvements or replacements of equivalents parts and elements without departing from the basic inventive idea, as set out in the following claims.

Claims

CLAIMS1. Method for learned image compression implemented in an autoencoder (1') comprising an image reconstruction encoder (fa) and an image reconstruction decoder (fs), said method comprising the steps of: a) extracting from an image (x) a main latent representation (y) by means of said image reconstruction encoder (fa); b) quantizing said main latent representation (y) by means of a quantizer (2) to obtain a discrete main latent representation (y); c) entropy coding said discrete main latent representation (y) by means of an entropy encoder (4) to obtain a bitstream (5); d) entropy decoding said bitstream (5) by means of an entropy decoder (6) to obtain an entropy decoded bitstream; e) feeding said entropy decoded bitstream to said image reconstruction decoder (fs); f) recovering a reconstructed image (x) by means of said image reconstruction decoder (fs); g) training said autoencoder (3) via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing said discrete main latent representation (y) by means of a soft- quantized main latent representation (y), said method being characterized in that said quantizer (2) is implemented as a parameterized activation functionwherein L is a number of sought quantization levels, [3 is an inverse temperature parameter regulating a slope of said quantization levels, and w and b are parameters respectively determining reconstruction levels and width of quantization intervals of said soft-quantized main latent representation (y).

2. Method according to claim 1, wherein said parameter w for determining said reconstruction levels and said parameter b for determining said width of said quantization intervals are learned to minimize said rate distortion cost function L.

3. Method according to claim 2, wherein said rate distortion cost function L is in the form J = R + A ■ D where A is a hyper-parameter that regulates the trade-off between bitrate R of main latent representation (y) and distortion D of said reconstructed image (x).

4. Method according to any of the claims 1 to 3, wherein said inverse temperature parameter [3 is gradually increased during training with an annealing procedure.

5. Method according to claim 4, wherein said annealing procedure relies on a function that considers both the number of training iterations and the difference between said soft-quantized main latent representation (y ) and said discrete main representation (y).

6. Method according to any of the claims 1 to 5, wherein said inverse temperature parameter [3 is computed as:wherein t is t-th step of the training, wherein™ 7 A ■■’■ -V are respectively quantization errors of the soft-quantized main latent representation (y) and of the discrete quantized latent representation (y), whereinf ; ;is the error between said quantization errors at training step t, wherein [3trepresents the value used at the t-th step, and wherein K is a factor that regulates the velocity with which said soft-quantized main latent representation (y) is frozen.

7. Method for learned image compression implemented in an autoencoder (1) comprising an image reconstruction encoder (fa) and an image reconstruction decoder (fs), said method comprising the steps of: a) extracting from an image (x) a main latent representation (y) by means of said image reconstruction encoder (fa) and a hyperprior latent representation (z) by means of a hyperprior analysis transform block (ha); b) quantizing said main latent representation (y) by means of a first quantizer (2) to obtain a discrete main latent representation (y) and said hyperprior latentrepresentation (z) by means of a second quantizer (3) to obtain a discrete hyperprior latent representation (z); c) entropy coding said discrete main latent representation (y) by means of a first entropy encoder (4) to obtain a first bitstream (5) and said discrete hyperprior latent representation (z) by means of a second entropy encoder (7) to obtain a second bitstream (8); d) entropy decoding said first bitstream (5) by means of a first entropy decoder (6) to obtain a first entropy decoded bitstream and said second bitstream (8) by means of a second entropy decoder (9) to obtain a second entropy decoded bitstream; e) feeding said first (5) and second (8) entropy decoded bitstreams to said image reconstruction decoder (fs); f) recovering a reconstructed image (x) by means of said image reconstruction decoder (fs); g) training said autoencoder (3) via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing said discrete main latent representation (y) by means of a soft- quantized main latent representation (y ) and said discrete hyperprior latent representation (z) by means of a soft-quantized hyperprior latent representation (z), said method being characterized in that said quantizers (2,3) are implemented as respective parameterized activation functions whereinwherein f indicates the f-th training step, L is a number of sought quantization levels, |3 is an inverse temperature parameter regulating a slope of said quantization levels, w and b are parameters respectively determining reconstruction levels and width of quantization intervals of said soft-quantized main latent representations (y, z), and the subscripts { / i, / i / } refer to the two different latent representations, namely the hyperprior and the main one, respectively.

8. Method according to claim 7, wherein said parameter w for determining said reconstruction levels and said parameter b for determining said width of said quantization intervals are learned to minimize said rate distortion cost function L.

9. Method according to claim 8, wherein said rate distortion cost function L is wherein d is a distortion metric, X is a hyper-parameter that regulates the trade-off between bitrate R of said main latent representation (y) and distortion D of said reconstructed image (x),represents a rate of said hyperprior latent representation (z), andrepresents a rate of said main latent representation (y) from which the image is reconstructed.

10. Method according to claim 8 or 9, wherein said inverse temperature parameters Ph,t and pm,t are gradually increased during training with an annealing procedure.

11. Method according to claim 10, wherein said annealing procedure relies on a function that considers both the number of training iterations and a difference between said soft-quantized main latent representation (y) and said discrete main representation (y) and a difference between said soft-quantized hyperprior latent representation (z) and said discrete hyperprior latent representation (z).

12. Method according to claim 10 or 11, wherein said inverse temperature parameter pm,t is computed as:wherein t is t-th step of the training, wherein> are respectively quantization errors of the soft-quantized main latent representation (y) and of the discrete quantized latent representation (y), whereinfis the error between said quantization errors at training step t, and wherein K is a factor that regulates the velocity with which said self-quantized latent representation is frozen;and said inverse temperature parameter h,t is computed as:wherein t is t-th step of the training, wherein et= ||z — z||2and et= ||z — z||2are respectively quantization errors of the soft-quantized hyperprior latent representation (z) and of said discrete quantized hyperprior representation (z), Et= |et— et] is the error between said quantization errors at training step t, and wherein K is a factor that regulates the velocity with which said soft-quantized hyperprior latent representation (z) is frozen.

13. Autoencoder (1') for learned image compression comprising:- an image reconstruction encoder (fa) configured to extract from an image (x) a main latent representation (y);- a quantizer (2) configured to quantize said main latent representation (y) to obtain a discrete main latent representation (y);- an entropy encoder (4) configured to entropy coding said discrete main latent representation (y) to obtain a bitstream (5);- an entropy decoder (6) configured to entropy decoding said bitstream (5) to obtain an entropy decoded bitstream;- an image reconstruction decoder (fs) configured to receive said entropy decoded bitstream;- an image reconstruction decoder (fs) configured to recover a reconstructed image (x);- means for training said autoencoder (1') via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing said discrete main latent representation (y) by means of a soft- quantized main latent representation (y), said autoencoder (1') being characterized in that said quantizer (2) is implemented as a parameterized activation functionwherein L is a number of sought quantization levels, [3 is an inverse temperature parameter regulating a slope of said quantization levels, and w and b are parameters respectively determining reconstruction levels and width of quantization intervals of said soft-quantized main latent representation (y).

14. An autoencoder (1) for learned image compression comprising:- an image reconstruction encoder (fa) configured to extract from an image (x) a main latent representation (y) and a hyperprior analysis transform block (ha) configured to extract from said image (x) a hyperprior latent representation (z);- a first quantizer (2) configured to quantize said main latent representation (y) to obtain a discrete main latent representation (y) and a second quantizer (3) configured to quantize said hyperprior latent representation (z) to obtain a discrete hyperprior latent representation (z);- a first entropy encoder (4) configured to entropy coding said discrete main latent representation (y) to obtain a first bitstream (5) and a second entropy encoder (7) configured to entropy coding said discrete hyperprior latent representation (z) to obtain a second bitstream (8);- a first entropy decoder (6) configured to entropy decoding said first bitstream (5) and a second entropy decoder (9) configured to entropy decoding said second bitstream (8) to obtain a second entropy decoded bitstream;- an image reconstruction decoder (fs) configured to receive said first (5) and second (8) entropy decoded bitstreams;- an image reconstruction decoder (fs) configured to recover a reconstructed image (x);- means for training said autoencoder (1) via standard gradient descent of the backpropagated error gradient by minimizing a rate distortion cost function L through relaxing said discrete main latent representation (y) by means of a soft- quantized main latent representation (y ) and said discrete hyperprior latent representation (z) by means of a soft-quantized hyperprior latent representation (z),said autoencoder (1) being characterized in that said first (2) and second (3) quantizers are implemented as respective parameterized activation functionswherein f indicates the f-th training step, L is a number of sought quantization levels, |3 is an inverse temperature parameter regulating a slope of said quantization levels, w and b are parameters respectively determining reconstruction levels and width of quantization intervals of said soft-quantized main latent representations (y, z), and the subscripts { / i, / i / } refer to the two different latent representations, namely the hyperprior and the main one, respectively.

15. A method for variable rate image compression implemented in a device, the method comprising:- training one or more anchor models from scratch by selecting appropriate X values for placement across a target rate-distortion (RD) curve;- generating one or more derivations from each of the trained anchor models by cloning the trained anchor model and setting a lower X to target a lower rate;- resuming the training procedure for each derivation, wherein only the quantization layers related to the parameterized activation function according to claims 1 and 7 are refined, and all other parameters of the derivation are kept unchanged from the reference anchor model;- storing only the refined quantization layers after the refinement process is completed, wherein all layers of the derivation, except for the refined quantization layers, are identical to the anchor model; and- enabling variable rate image compression in said device by replacing a current quantization layer with that of a refined anchor to switch rates without the necessity of training a separate model for each rate.

Images