Method for enhanced image encoding and decoding using a graph-based attention block, and related encoder and decoder

Abstract

It is described a method for image encoding using a graph-based attention module (G-WAM) comprising a graph-based window block (112) configured to process an input feature map by applying graph-based attention mechanisms within a localized window of said input feature map, wherein said method comprises the steps of: - providing an output of said graph-based window block (112) to a first sequence (114) of residual blocks (RB), said first sequence (114) of residual blocks (RB) being configured to apply a first series of transformations to refine feature representations of said image; - providing an output of said first sequence (114) of residual blocks (RB) to a 1x1 convolutional layer (118), said 1x1 convolutional layer (118) being configured to perform a linear transformation for each feature independently; - passing an output of said 1x1 convolutional layer through an activation function (120); - element-wise multiplying, in a multiplication block (122), an output of said activation function with an output of a second sequence (116) of residual blocks (RB), said second sequence (116) of residual blocks (RB) taking as input said input feature map and being configured to apply a second series of transformations to further refine said feature representations of said image; adding an output of said multiplication block (122) to said input feature map by means of an element-wise addition block (124).

Description

[0001] -SVT078-PC00-

[0002] METHOD FOR ENHANCED IMAGE ENCODING AND DECODING USING A GRAPH-BASED ATTENTION BLOCK, AND RELATED ENCODER AND DECODER

[0003] DESCRIPTION

[0004] The present invention relates to a method for enhanced image encoding and decoding using a graph-based attention block.

[0005] The present invention also relates to an encoder and to a decoder implementing the method for enhanced image encoding and decoding using a graph-based attention block.

[0006] Standardized image codecs like JPEG (see Gregory K Wallace, "The JPEG still picture compression standard", CACM, 1991, reference [1]), the HEVC-based BPG (Bellard Fabrice, "BPG image format," 2014, reference [2], and WC (Benjamin Bross et al., "Overview of the versatile video coding (WC) standard and its applications", IEEE TCSVT, 2021, reference [3]) rely on a clever combination of coding tools such as transform coding, quantization, and entropy coding to achieve state-of-the-art in terms of Rate-Distortion (RD) efficiency.

[0007] Recently, Learned Image Compression (LIC) methods based on deep neural architectures such as Variational Autoencoders (VAEs) have gained traction in reason of their encoding efficiency.

[0008] After the seminal work by Johannes Balle et al., "End-to-end optimized image compression", ICLR, 2017, reference [4], most LIC architectures have predominantly used Convolutional Neural Networks (CNNs) (see Johannes Balle et al., "Variational image compression with a scale hyperprior", ICLR, 2018, reference [5]; David Minnen, Johannes Balle and George D Toderici, "Joint autoregressive and hierarchical priors for learned image compression", NeurlPS, 2018, reference [6]; Zhengxue Cheng et al., "Learned image compression with discretized gaussian mixture likelihoods and attention modules", CVPR, 2020, reference [7]; Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang, "The devil is in the details: Window-based attention for image compression", CVPR, 2022, reference [8]). The architectures of references [5], [6], [7] and [8] are improved -SVT078-PC00- with Generalized Divisive Normalization (GDN) layers (see Johannes Balle, Valero Laparra, and Eero P Simoncelli, "Density modeling of images using a generalized normalization transformation", ICLR, 2016, reference [9]) within a VAE framework.

[0009] Recently, attention mechanisms from Vision Transformers (ViTs) have been incorporated into LIC models (see reference [7], reference [8], and Jinming Liu, Heming Sun, and Jiro Katto, "Learned image compression with mixed transformer-CNN architectures", CVPR, 2023, reference

[0010] ) showing improved compression efficiency (see reference [8]).

[0010] However, the windowing-based self-attention mechanism used, e.g. in Swin Transformers (see Ze Liu et al., "Swin transformer: Hierarchical vision transformer using shifted windows", ICCV, 2021, reference

[0011] ), may result in redundant features that add to the rate of the compressed latent representation. It is pointed out that the self-attention mechanism in ViTs can be seen as a graph attention network (see Petar Velickovic et al., "Graph attention networks", ICLR, 2018, reference

[0012] ) operating within a locally fully connected graph.

[0011] Understanding how to efficiently constrain the attention mechanism is a challenging problem that, to the best of the Applicants' knowledge, has not been tackled yet.

[0012] Learned Image Compression

[0013] In LIC, an image is compressed employing a parametric autoencoder, trained end-to-end using backpropagation (see Siwei Ma et al., "Image and video compression with neural networks: A review," IEEE TCSVT, 2019, reference

[0013] ).

[0014] In particular, an encoder fa(-) projects an image x to a latent representation y, that is first quantized into y and then entropy-coded as a compressed bitstream. On the receiving side, the latter is first retrieved and then projected back to its original domain by a decoders(-) / which outputs the reconstruction of the image x. The entire framework is trained by optimizing the well-known Rate-Distortion loss function: -SVT078-PC00- where <\ is the hyper-parameter that controls the trade-off between the rate R(-) and the distortion D(-), that can be any distance metric between input x and reconstruction x (like a mean squared error), and R(-), representing the estimated rate of y. The loss in formula (1) accounts for both quality of the reconstruction x and compressibility of the latent representation y.

[0015] The first studies like those of reference [4] exploited a simple structure with only one latent representation, modeled as a fully factorized distribution extracted either with an auxiliary neural network or in an analytical way (see Alberto Presta et al., "A differentiable entropy model for learned image compression", ICIAP, 2023, reference

[0014] ). More recent architectures rely on a further representation z, called hyperprior (see reference [5]), extracted from a second encoder-decoder pair { / / ,;(-), fe(-)} to capture spatial correlations from y. In this scenario, y and z are modeled with a Gaussian and channel-wise fully-factorized distribution, respectively.

[0016] Various works have been developed upon this architecture, to increase the overall performance. For example, Minnen et al., see reference [6], introduced an autoregressive model based on mask convolution to extract more context information. Minnen and Singh, "Channel-wise autoregressive entropy models for learned image compression", ICIP, 2020, reference

[0015] , developed a channel-wise entropy estimation model that first divides the latent representation into slices and then encodes them sequentially. Other works tried to exploit other techniques, e.g., Yueqi Xie et al., "Enhanced invertible encoding for learned image compression", ACMMM, 2021, reference

[0016] , introduced invertible modules to preserve information through layers, while Yang et al., "Graph-convolution network for image compression", ICIP, 2021, reference

[0017] , exploited graph-convolution to extract both local and global features at the same time.

[0017] Attention Mechanism in LIC

[0018] Motivated by the success of attention mechanisms in tasks related to natural language processing and computer vision, many tried to introduce attention modules in the compression framework to optimize the bitrate allocation. To -SVT078-PC00- capture long-range dependencies between pixels, non-local attention blocks were first integrated within the end-to-end image compression frameworks (see reference [7], Lei Zhou et al., "End-to-end optimized image compression with attention mechanism", CVPR Workshops, 2019, reference

[0018] , and Haojie Liu et al., "Non-local attention optimized deep image compression", ArXiv, 2019, reference

[0019] ).

[0019] In reference [8], the authors notice instead that generating attention maps based on spatially neighboring elements can improve RD performance with fewer computations. For this reason, they replaced the non-local attention module of reference

[0019] with a window block for focusing on regions with high contrast and to put more effort into image details. In a similar spirit, reference

[0010] proposed a parallel Transformer-CNN mixture block with a controllable complexity, able to join the local modeling ability of the convolution and the non-local modeling ability of transformers. Besides, they introduced a Swin-attention module in the channel-wise entropy estimation architecture. Vision Graph Neural Networks

[0020] Meanwhile, Graph Neural Networks (GNNs), typically used for graph-based data (see Thomas N Kipf and Max Welling, "Semi-supervised classification with graph convolutional networks", ICLR, 2017, reference

[0020] ), have demonstrated remarkable success when applied to tasks such as image classification (see Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu, "Vision GNN: An image is worth graph of nodes", NeurlPS, 2022, reference

[0021] , and Yan Han et al., "Vision HGNN: An image is more than a graph of nodes", ICCV, 2023, reference

[0022] ), and segmentation (Jhony H. Giraldo et al., "Hypergraph convolutional networks for weakly-supervised semantic segmentation", ICIP, 2022, reference

[0023] ).

[0021] Drawing inspiration from the partition concept introduced in ViTs (see Alexey Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale", ICLR, 2021, reference

[0024] ), the input image undergoes segmentation into smaller patches, with each patch representing a node in the graph of the image. Connections between nodes are established using the k-NN -SVT078-PC00- technique in the feature space. Subsequently, graph convolution operators are employed to update the features of each node, considering its position in the graph (see Justin Gilmer et al., "Neural message passing for quantum chemistry", ICML, 2017, reference

[0025] ). Following a similar paradigm as Transformers, these features contribute to the classification of the entire graph, thereby classifying the entire image.

[0022] Several works showed that the attention mechanism is key for LIC models' compression efficiency. For example, reference

[0019] introduced the Non-Local Attention Module (NLAM), consisting of a cascade involving a non-local block and regular convolutional layers. Subsequently, reference [8] replaced the non-local block with a window block to capture local dependencies. However, none of the above approaches exploits the full potential of GNNs that, to the present date, remain largely untapped.

[0023] The present invention aims at solving these and other problems by providing a method for enhanced image encoding and decoding using a graph-based attention block, and related encoder and decoder, that avoids the aggregation of redundant visual features, thereby enhancing encoding and decoding performance.

[0024] A further scope of the present invention is to provide a method for enhanced image encoding and decoding using a graph-based attention block, and related encoder and decoder, that improves encoding efficiency, particularly at high bit rates, by better preserving local details and high-frequency components.

[0025] A further scope of the present invention is to provide a method for enhanced image encoding and decoding using a graph-based attention block, and related encoder and decoder, that ensures that high-contrast and high-detail areas receive adequate bit allocation without unnecessary redundancy.

[0026] A further scope of the present invention is to provide a method for enhanced image encoding and decoding using a graph-based attention block, and related encoder and decoder, that reduces computational overhead while still achieving superior encoding results.

[0027] A further scope of the present invention is to provide a method for enhanced -SVT078-PC00- image encoding and decoding using a graph-based atention block, and related encoder and decoder, that is versatile and can be applied to different types of images.

[0028] In a nutshell, the present patent application introduces a novel atention mechanism called Graph-based Atention Block for Image Compression (GABIC).

[0029] GABIC preferably relies on a graph-based k-Nearest Neighbors (k-NN) mechanism at attention time to reduce the number of redundant features.

[0030] The contributions of the method according to the present application can be summarized as follows:

[0031] • to the best of the Applicant's knowledge, GABIC is the first local graphbased attention mechanism that allows clustering redundant features employed in an end-to-end compression model;

[0032] • the experimental results show that the method according to the present invention improves the encoding efficiency with respect to comparable approaches, especially at lower distortion.

[0033] In accordance with the invention, it is disclosed a method for image encoding using a graph-based atention module comprising a graph-based window block configured to process an input feature map by applying graph-based atention mechanisms within a localized window of the input feature map, wherein the method comprises the steps of: providing an output of the graph-based window block to a first sequence of residual blocks, the first sequence of residual blocks being configured to apply a first series of transformations to refine feature representations of the image; providing an output of the first sequence of residual blocks to a 1x1 convolutional layer, the 1x1 convolutional layer being configured to perform a linear transformation for each feature independently; passing an output of the 1x1 convolutional layer through an activation function; element-wise multiplying, in a multiplication block, an output of the activation function with an output of a second sequence of residual blocks, the second sequence of residual blocks taking as input the input feature map and being configured to apply a second series of -SVT078-PC00- transformations to further refine the feature representations of the image; adding an output of the multiplication block to the input feature map by means of an element-wise addition block.

[0034] Following an aspect of the encoding method according to the invention, the graph-based window block is configured to: associate each patch of the localized window with a node; establish connections between nodes to obtain connected nodes, wherein features of the connected nodes form a local graph structure; apply the graph-based attention mechanism to compute attention coefficients between the connected nodes, wherein the attention coefficients are learnable and measure the importance of each connection between the nodes in the local graph structure.

[0035] Following a further aspect of the encoding method according to the invention, the features associated with each node are updated considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

[0036] Following a further aspect of the encoding method according to the invention, establishing connections between nodes is obtained through a k-Nearest Neighbors algorithm.

[0037] Following a further aspect of the encoding method according to the invention, the attention coefficients are obtained through a softmax function.

[0038] In accordance with the invention, it is further disclosed an encoder for image encoding using a graph-based attention module comprising a graph-based window block configured to process an input feature map by applying graph-based attention mechanisms within a localized window of the input feature map, wherein the encoder comprises: a first sequence of residual blocks configured to receive an output of the graph-based window block and to apply a first series of transformations to refine feature representations of the image; a 1x1 convolutional layer configured to receive an output of the first sequence of residual blocks and to perform a linear transformation for each feature independently; an activation function configured to receive and process an output of the 1x1 convolutional layer; a multiplication block configured to -SVT078-PC00- element-wise multiply an output of the activation function with an output of a second sequence of residual blocks, the second sequence of residual blocks being configured to take as input the input feature map and to apply a second series of transformations to further refine the feature representations of the image; an element-wise addition block configured to add an output of the multiplication block to the input feature map.

[0039] Following an aspect of the encoder according to the invention, the graph-based window block is configured to: associate each patch of said localized window with a node; establish connections between nodes to obtain connected nodes, wherein features of the connected nodes form a local graph structure; apply the graph-based attention mechanism to compute attention coefficients between the connected nodes, wherein the attention coefficients are learnable and measure the importance of each connection between the nodes in the local graph structure.

[0040] Following a further aspect of the encoder according to the invention, the graph-based attention module is configured to update the features associated with each node considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

[0041] Following a further aspect of the encoder according to the invention, the graphbased window block is configured to establish connections between nodes through a k-Nearest Neighbors algorithm.

[0042] Following a further aspect of the encoder according to the invention, the graphbased attention module is configured to calculate the attention coefficients through a softmax function.

[0043] In accordance with the invention, it is further disclosed a method for image decoding using a graph-based attention module comprising a graph-based window block configured to process a discrete main latent representation of the image by applying graph-based attention mechanisms within a localized window of the input feature map, wherein the method comprises the steps of: providing an output of the graph-based window block to a first sequence of residual blocks, the first sequence of residual blocks being -SVT078-PC00- configured to apply a first series of transformations to refine feature representations of the image; providing an output of the first sequence of residual blocks to a 1x1 convolutional layer, the 1x1 convolutional layer being configured to perform a linear transformation for each feature independently; passing an output of the 1x1 convolutional layer through an activation function; element-wise multiplying, in a multiplication block, an output of the activation function with an output of a second sequence of residual blocks, the second sequence of residual blocks taking as input the input feature map and being configured to apply a second series of transformations to further refine the feature representations of the image; adding an output of the multiplication block to the input feature map by means of an element-wise addition block.

[0044] Following an aspect of the decoding method according to the invention, the graph-based window block is configured to: associate each patch of the localized window with a node; establish connections between nodes to obtain connected nodes, wherein features of the connected nodes form a local graph structure; apply the graph-based attention mechanism to compute attention coefficients between the connected nodes, wherein the attention coefficients are learnable and measure the importance of each connection between the nodes in the local graph structure.

[0045] Following a further aspect of the decoding method according to the invention, the features associated with each node are updated considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

[0046] Following a further aspect of the decoding method according to the invention, establishing connections between nodes is obtained through a k-Nearest Neighbors algorithm.

[0047] Following a further aspect of the decoding method according to the invention, the attention coefficients are obtained through a softmax function.

[0048] In accordance with the invention, it is further disclosed a decoder for image decoding using a graph-based attention module comprising a graph-based window block configured to process a discrete main latent representation of the -SVT078-PC00- image by applying graph-based attention mechanisms within a localized window of the input feature map, wherein the encoder comprises: a first sequence of residual blocks configured to receive an output of the graph-based window block and to apply a first series of transformations to refine feature representations of the image; a 1x1 convolutional layer configured to receive an output of the first sequence of residual blocks and to perform a linear transformation for each feature independently; an activation function configured to receive and process an output of the 1x1 convolutional layer; a multiplication block configured to element-wise multiply an output of the activation function with an output of a second sequence of residual blocks, the second sequence of residual blocks being configured to take as input the input feature map and to apply a second series of transformations to further refine the feature representations of the image; an element-wise addition block configured to add an output of the multiplication block to the input feature map.

[0049] Following an aspect of the decoder according to the invention, the graph-based window block is configured to: associate each patch of the localized window with a node; establish connections between nodes to obtain connected nodes, wherein features of the connected nodes form a local graph structure; apply the graph-based attention mechanism to compute attention coefficients between the connected nodes, wherein the attention coefficients are learnable and measure the importance of each connection between the nodes in the local graph structure.

[0050] Following a further aspect of the decoding method according to the invention, the graph-based attention module is configured to update the features associated with each node considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients. Following a further aspect of the decoding method according to the invention, the graph-based window block is configured to establish connections between nodes through a k-Nearest Neighbors algorithm.

[0051] Following a further aspect of the decoding method according to the invention, the graph-based attention module is configured to calculate the attention -SVT078-PC00- coefficients through a softmax function.

[0052] The invention further discloses an autoencoder that comprises an encoder and a decoder as described.

[0053] The invention will be described in detail hereinafter through non-limiting embodiments with reference to the attached Figures, wherein:

[0054] Figure 1 shows an image divided into windows and patches where each patch represents a node in a graph;

[0055] Figure 2a shows a general architecture of a state-of-the-art LIC model enhanced with attention mechanisms according to the invention;

[0056] Figure 2b shows in detail an hyper-prior architecture which is part of the general architecture of Figure 2a;

[0057] Figure 2c shows a Graph Window Attention Module (G-WAM) according to the invention;

[0058] Figure 3a shows a traditional local window block scheme;

[0059] Figure 3b shows a Local Graph-based Window block according to the invention;

[0060] Figure 4 represents a table showing a comparison between the method according to the invention and prior art methods in relationship to the averaged encoding and decoding time, FLOPs and memory footprint on Kodak dataset using a GPU of the NVIDIA A40 type;

[0061] Figure 5a and 5b show rate distortion plots for the Kodak and CLIC datasets, respectively;

[0062] Figure 6 represents original images (a, d), bits allocation for low bitrates (b, e) and for high bitrates (c, f), where black indicates more bits for the method according to the present invention and white means less bits for the method according to the present invention compared to the prior art architecture named Zou2022.

[0063] The intuition behind the present invention is that if the self-attention mechanism can be locally constrained within a local graph, that may avoid the aggregation of redundant visual features, as illustrated in Figure 1.

[0064] Figure 1 thus shows that by dividing an image x into patches 1, it is possible to -SVT078-PC00- compress their content using a GNN. This allows better preserving of local details due to the dynamic graph structure on which the attention is computed, enabling higher compressibility at high bit rates.

[0065] Figure 2a shows the general architecture of a state-of-the-art LIC model enhanced with an attention mechanism GABIC according to the invention.

[0066] The model comprises an encoder-decoder pair, with attention computation facilitated by a graph-based window block within a graph structure according to the invention.

[0067] Additionally, the architecture builds upon the hyper-prior architecture of reference [5] and incorporates a channel-wise entropy model described in reference

[0015] .

[0068] The encoder-decoder pair, or autoencoder 10, comprises an encodera(-) and a decoder s(-)-

[0069] The encodera(-) receives an input image x which is treated by a series of layers and modules comprising:

[0070] -a first convolutional layer 102 followed by a first GDN (Generalized Divisive Normalization) layer 104;

[0071] -a second convolutional layer 106 followed by a second GDN layer 108;

[0072] -a first Graph-based Window Attention Module 110 (G-WAM) to reduce redundancy in the feature representations by focusing attention on relevant patches within a localized region of the image;

[0073] -a third convolutional layer 112 followed by a third GDN layer 114;

[0074] -a fourth convolutional layer 116 followed by a second Graph-based Window Attention Module 118 (G-WAM).

[0075] The first, second and third GDN layers 104,108,114 normalize the activations, reducing redundancy and enhancing the feature representation, while the first, second, third and fourth convolutional layers 102,106,112,116 perform convolution with N filters, each with a size of 5x5 pixels. The convolution is performed with a stride of 2, which means that the filter moves 2 pixels at a time. This results in a reduction of the spatial dimension of the image or feature map by a factor of 2 (both in width and height), performing a downsampling. -SVT078-PC00-

[0076] Obviously, the number of filters N, the size of pixels and the stride may vary according to the image compression needs.

[0077] The output of the encoder / •) is a main latent representation y which is then quantized by a quantizer 152, thereby obtaining a discrete main latent representation y : y is referred to as main latent representation, since the image is directly reconstructed from it.

[0078] To allow the transmission of the discrete main latent representation y from the encoding side to the decoding side of the autoencoder 10, a second entropy encoder 154, in particular an arithmetic encoder, is configured to process the discrete main latent representation y to provide a second compressed latent space 156 in the form of a bitstream which is then fed to a second entropy decoder 158, in particular an arithmetic decoder, for providing a second decompressed latent space corresponding to the discrete main latent representation y.

[0079] With reference to Figure 2b, the main latent representation y is divided into slices 173,174 and input to the channel-wise autoregressive entropy model block 150 and each slide is encoded in order using those previously encoded.

[0080] The channel-wise autoregressive entropy model block 150 comprises slice encoders 175,176. In addition, the slice encoder blocks 175,176 exploit the output of the hyperprior encoder / decoder 177.

[0081] The hyperprior encoder / decoder 177 comprises a hyper encoder 162 which is responsible for encoding the main latent representation y into a hyperprior latent representation z. This process involves capturing higher-order spatial dependencies that are not represented in the primary latent space.

[0082] The hyperprior latent representation z is then fed into an arithmetic encoder 164. This encoder compresses the hyperprior latent representation into a more compact form, suitable for efficient storage and transmission. The arithmetic encoder 164 applies entropy coding techniques, such as arithmetic coding, to encode z.

[0083] The output of the arithmetic encoder 164 is a compressed bitstream representing the hyperprior latent representation. This bitstream is transmitted or stored for -SVT078-PC00- later decoding and serves as an intermediate representation that efficiently captures the information in z.

[0084] On the decoding side, the compressed bitstream is fed into an arithmetic decoder 168. This decoder reconstructs the hyperprior latent representation from the compressed bitstream. The arithmetic decoder reverses the entropy coding process applied by the arithmetic encoder, ensuring accurate reconstruction of The reconstructed hyperprior latent representation is then passed through a hyper decoder 170. The hyper decoder 170 generates the parameters (mean / / and standard deviation u) for the Gaussian distribution used to model the main latent representation y. The output of the hyper decoder includes the parameters required for accurate entropy modeling of y, enhancing the overall compression efficiency.

[0085] Residual vectors r are the outputs of blocks 178 called LRPi and LRP2, wherein LRP stands for Layer-wise Relevance Propagation.

[0086] Towards this end, the discrete main latent representation y, or feature map, is divided into a predetermined number of slices, and then the spatial context of a specific slice is extracted by integrating information, via an adder block 160 (see Figure 2a), from both the hyperprior and a channel context model that receives previously decoded slices as input.

[0087] In addition, also a residual vector r obtained during the quantization step is estimated by the channel-wise autoregressive entropy model block 150, which is then added to the discrete main latent representation y, via the adder block 160, to reduce quantization error.

[0088] An output 157 of the channel-wise autoregressive entropy model block 150 is represented in terms of mean / / and standard deviation o representing spatial correlation for each element of the discrete main latent representation y since the latter is modeled as a Gaussian distribution.

[0089] The standard deviation o of the output 157 is fed both to the first arithmetic encoder 154 and to the second arithmetic decoder 156, while the mean / / of the output 157 is used to translate the discrete main latent representation y so that it has zero mean when compressed. -SVT078-PC00-

[0090] Finally, the discrete main latent representation y is fed to an image reconstruction decoders(-), thereby obtaining a reconstructed image .

[0091] The decoder s(-) includes a series of layers and modules comprising:

[0092] -a third Graph-based Window Attention Module 202 (G-WAM);

[0093] -a first deconvolutional layer 204 followed by a fourth GDN layer 206;

[0094] -a third deconvolutional layer 208;

[0095] -a fourth Graph-based Window Attention Module 210 (G-WAM);

[0096] -a fifth GDN layer 212 followed by a third deconvolutional layer 214;

[0097] -a sixth GDN layer 216 followed by a fourth deconvolutional layer 218.

[0098] The fourth, fifth and sixth GDN layers 206, 212, 216 normalize the activations at each stage of the decoding process, while the first, second, third and fourth deconvolutional layers 204,208,214,218 perform deconvolution with N filters, each with a size of 5x5 pixels. The deconvolution is performed with a stride of 2, which means that the filter moves 2 pixels at a time. This results in an increase of the spatial dimension of the image or feature map by a factor of 2 (both in width and height), performing an upsampling.

[0099] It is highlighted that the above described encodera(-) and decoder ?(•) are just ones of the possible implementations of an encoder and a related decoder. However, it is essential that both the encoder and the decoder comprise at least one Graph-based Window Attention Module according to the invention.

[0100] Figure 2c illustrates the internal structure of the Graph-based Window Attention Module 110 (G-WAM), a key component of the Graph-based Attention Block for Image Compression (GABIC), while Figure 2a showcases the integration of the G-WAM modules into both encoder and decoder pair of the architecture according to the invention.

[0101] The block 110 (G-WAM) comprises a graph-based Window Block 112 which is the core component that processes the input feature map by leveraging graphbased attention mechanisms within localized windows. It focuses on reducing redundancy and improving feature representation by dynamically adjusting the attention based on the similarity between patches within a window.

[0102] The block 110 further comprises multiple residual blocks RB, each responsible for -SVT078-PC00- refining the features extracted from the input. The residual blocks consist of a sequence of layers that learn residual mappings, which are added to the original features to enhance the overall representation.

[0103] A first sequence 114 of residual blocks RB takes the output from the Graph-based Window Block 112 and applies a series of transformations to refine the feature representations, while a second sequence 116 of residual blocks takes the original input and applies similar transformations in parallel to the first sequence.

[0104] A 1x1 convolutional layer 118 is applied to the output of the first sequence of residual blocks 114. This layer serves to reduce the dimensionality of the feature maps and to fuse the features extracted by the residual blocks.

[0105] The output of the 1x1 convolutional layer 118 is passed through an activation function 120. The activation function 120 squashes the feature values to be within the range [0, 1]. Preferably, the activation function 120 is a Sigmoid activation function.

[0106] Further, the output from the activation function 120 is element-wise multiplied in a multiplication block 122 with the output of the second sequence of residual blocks. This multiplication serves as an attention mechanism that modulates the features based on their importance. By multiplying the features, the model effectively prioritizes the most relevant features while suppressing less important ones.

[0107] Finally, the result of the element-wise multiplication is then added to the original input feature map by means of an element-wise addition 124. This residual connection helps in preserving the original information while integrating the enhanced features learned by the residual blocks. The addition operation ensures that the final output retains both the original and the refined feature information, leading to a more robust representation.

[0108] Preliminaries

[0109] Let us represent a graph as the mathematical entity G = (V, ), where V = {1, . . . , N] is the set of N nodes and E £ {(zy) | j e V and i j} is the set of edges between nodes i and j. In GNNs, each node is typically updated by -SVT078-PC00- embedding i using a message-passing mechanism within the set of neighbors N(z) of node i (see reference

[0025] ). The attention mechanism GABIC according to the invention also uses the message-passing operations coupled with attention.

[0110] Architecture

[0111] The above LIC model is characterized by a hyperprior-based architecture (see reference [5]) and a channel-wise entropy model (see reference

[0015] ) as shown in Figure 2a, Figure 2b and Figure 2c. Thus, the model can be formulated as: where 0 and 0 are trainable parameters of the encoder fa(-) and the decoder / ?(•) respectively, while Q(-) is the rounding quantization function.

[0112] As previously mentioned, y is estimated to have Gaussian distribution with mean p. Following previous works (see references [6] and [8]), it is rounded and encoded [y - p] instead of [y], as it has been proven to benefit the entropy models; it is exploited a range coder to loosely encode [y - |i], which is modeled with a Gaussian distribution with standard deviation u, and it is transmitted to the decoder / s(-)- The mean / i and the standard deviation o are extracted by leveraging both the hyperprior encoder-decoder lzfl(-), / / ?(•) and the channel-wise entropy model.

[0113] In particular, y is divided into s slices which are then encoded sequentially so that the already encoded slices can improve the encoding of the current one; furthermore, a slice network <p;(-) has been introduced to reduce the error due to quantization artifact. The overall pipeline can be formulated as follows: y< - Q(yi ~~ / A) 4- / A > v-• ; <• r. <<• 4- v;x■ where 0 < i < s and represents the slices already -SVT078-PC00- encoded. As presented in formula (3), the hyperprior autoencoder is meant to capture spatial dependencies among y employing (D / ;, Da); for a specific slice, the latter are used in combination with and y; as input to the slice network <p((-)zyielding both the Gaussian parameters and a residual vector r;, which has been introduced to reduce the quantization error. Once passed toward all the slices, y is entered in the decoder / ?(•) instead of y in formula (2), obtaining the reconstructed image x.

[0114] The model is finally trained minimizing a rate-distortion loss function formulated as:

[0115] The terms of formula (4) are as follows:

[0116] L represents the overall loss function that the model aims to minimize during training; is an expectation operator, indicating that the loss is computed as an average over the distribution of input images (x) sampled from the data distribution (px);

[0117] — k%2 PyjsdS =^J represents the negative log-likelihood of the quantized latent representation y given the quantized hyperprior z wherein is the conditional probability distribution of y given z;

[0118] — represents the negative log-likelihood of the quantized hyperprior z; PMz?is the probability distribution of z;

[0119] X is a hyperparameter that controls the trade-off between the rate (bitrate) and the distortion (image quality). It balances the two terms in the loss function; • 7 > represents the expected distortion between the original image x and the reconstructed image x; -Ax'X !is a distortion metric, such as mean squared error (MSE), which measures the difference between the original and reconstructed images x, x. -SVT078-PC00-

[0120] Graph Window Attention Block

[0121] GABIC computes local attention maps over non-overlapping windows. However, the self-attention mechanism is not computed as in regular Swin Transformer models (see references [8] and

[0011] ), but instead each patch of these windows is treated as a node of a graph. Therefore, it is used the graph to compute the attention operation.

[0122] Figure 3b shows an illustration of the updating operation according to the present invention (in contrast to the conventional self-attention in Figure 3a), where the input feature map is divided into M x M windows. Let x(z) be the embedding of the patch i in the local window. The embedding of the patch i is updated using a graph convolution operation described as follows: where Wzand W« are learnable parameters for the specific window, and az,y are the attention coefficients computed as:

[0123] The neighbors of i are dynamically updated on each iteration using a k-NN in the feature space such that:

[0124] JV(£) ™ -{(-L J) : j E fc- where k-NN[x(z)] is the set of k-NN of the feature x(z).

[0125] Training. To facilitate comparisons with other end-to-end compression models, the graph-based architecture according to the invention was implemented in the CompressAI platform (see Jean Begaint et al.,"CompressAI: a PyTorch library and evaluation platform for end-to-end compression research," arXiv, 2020, reference

[0027] ), and the local Graph-based Window Block in Pytorch-Geometric (see Matthias Fey and Jan E. Lenssen, "Fast graph representation learning with -SVT078-PC00-

[0126] PyTorch Geometric", ICLR Workshops, 2019, reference

[0028] ).

[0127] The training was conducted by randomly selecting 300k images from the Openimages dataset (see Ivan Krasin et al., "Openimages: A public dataset for large-scale multi-label and multi-class image classification", Dataset available from https: / / github.com / openimages, 2016, reference

[0029] ) and randomly cropping them with the size of 256 x 256.

[0128] The models were trained for 200 epochs using Adam as optimizer (see Diederik P Kingma and Jimmy Ba, "Adam: A method for stochastic optimization", ICLR, 2015, reference

[0030] ), a batch size of 16, and an initial learning rate equal to 1 x 10-4and then decremented by a factor of 0.3 with patience of 10 epochs. Following the setting reported in reference

[0027] , the models were optimized using the Mean Squared Error (MSE) as distortion term in the loss, and using lambda values X belonging to [0.0067, 0.0130, 0.025, 0.0483}.

[0129] As done by Zou et al. (see reference [8]), the number of channels for the latent and hyper-latent spaces was configured to 320 and 192, respectively. Additionally, the window size for the first and second Graph-based Window Blocks was set in both the encoder and decoder to 8 and 4, respectively.

[0130] Evaluation

[0131] GABIC was evaluated in terms of RD performance (PSNR) on the commonly used Kodak image set (see Eastman Kodak, "Kodak lossless true color image suite (photocd pcd0992)", 1993, http: / / rOk. us / graphics / kodak / , reference

[0031] ) and the first 30 images of the CLIC professional dataset (see "Workshop and challenge on learned image compression", 2020, https: / / www.compression.cc / , reference

[0032] ).

[0132] The method according to the present invention was compared with the VCC- VTM standard codec (see reference [3]) and other well-known learned compression models, including the context-free hyperprior model (Balle2018), reference [5], auto-regressive hyper-prior model (Minnen2018), reference [6], and auto-regressive hyper-prior model with GMM and simplified attention (Cheng2020), reference [7], The results were also compared with Zou2022, reference [8], in which the standard local attention is implemented. -SVT078-PC00-

[0133] In Figure 4 it is shown that the graph-based module according to the invention preserves the same complexity as the local-attention one.

[0134] Rate-distortion comparison

[0135] Figure 5a and Figure 5b shows the rate-distortion curves for both Kodak and CLIC datasets, respectively. The method according to the present invention (labelled "our" in the Figures 5a and 5b) improves over Zou2022, reference [8], where the local attention maps are computed differently. Regarding this architecture, GABIC yields a BD-Rate gain of about 1.50% on Kodak and 0.89% on CLIC in terms of Bjontegaard metrics. In particular, the method according to the present invention performs especially better in the top-right part of the rate-distortion curves, at high PSNR and bit rate regime.

[0136] The analysis showed that the attention mechanism according to the invention indeed tries to capture irregular local shapes trying to preserve them. Thanks to this, fewer bits are allocated to high-contrast areas of the image, while preserving the correct level of details.

[0137] Allocation Maps Studies

[0138] Figure 6 shows the difference between the bit allocation map obtained using the Graph-based Window Block for coding according to the invention concerning the use of a standard Window Block at different compression regimes.

[0139] In white are highlighted the areas of the image where GABIC manages to allocate fewer bits than Zou2022, see reference [8], and in black the opposite.

[0140] It is observed that in a high-quality compression regime (top-right part of the RD curve), GABIC allocates fewer bits almost anywhere, especially in the high- frequency parts of the image that bear more perceptual importance and still obtain higher PSNR values.

[0141] However, at low-quality regimes the standard Window Block attention method is better at summarizing flat areas of the image, while the method according to the present invention captures details in these areas as well, allocating more bits than necessary for this compression regime. Indeed, in this case, it was also obtained higher PSNR values paid with a slightly higher bitrate.

[0142] To sum up, it was studied how local attention mechanisms work in image -SVT078-PC00- compression, and it was introduced GABIC, a novel method to compute the attention based on a graph constructed in each local window independently. Through extensive evaluations on well-established datasets like Kodak and CLIC, it was demonstrated that GABIC achieves superior results compared to state-of-the-art models. The gains come mainly at the more challenging high bit rates, where the fidelity is already high. The method according to the invention excels in capturing irregular local shapes and preserving them, which is reflected in the improved performance. To further highlight the capabilities of GABIC, a detailed study was conducted comparing the allocation bit maps produced by the model according to the present invention with those from models employing standard local attention mechanisms, with empirical results supporting the Applicant's hypotheses.

[0143] The present description has tackled some of the possible variants, but it will be apparent to the man skilled in the art that other embodiments may also be implemented, wherein some elements may be replaced with other technically equivalent elements. The present invention is not therefore limited to the explanatory examples described herein, but may be subject to many modifications, improvements or replacements of equivalents parts and elements without departing from the basic inventive idea, as set out in the following claims.

Claims

-SVT078-PC00-CLAIMS1. A method for image encoding using a graph-based attention module (G-WAM) comprising a graph-based window block (112) configured to process an input feature map by applying graph-based attention mechanisms within a localized window of said input feature map, wherein said method comprises the steps of: providing an output of said graph-based window block (112) to a first sequence (114) of residual blocks (RB), said first sequence (114) of residual blocks (RB) being configured to apply a first series of transformations to refine feature representations of said image; providing an output of said first sequence (114) of residual blocks (RB) to a 1x1 convolutional layer (118), said 1x1 convolutional layer (118) being configured to perform a linear transformation for each feature independently; passing an output of said 1x1 convolutional layer through an activation function (120); element-wise multiplying, in a multiplication block (122), an output of said activation function with an output of a second sequence (116) of residual blocks (RB), said second sequence (116) of residual blocks (RB) taking as input said input feature map and being configured to apply a second series of transformations to further refine said feature representations of said image; adding an output of said multiplication block (122) to said input feature map by means of an element- wise addition block (124).

2. A method for image encoding according to claim 1, wherein said graph-based window block (112) is configured to:- associate each patch of said localized window with a node;- establish connections between nodes to obtain connected nodes, wherein features of said connected nodes form a local graph structure;- apply said graph-based attention mechanism to compute attention coefficients between said connected nodes, wherein said attention coefficients are learnable and measure the importance of each connection between said nodes in said local graph structure.-SVT078-PC00-3. A method for image encoding according to claim 2, wherein said features associated with each node are updated considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

4. A method for image encoding according to claim 2 or 3, wherein establishing connections between nodes is obtained through a k-Nearest Neighbors algorithm.

5. A method for image encoding according to any of claims 2 to 4, wherein said attention coefficients are obtained through a softmax function.

6. An encoder for image encoding using a graph-based attention module (G-WAM) comprising a graph-based window block (112) configured to process an input feature map by applying graph-based attention mechanisms within a localized window of said input feature map, wherein said encoder comprises: a first sequence (114) of residual blocks (RB) configured to receive an output of said graph-based window block (112) and to apply a first series of transformations to refine feature representations of said image; a 1x1 convolutional layer (118) configured to receive an output of said first sequence (114) of residual blocks (RB) and to perform a linear transformation for each feature independently; an activation function (120) configured to receive and process an output of said 1x1 convolutional layer; a multiplication block (122) configured to element-wise multiply an output of said activation function with an output of a second sequence (116) of residual blocks (RB), said second sequence (116) of residual blocks (RB) being configured to take as input said input feature map and to apply a second series of transformations to further refine said feature representations of said image; an element-wise addition block (124) configured to add an output of said multiplication block (122) to said input feature map.

7. An encoder for image encoding according to claim 6, wherein said graph-based window block (112) is configured to:- associate each patch of said localized window with a node;-SVT078-PC00-- establish connections between nodes to obtain connected nodes, wherein features of said connected nodes form a local graph structure;- apply said graph-based attention mechanism to compute attention coefficients between said connected nodes, wherein said attention coefficients are learnable and measure the importance of each connection between said nodes in said local graph structure.

8. An encoder for image encoding according to claim 7, wherein said graph-based attention module (G-WAM) is configured to update said features associated with each node considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

9. An encoder for image encoding according to claim 7 or 8, wherein said graph-based window block (112) is configured to establish connections between nodes through a k-Nearest Neighbors algorithm.

10. An encoder for image encoding according to any of claims 7 to 9, wherein said graph-based attention module (G-WAM) is configured to calculate said attention coefficients through a softmax function.

11. A method for image decoding using a graph-based attention module (G-WAM) comprising a graph-based window block (112) configured to process a discrete main latent representation of said image by applying graph-based attention mechanisms within a localized window of said input feature map, wherein said method comprises the steps of: providing an output of said graph-based window block (112) to a first sequence (114) of residual blocks (RB), said first sequence (114) of residual blocks (RB) being configured to apply a first series of transformations to refine feature representations of said image; providing an output of said first sequence (114) of residual blocks (RB) to a 1x1 convolutional layer (118), said 1x1 convolutional layer (118) being configured to perform a linear transformation for each feature independently; passing an output of said 1x1 convolutional layer through an activation function (120);-SVT078-PC00- elem ent-wise multiplying, in a multiplication block (122), an output of said activation function with an output of a second sequence (116) of residual blocks (RB), said second sequence (116) of residual blocks (RB) taking as input said input feature map and being configured to apply a second series of transformations to further refine said feature representations of said image; adding an output of said multiplication block (122) to said input feature map by means of an element- wise addition block (124).

12. A method for image decoding according to claim 11, wherein said graph-based window block (112) is configured to:- associate each patch of said localized window with a node;- establish connections between nodes to obtain connected nodes, wherein features of said connected nodes form a local graph structure;- apply said graph-based attention mechanism to compute attention coefficients between said connected nodes, wherein said attention coefficients are learnable and measure the importance of each connection between said nodes in said local graph structure.

13. A method for image decoding according to claim 12, wherein said features associated with each node are updated considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

14. A method for image decoding according to claim 12 or 13, wherein establishing connections between nodes is obtained through a k-Nearest Neighbors algorithm.

15. A method for image decoding according to any of claims 12 to 14, wherein said attention coefficients are obtained through a softmax function.

16. A decoder for image decoding using a graph-based attention module (G-WAM) comprising a graph-based window block (112) configured to process a discrete main latent representation of said image by applying graph-based attention mechanisms within a localized window of said input feature map, wherein said encoder comprises: a first sequence (114) of residual blocks (RB) configured to receive an-SVT078-PC00- output of said graph-based window block (112) and to apply a first series of transformations to refine feature representations of said image; a 1x1 convolutional layer (118) configured to receive an output of said first sequence (114) of residual blocks (RB) and to perform a linear transformation for each feature independently; an activation function (120) configured to receive and process an output of said 1x1 convolutional layer; a multiplication block (122) configured to element-wise multiply an output of said activation function with an output of a second sequence (116) of residual blocks (RB), said second sequence (116) of residual blocks (RB) being configured to take as input said input feature map and to apply a second series of transformations to further refine said feature representations of said image; an element-wise addition block (124) configured to add an output of said multiplication block (122) to said input feature map.

17. A decoder for image decoding according to claim 16, wherein said graph-based window block (112) is configured to:- associate each patch of said localized window with a node;- establish connections between nodes to obtain connected nodes, wherein features of said connected nodes form a local graph structure;- apply said graph-based attention mechanism to compute attention coefficients between said connected nodes, wherein said attention coefficients are learnable and measure the importance of each connection between said nodes in said local graph structure.

18. A decoder for image decoding according to claim 17, wherein said graph-based attention module (G-WAM) is configured to update said features associated with each node considering the features of the node itself and the ones of its neighbors properly weighted with respect to the learned attention coefficients.

19. A decoder for image decoding according to claim 17 or 18, wherein said graph-based window block (112) is configured to establish connections between nodes through a k-Nearest Neighbors algorithm.-SVT078-PC00-20. A decoder for image decoding according to any of claims 17 to 19, wherein said graph-based attention module (G-WAM) is configured to calculate said attention coefficients through a softmax function.

21. An autoencoder (10) comprising an encoder according to any of the claims 1 to 5 and a decoder according to any of the claims 6 to 10.

Images