Scalable learned image compression with adaptive learned sparse image representation

Abstract

A method for performing scalable Learned Image Compression (LIC) with adaptive Learned Sparse Image Representation (LSLR). The method, implemented by a sending device, includes encoding an input image x by K input encoders of the sending device to generate K image feature tensors wherein K is an integer greater than 1; computing K sparse codebook-based latent features respectively based on the K image feature tensor and corresponding basis codebook; computing a weight map that provides information for weighted combining K decoded image embedding features, wherein the K decoded image embedding features is respectively based on the K sparse codebook-based latent features and the corresponding basis codebook; computing a compact weight map based on the weight map and a target compression rate; and transmitting the compact weight map and the K sparse codebook-based latent features to a receiving device

Description

Atty. Docket No.4502-81701 (6000635PCT02) SCALABLE LEARNED IMAGE COMPRESSION WITH ADAPTIVE LEARNED SPARSE IMAGE REPRESENTATION CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No.63 / 527,278 filed on July 17, 2023. The disclosure of the aforementioned application is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to Learned Image Compression (LIC), and in particular, to scalable LIC with adaptive Learned Sparse Image Representation (LSLR). BACKGROUND

[0003] LIC based on neural networks has been largely studied in recent years and has shown superior performance over traditional coding methods like Joint Photographic Experts Group (JPEG), Versatile Video Coding (VVC), and High Efficiency Video Coding (HEVC). Sparse representation learning has been an interesting research topic for modelling generic images. Learned Sparse Image Representation (LSIR) learns a highly compressed codebook in the image domain by a vector-quantized autoencoder that is trained with adversarial and perceptual loss. SUMMARY

[0004] A first aspect relates to a method, implemented by a sending device, that includes encoding an input image x by K input encoders of the sending device to generate K image feature tensors ^^^, ⋯ , ^^^, wherein K is an integer greater than 1; computing K sparse codebook-based latent features ^^^^, ⋯ , ^^^^respectively based on the K image feature tensor ^^^, ⋯ , ^^^andAtty. Docket No.4502-81701 (6000635PCT02) corresponding basis codebook ^^, ⋯ , ^^; computing a weight map W that provides information for weighted combining K decoded image embedding features ^̂^^, ⋯ , ^̂^^, wherein the K decoded image embedding features ^̂^^, ⋯ , ^̂^^is respectively based on the K sparse codebook-based latent features ^^^^, ⋯ , ^^^^and the corresponding basis codebook ^^, ⋯ , ^^; computing a compact weight map ^^based on the weight map W and a target compression rate R; and transmitting the compact weight map ^^and the K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^to a receiving device.

[0005] Optionally, in a first implementation according to the first aspect, computing the compact weight map ^^based on the weight map W and the target compression rate R includes computing a masking rate ^ based on the target compression rate R; generating a weight mask M based on the masking rate ^; computing the compact weight map ^^based on the weight map W and the weight mask.

[0006] Optionally, in a second implementation according to the first aspect or any implementation thereof, the masking rate τ is a ratio of masked-out weights against all weights.

[0007] Optionally, in a third implementation according to the first aspect or any implementation thereof, the method further includes generating the weight mask M based on the masking rate τ and a masking strategy.

[0008] Optionally, in a fourth implementation according to the first aspect or any implementation thereof, the method further includes computing a target bit count ^^^^^based on the target compression rate R; and determining the masking strategy based on the target bit count ^^^^^.

[0009] Optionally, in a fifth implementation according to the first aspect or any implementation thereof, the method further includes transmitting the weight mask M to the receiver.

[0010] Optionally, in a sixth implementation according to the first aspect or any implementation thereof, the method further includes computing a distortion loss D measuring a distortion between the input image x and a reconstructed output image ^^, and / or between a target combined latent feature ^̂^∗and a combined latent feature ^̂^, and modifying weights corresponding to non-masked- out pixels in the compact weight map ^^to reduce the distortion loss D.Atty. Docket No.4502-81701 (6000635PCT02)

[0011] Optionally, in a seventh implementation according to the first aspect or any implementation thereof, the method further includes computing, based on a plurality of candidate masking strategies, a distortion loss D measuring a distortion between the input image x and a reconstructed output image ^^, and / or between a target combined latent feature ^̂^∗and a combined latent feature ^̂^, and selecting a masking strategy from the plurality of candidate masking strategies having a minimum distortion loss D.

[0012] A second aspect relates to a method, implemented by a receiving device, that includes receiving K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^and a compact weight map ^^from a sending device, wherein K is an integer greater than 1; computing K decoded image embedding features ^̂^^, ⋯ , ^̂^^based on the K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^; computing a filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^; computing a combined latent feature ^̂^based on the filled weight map ^^ ^^^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^; and computing a reconstructed output image ^^ based on the combined latent feature ^̂^.

[0013] Optionally, in a first implementation according to the second aspect, computing the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^includes predicting weights for masked-out weights in the compact weight map^^.

[0014] Optionally, in a second implementation according to the second aspect or any implementation thereof, computing the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^,, comprises predicting weights for unused basis codebooks in computing the K decoded image embedding features ^̂^^, ⋯ , ^̂^^.

[0015] Optionally, in a third implementation according to the second aspect or any implementation thereof, computing the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^computing a set of new weights for recovering a target combined latent feature ^̂^∗.Atty. Docket No.4502-81701 (6000635PCT02)

[0016] Optionally, in a fourth implementation according to the second aspect or any implementation thereof, computing, by a neural network model, the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^.

[0017] Optionally, in a fifth implementation according to the second aspect or any implementation thereof, the neural network model is optimized for a masking strategy corresponding to the compact weight map ^^.

[0018] Optionally, in a sixth implementation according to the second aspect or any implementation thereof, computing the combined latent feature ^̂^based on the filled weight map ^^ ^^^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^comprises computing ^̂^,^(^ = 1, … , ^^× ℎ^), for each pixel in ^̂^as follows:

[0019] ^̂^,^=∑^^#^^"^^^^^,^∗ ^̂^^,^corresponding to l-th pixel in the filled weight map ^^ ^^^^.

[0021] A third aspect relates to an apparatus comprising a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the apparatus to perform the method according to the first aspect, the second aspect, or any implementation thereof.

[0022] A fourth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processors of an apparatus, cause the apparatus to perform a method according to the first aspect, the second aspect, or any implementation thereof.

[0023] For clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0024] These and other features, and the advantages thereof, will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.Atty. Docket No.4502-81701 (6000635PCT02) BRIEF DESCRIPTION OF DRAWINGS

[0025] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals between drawings represent like parts.

[0026] FIG.1 is a diagram illustrating a general framework for LIC.

[0027] FIG.2 is a diagram illustrating a general framework for LSIR.

[0028] FIG.3 is a diagram illustrating a general framework for adaptive LSIR according to an embodiment of the present disclosure.

[0029] FIG.4 is a diagram illustrating a general framework for scalable LIC with adaptive LSLR according to an embodiment of the present disclosure.

[0030] FIG.5 is a diagram illustrating an adaptive weight compression module according to an embodiment of the present disclosure.

[0031] FIG.6 is a flowchart illustrating a processing workflow of a sending device according to an embodiment of the present disclosure.

[0032] FIG.7 is a flowchart illustrating a processing workflow of a receiving device according to an embodiment of the present disclosure.

[0033] FIG.8 is a diagram illustrating an apparatus according to an embodiment of the present disclosure. DESCRIPTION OF EMBODIMENTS

[0034] It should be understood at the outset that, although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0035] Disclosed herein are various systems and methods for encoding and decoding an image. In particular, the present disclosure describes a general framework of LIC by using an adaptiveAtty. Docket No.4502-81701 (6000635PCT02) LSLR. The disclosed framework provides scalability to utilize increasingly large and complex LSIR to largely improve the compression performance of LIC by achieving high compression rate and high reconstruction quality at the same time. Specifically, the disclosed framework uses an adaptive weighting method with effective weight compression to flexibly select compression rates according to the input image and compression needs.

[0036] FIG.1 is a diagram illustrating a general framework for LIC. LIC is a modern approach to image compression that utilizes deep learning techniques to learn efficient representations of images. Traditional image compression techniques rely on handcrafted algorithms that transform the image data into a compressed format. However, LIC aims to improve compression efficiency by training neural networks to automatically learn the most effective compression strategies directly from the data. As shown in FIG. 1, on a sender side (i.e., implemented by a sending device), an input image x is passed through an input encoder 102 to generate an image embedding feature ^̂^, which is a numerical representation of the input image x. For instance, in an embodiment, theencoder 102 is a neural network that is configured to convert the raw pixel values of the input image x into a compressed and semantically meaningful numerical representation of the input image x in a high-dimensional vector space. In some embodiments, the image embedding feature ^̂^is further compressed through quantization and arithmetic coding into a data string that is efficient for storage and transmission from the sender to a receiver.

[0037] On the receiver side (i.e., implemented by a receiving device), the image embedding feature ^̂^is recovered (i.e., decoded) from the received data string using arithmetic decoding and dequantization. The decoded image embedding feature ^̂^is inputted into a decoder 104. The decoder 104 is a neural network configured to reconstruct an output image ^^ based on the decoded image embedding feature ^̂^. The goal of LIC is to minimize the restoration loss between the reconstructed output image $ ^ and the original input x, and to minimize the bits used to represent the image embedding feature ^^for storage and transmission.

[0038] The current LIC framework, as described in FIG.1, relies on learning a general compact image representation, i.e., a latent space where the image embedding feature ^^can capture theAtty. Docket No.4502-81701 (6000635PCT02) details of the input x to reconstruct ^^. The LIC framework has several severe limitations. First, the compression performance is innately bounded by the model capacity (e.g., the network structure and number of parameters of the input encoder and the decoder) in learning the general prior %(^|^^) in the image domain (i.e., in learning the characteristics and properties of images (i.e., prior image knowledge) that are used for generating the image embedding feature ^̂^representing the input image x). Due to the limited model capacity, limited training data, and limited computation resources in both training and test stage, it is hard to further improve the compression performance beyond a good baseline. Second, the LIC models are learned to balance the competing goals in the rate- distortion (RD) loss, where reducing reconstruction distortion and reducing bitrate contradicts with each other. Therefore, it is hard to improve the compression performance and the perceptual quality at the same time using LIC, due to the difficulty in balancing different loss terms in end-to-end training.

[0039] FIG. 2 is a diagram illustrating a general framework for LSIR. In LSIR, a vector- quantized autoencoder in the image domain is trained based on adversarial and perceptual loss (e.g., using the Vector Quantized Generative Adversarial Network (VQGAN) method) to learn a highly compressed codebook 202. The learned codebook 202 comprises a collection of codewords used in a compression algorithm. The goal is to represent an image using a set of codewords from the codebook 202 more efficiently than directly encoding each vector or group of pixels within the image. The learned codebook 202 is optimized end-to-end to balance codebook efficiency and reconstruction quality. As shown in FIG.2, an input image x is passed through the input encoder 102 to generate the image embedding feature ^̂^as described in FIG. 1. For LSIR, on the sender side, the image embedding feature ^̂^is mapped into a sequence of code indices ^^^using the learned codebook 202. The code indices ^^^are integers that can be effectively stored or transmitted from the sender to a receiver. On the receiver side, the same learned codebook 202 is used to recover / decode the image embedding feature ^̂^(e.g., by using the codewords in the codebook 202 corresponding to the received code indices ^^^). The decoder 104 is then configured to reconstruct an output image ^^ based on the decoded image embedding feature ^̂^.Atty. Docket No.4502-81701 (6000635PCT02)

[0040] In comparing LSIR as described in FIG.2 with LIC as described in FIG.1, LSIR shows inferior performance in image compression. This is because when used for compression, LSIR has an aggressive goal of high compression rate by using a compact codebook to model the complicated generic image prior %(^|^^). Thus, the reconstructed image usually lacks expressive and fidelity details.

[0041] FIG.3 is a diagram illustrating a general framework for adaptive LSIR according to an embodiment of the present disclosure. The goal of adaptive LSIR is to improve the performance of general image restoration over arbitrary image content. In adaptive LSIR, the learned codebook 202 as described in FIG.2, is diversified into K basis codebooks ^^, ⋯ , ^^, where each codebook targets a particular class category of images. For example, in some embodiments, the class categories are semantic categories of images, such as indoor objects, natural scenes, portraits and so on. Adaptive LSIR can also be applied to face images (i.e., for face semantic category only), and the class categories for diversifying the basis codebooks are the image distributions. As shown in FIG.3, on the sender side, an input image x is encoded using K input encoders (e.g., input encoder 1302A, …, input encoder K 302K) to generate K image embedding features ^^^, ⋯ , ^^^, respectively. In some embodiments, the input encoder 1 302A, …, input encoder K 302K are the same, and thus ^^^, ⋯ , ^^^are also the same. In other embodiments, the input encoder 1302A, …, input encoder K 302K are different, and thus ^^^, ⋯ , ^^^are different. The K image embedding features ^^^, ⋯ , ^^^are then respectively mapped into K sequences of code indices ^^^^, ⋯ , ^^^^by code generation module 1 306A, …, code generation module K 306K using the corresponding basis codebooks ^^304A, ⋯ , ^^304K . Additionally, on the sender side, the K image embedding features ^^^, ⋯ , ^^^are passed (i.e., inputted) to a weight predictor module 310 that is configured to compute a weight map W. The weight map W guides how the decoded image embedding features ^̂^^, ⋯ , ^̂^^on the receiver side can be weighted combined to compute the final image embedding feature ^̂^. The code indices ^^^^, ⋯ , ^^^^and the weight map W are transmitted from the sender to a receiver.Atty. Docket No.4502-81701 (6000635PCT02)

[0042] On the receiver side, feature retrieval module 1308A, …, feature retrieval module K 308K are configured to recover / decode the respective image embedding features ^^^, ⋯ , ^^^using the corresponding basis codebooks ^^304A, ⋯ , ^^304K to obtain decoded image embedding features ^̂^^, ⋯ , ^̂^^. The weight map W and the decoded image embedding features ^̂^^, ⋯ , ^̂^^are passed to a weighted combine module 312. The weighted combine module 312 is configured to weighted combine the decoded image embedding features ^̂^^, ⋯ , ^̂^^based on the weight map W to generate a combined latent feature ^̂^. The combined latent feature ^̂^is passed to a decoder 314 that is configured to reconstruct an output image ^^ based on the combined latent feature ^̂^.

[0043] Adaptive LSIR has been used for face-centric image compression. For instance, due to the highly structural characteristics of human faces, a high-quality (HQ) codebook can be robustly learned for face reconstruction. However, such an approach cannot easily extend to general images with general image content. For instance, in some uses of adaptive LSIR, image reconstruction performance over general image content can be improved by combining features from multiple basis codebooks that are diversified according to semantic categories. That is, semantic-class-guided training is used to learn semantic-class-guided codebooks so that improved semantic-class-agnostic restoration performance can be achieved. However, the image restoration performance relies on predicted image-adaptive weight maps (e.g., weight map W in FIG.3), which are not compression friendly. Therefore, the dense weight maps are impractical to transmit due to the heavy transmission overhead.

[0044] To address one or more of the above technical problems, the present disclosure describes a general framework that uses adaptive LSIR for highly effective LIC. The disclosed framework uses an adaptive weighting method with weight compression, which provides scalability to utilize increasingly large and complex adaptive LSIR that comprises basis codewords that are arbitrarily diversified according to semantic categories, image distributions, qualities, etc. Thus, the disclosed adaptive weighting mechanism can flexibly select compression rates according to the input image and the compression needs. As a result, the compression performance of LIC is largely improved to achieve both high compression rate and high reconstruction quality at the same time.Atty. Docket No.4502-81701 (6000635PCT02)

[0045] FIG.4 is a diagram illustrating a general framework for scalable LIC with adaptive LSLR according to an embodiment of the present disclosure. As shown in FIG.4, the disclosed scalable LIC with adaptive LSLR framework includes the general framework for adaptive LSIR as described in FIG.3 (i.e., like reference numerals in FIG.3 and FIG.4 represent like parts and perform similar functions as described in FIG.3). In contrast to the adaptive LSIR framework in FIG.3, the disclosed scalable LIC with adaptive LSLR framework in FIG. 4 includes an adaptive weight compression module 402 on the sender side and a weight filling module 404 on the receiver side. As described herein, the adaptive weight compression module 402 enables effective LIC with adaptive LSIR, which also provides flexibility to adaptively select compression rates according to the actual input image and compression needs.

[0046] As described in FIG. 3, on the sender side, an input image x is encoded using K input encoders (e.g., input encoder 1302A, …, input encoder K 302K) to generate K image embedding features ^^^, ⋯ , ^^^, respectively. In an embodiment, the image x is represented using a general three-dimensional (3D) tensor. A 3D tensor is a mathematical object that extends the concept of matrices to three dimensions. For example, in an embodiment, the general framework for scalable LIC with adaptive LSLR is provided with the image x, where x is a general 3D tensor with shape ^ × ℎ × ., where W, h, c are the width, height, and number of channels of the image. For example, . = 3 for color images, . = 1 for spectral images, or . = 4 for color and depth (RGB-D) images. As described above, the image x is encoded using K input encoders (e.g., input encoder 1302A, …, input encoder K 302K) to generate K image embedding feature tensors ^^^, ⋯ , ^^^, respectively. In some embodiments, the K input encoders (i.e., input encoder 1302A, …, input encoder K 302K) can be individually different or be all the same. For example, in some embodiments, a same input encoder can be used that embeds the input image x to a visual embedding space that models the visual representation of all natural images with different content and different domain distributions. In some embodiments, different input encoders can be used that embeds the input image x into different visual embedding spaces targeting at the specific class category of images that each basis codebooks ^^304A, ⋯ , ^^304K are learned to represent. Still, in some embodiments, some, butAtty. Docket No.4502-81701 (6000635PCT02) not all, of the K input encoders can be the same to model a shared visual embedding space across several class categories of images.

[0047] In an embodiment, using the k-th processing pipeline for the k-th basis codebook in FIG. 4 as an example, the input encoder K 302K encodes the input x into a feature tensor ^^^with shape ^^× ℎ^× / ^where the width and height ^^and ℎ^depend on the input width and height as well as the network structure of the input encoder K 302K, and where / ^is the number of feature channels. In some embodiments, the width and height ^^and ℎ^are designed to be the same for different k, so that ^^^, ⋯ , ^^^have the same width and height. Various neural networks can be used as the input encoder K 302K such as, but not limited to, a Visual Transformer (ViT), which divides the input image x into patches and encodes them as a sequence. In another embodiment, the Convolutional Neural Network (CNN) structure is used where the entire image is encoded in a parallel manner.

[0048] Code generation module K 306K is configured to compute a sparse codebook-based latent feature ^^^^based on the feature tensor ^^^and the corresponding basis codebook ^^304K. For instance, in an embodiment, the Basis Codebook ^^comprises of 0^codewords, each having / ^dimensions. Each “pixel’’ ^^^^,^in ^^^^(^ = 1, … , ^^× ℎ^) corresponds to a codeword .^^∈ ^^that is nearest to the.^^= 23456789∈ :9;6<=(.^, ^^^,^),

[0049] where ;6<=(∙) is a distance metric, such as L1 or L2 norm. In accordance with the disclosed embodiments, the entire ^^^^has ^^× ℎ^integers corresponding to the indices of ^^× ℎ^codewords, can be efficiently transmitted to the decoder in a lossless manner with very little bit consumption (i.e., using a small number of bits).

[0050] On the receiver side, the received ^^^^is fed into feature retrieval module K 308K to obtain a decoded image embedding feature ^̂^^of shape ^^× ℎ^× / ^based on the same basis codebook ^^304K as the sender. For example, each “pixel” of ^̂^^,^(^ = 1, … , ^^× ℎ^) is assignedAtty. Docket No.4502-81701 (6000635PCT02) with the codeword with index ^^^^,^. As shown in FIG.4, a similar process is performed for each k processing pipeline (i.e., k=1, …, k=K) to generate the corresponding feature tensors ^^^, ⋯ , ^^^, the sparse codebook-based latent features ^^, ⋯ , ^^^^ ^^, and the decoded image embedding features ^̂^^, ⋯ , ^̂^^.

[0051] In an embodiment, based on the feature tensors ^^^, ⋯ , ^^^, the weight predictor module 310 is configured to compute a weight map W of shape ^^× ℎ^× ?, e.g., using a weight prediction method with transformer blocks. In an embodiment, the weight map W guides how the decoded image embedding features ^̂^^, ⋯ , ^̂^^on the receiver side can be weighted combined to compute a target combined latent feature ^̂^∗. The target combined latent feature ^̂^∗utilizes the rich information from all basis codebooks in the adaptive LSIR through the dense weight map W, which gives a target (optimal) reconstructed output image ^^∗. Specifically, in an embodiment, for each “pixel” in ^̂^∗, ^̂^∗,^(^ = 1, … , ^^× ℎ^) is computed as: ^ ^̂^∗,^= @ ^^,^∗ ^̂^^,^

[0052] where ^^,^is the k-th l-th “pixel” in the weight map W. Inaccordance with the disclosed embodiments, because the dense weight map W is too heavy to transmit, an adaptive weight compression module 402 is used to compute a compact weight map ^^that is easy to transmit, based on the weight map W and a target compression rate R. An embodiment of the adaptive weight compression module 402 is described in FIG.5.

[0053] On the receiver side, a weight filling module 404 computes a filled weight map ^^ ^^^^based on the compact weight map ^^and the decoded image embedding features ^̂^^, ⋯ , ^̂^^. The filled weight map ^^ ^^^^is a weight map that fills in missing weights in the compact weight map ^^. There are many ways to implement the weight filling module 404. For example, in some embodiments, weights for masked-out “pixels” are predicted, and weights for unused basis codebooks are predicted. In some embodiments, a neural network model, e.g., a transformer, may be used to implement the weight filling module 404. In some embodiments, one model can be optimized for each masking strategy respectively. In other embodiments, different masking strategies can share a same model. The present disclosure does not place restrictions on the neuralAtty. Docket No.4502-81701 (6000635PCT02) network structure or how it is trained. In an embodiment, a set of new weights are computed by the weight filling module 404 so that the filled weight map ^^ ^^^^can optimally recover / compute the target combined latent feature ^̂^∗. In some embodiments, the weight filling module 404 can be skipped when the filled weight map ^^ ^^^^is the same as the compact weight map ^^.

[0054] As shown in FIG. 4, the weighted combine module 312 is configured to use the filled weight map ^^ ^^^^to compute a combined latent feature ^̂^based on the decoded image embedding features ^̂^^, ⋯ , ^̂^^. In an embodiment, similar to the computation of the target combined feature ^̂^∗, to compute a combined latent feature ^̂^, for each “pixel” in ^̂^, ^̂^,^(^ = 1, … , ^^× ℎ^) is computed as: ^ ^̂^,^= @ ^"^^^^^,^∗ ^̂^^,^

[0055] where ^"^^^^^,^is the k-to l-th “pixel” in the filled weight map ^^ ^^^^. The decoder 314 then computes the reconstructed output image ^^ based on the combined latent feature ^̂^. In some embodiments, the decoder 314 can have a network structure of multiple CNN layers like the decoding network of a Varational Autoencoder (VAE). In other embodiments, the decoder 314 can be a decoder diffusion model such as an image generation model.

[0056] In some embodiments, the general framework for scalable LIC with adaptive LSLR of FIG.4 may also include a compute criterion module 406 that is configured to measure the distortion D between the original input x and the reconstructed output image ^^, and / or between the target combined latent feature ^̂^∗and the combined latent feature ^̂^. As further described in FIG. 5, in some embodiments, the distortion D or loss may be minimized using the adaptive weight compression module 402 to achieve the target compression quality. It should be noted that while FIG. 4 illustrates the compute criterion module 406 as being on the receiver side, the present disclosure does not place restrictions on the location of the compute criterion module 406 or whether the compute criterion module 406 is a standalone module or is implemented as part of another module (e.g., within the adaptive weight compression module 402).

[0057] FIG.5 is a diagram illustrating an adaptive weight compression module 500 according to an embodiment of the present disclosure. The adaptive weight compression module 500 is anAtty. Docket No.4502-81701 (6000635PCT02) example of the adaptive weight compression module 402 in FIG. 4. The adaptive weight compression module 500 includes a mask rate selection module 502 and a masking module 504. As shown in FIG.5, the mask rate selection module 502, given a target compression rate R, is configured to compute a masking rate ^ (may also be referred to as a masking ratio). The masking rate ^ indicates the rate or amount of masking to apply based on the target compression rate R. The masking module 504, given a masking strategy, is configured to generate a weight mask M based on the masking rate ^. An example of a masking strategy may be balancing compression rate and reconstruction performance. The weight mask M is a mask that can be applied to a weight map W to mask out (e.g., set to zero) items in the weight map W to reduce the size of the weight map W. For instance, as shown in FIG.5, the masking module 504 applies the weight mask M to the weight map W (computed by the weight predictor module 310 in FIG.4) to generate the masked version of the weight map as the compact weight map ^^. For example, assume the indices of the k-th basis codebook ^^can be represented by a A^bit integer (e.g., A^= 8 or 10 if ^^has 256 or 1024 codewords). If using the full weight map W to compute the target combined latent feature ^̂^∗, the maximum number of bits that need to transfer is:

[0058] ^^^^= ∑^^#^(^^× ℎ^× A^+ ^^× ℎ^× A^^D^E) ,weight in the weight map W (e.g., A^^D^E=16 or 32 for single or double floating precisions). The weight mask M is usually a binary mask having the same shape ^^× ℎ^× ? as the weight map W. In an embodiment, the masked-out weight items in W by the mask M are set to zero. That is, the resulting compact weight map ^^is a sparse weight map that is much more lightweight to transfer than the original weight map W. There are many ways to compute the mask M. In an embodiment, for each “pixel” ^̂^,^( ^ = 1, … , ^^× ℎ^), the pixel is either completely masked-out where the corresponding items 5^,^, F = 1, … , ? in the mask M are set to zero, or the pixel is computed using codewords from a number of J basis codebooks, G = 1, … , ?. For instance, if a pixel is not masked-out, ^̂^,^can be computed as:Atty. Docket No.4502-81701 (6000635PCT02)

[0060] M^^, … , M^Kdenote the index of the basis codebook to use for ^̂^,^if it is not masked-out, 5^,^= 1 for F = M^L(N = 1, … , G), and 5^,^= 0 otherwise.

[0061] If there is not a completely masked-out “pixel”, the number of bits that need to transferred is: ^^^^^= ^^× ℎ^× AH+∑KL#^(^^× ℎ^× AHIJ+ ^^× ℎ^× A^^D^E) , for G > 1

[0062] AH= 3 or 4 if there are 8 or 16 basis codebooks.

[0063] When there are completely masked-out “pixels”, the corresponding items will be removed from the bit count to transfer as shown in the following example: ^^^^^=∑V^#W×X^WQ^[AH+∑KL#^SAHIJ+ A^^D^ET ] , for G > 1

[0064] where Q^= 0 or 1 if the “pixel” is completely masked-out or not.

[0065] As stated above, given a target compression rate R, the mask rate selection module 502 computes a masking rate ^. In an embodiment, the masking rate ^ can be computed as the ratio of completely masked-out “pixel” against all pixels: ^ = ∑V^#W×X^WQ^ / (^^× ℎ^)

[0066] For the given target compression rate R, a corresponding target bit count ^^^^^can be computed. For example, ^^^^^= 3 × ^^^^. A corresponding masking strategy (i.e., the number of J basis codebooks for each “pixel” and the ratio of completely masked-out “pixels”) can then be determined. For example, the masking strategy that produces the compression rate closest to the target compression rate R can be used. Alternatively, a set of masking strategies that producesAtty. Docket No.4502-81701 (6000635PCT02) compression rates close to the target compression rate R can be determined as candidates, and the optimal strategy that produces the balanced compression rate and reconstruction performance can be used.

[0067] As stated above, given a masking strategy, the masking module 504 computes the mask M and the compact weight map ^^. In one embodiment, for each “pixel” ^̂^,^(^ = 1, … , ^^× ℎ^), the basis codebooks that have the maximum values of corresponding weights of that “pixel” in the weight map W are selected to be used. In some embodiments, a uniform “pixel” sampling is performed to mask-out “pixels” according to the masking ratio ^. In other embodiments, a prediction neural network model can be used to predict the position of the masked-out “pixels” and compute the weights for the non-masked-out “pixels”, based on the masking ratio ^ and the weight map W, so that the resulting compact weight map ^^can better recover the target combined latent feature ^̂^∗as described in FIG.4. In some embodiments, the prediction neural network model can be trained in a way that one model is optimized for each masking strategy respectively, or in a way that all different masking strategies share a same model. The present disclosure does not place restrictions on the neural network structure or how it is trained.

[0068] In some embodiments, the weight mask M and the compact weight map ^^is transmitted to the receiver side directly. As described in FIG.4, on the receiver side, the weight filling module 404 is configured to compute a filled weight map ^^ ^^^^based on the compact weight map ^^and the decoded image embedding features ^̂^^, ⋯ , ^̂^^, which is then used by the weighted combine module 312 to compute the combined latent feature ^̂^. The decoder 314 then computes the reconstructed output image ^^ based on the combined latent feature ^̂^.

[0069] Further, as shown in FIG.4 and FIG.5, in some embodiments, distortion loss D can be computed in the compute criterion module 406 to measure the distortion between the original input x and the reconstructed output image ^^, and / or between the target combined latent feature ^̂^∗and the combined latent feature ^̂^, such as Mean Square Error (MSE). In some embodiments, this loss may be minimized to achieve the target compression quality. There are different ways to minimize this loss. For example, in some embodiments, given a masking strategy with the mask M, the weightsAtty. Docket No.4502-81701 (6000635PCT02) corresponding to the non-masked-out “pixels” in the compact weight map ^^can be updated by backpropagating the gradient of the loss to these weight items, while maintaining the masking strategy (i.e., the mask M and the number of codebooks for each unmasked “pixel”) unchanged. Another option is to compute the loss for various candidate masking strategies with the already computed mask M and compact weight map ^^, without updating ^^using backpropagation, and simply select the optimal strategy with the minimum loss. After this, the updated weight mask M and the updated compact weight map ^^are sent to the receiver side. The present disclosure does not place restrictions on how to compute the loss in the compute criterion module 406 or how to use the loss to update the mask M and the compact weight map ^^.

[0070] FIG.6 is a flowchart illustrating a processing workflow 600 according to an embodiment of the present disclosure. The processing workflow 600 is an example of a method performed of a sender or a sending device according to an embodiment of the present disclosure. The sending device, at step 602, encodes an input image x by K input encoders of the sending device (e.g., input encoder 1 302A, …, input encoder K 302K of FIG. 4) to generate K image feature tensors ^^^, ⋯ , ^^^, wherein K is an integer greater than 1. The sending device, at step 604, computes, by one or more processors of the sending device, K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^respectively based on the K image feature tensor ^^^, ⋯ , ^^^and corresponding basis codebook ^^, ⋯ , ^^(e.g., by executing the instructions associated with the code generation module 1306A, …, code generation module K 306K and the corresponding basis codebooks ^^304A, ⋯ , ^^304K as described in FIG.4). The sending device, using the one or more processors of the sending device, then computes, at step 606 (e.g., by executing the instructions associated with the weight predictor module 310 in FIG.4), a weight map W that provides information for weighted combining K decoded image embedding features ^̂^^, ⋯ , ^̂^^, wherein the K decoded image embedding features ^̂^^, ⋯ , ^̂^^is respectively based on the K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^and the corresponding basis codebook ^^, ⋯ , ^^. At step 608, the sending device, using the one or more processors of the sending device, computes (e.g., by executing the instructions associated with the adaptive weight compression module 402 in FIG.4) a compact weight map ^^based on the weightAtty. Docket No.4502-81701 (6000635PCT02) map W and a target compression rate R. At step 610, the sending device transmits (e.g., by a transmitter or network communication device or module of the sending device) the compact weight map ^^and the K sparse codebook-based latent features ^^^^, ⋯ , ^^^^to a receiving device. Additional details of the particular steps of the processing workflow 600 are described above in the present disclosure.

[0071] FIG.7 is a flowchart illustrating a processing workflow 700 according to an embodiment of the present disclosure. The processing workflow 700 is an example of a method performed of a receiver or a receiving device according to an embodiment of the present disclosure. The receiving device, at step 702, receives (e.g., by a transceiver or network communication device or module of the receiving device) K sparse codebook-based latent features ^^^^, ⋯ , ^^^^and a compact weight map ^^from a sender or sending device, wherein K is anthan 1. As described above, the K sparse codebook-based latent features ^^^^, ⋯ , ^^^^were generated by the sender based on K feature tensor ^^^and K corresponding basis codebook. At step 704, the receiving device computes, by one or more processors of the receiving device, K decoded image embedding features ^̂^^, ⋯ , ^̂^^based on the K sparse codebook-based latent features ^^^^, ⋯ , ^^^^using the corresponding K basis codebook (e.g., by executing the instructions associated with the feature retrieval module 1308A, …, feature retrieval module K 308K and the corresponding basis codebooks ^^304A, ⋯ , ^^304K as described in FIG. 4) . The receiving device, at step 706, computes, using the one or more processors of the receiving device, a filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^(e.g., by executing the instructions associated with weight filling module 404 in FIG.4). The receiving device then computes, at step 708, , using the one or more processors of the receiving device, a combined latent feature ^̂^based on the filled weight map ^^ ^^^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^(e.g., by executing the instructions associated with the weighted combined module 312 in FIG.4). At step 710, the receiving device computes, using the one or more processors of the receiving device, a reconstructed output image ^^ based on the combined latent feature ^̂^(e.g., by executing theAtty. Docket No.4502-81701 (6000635PCT02) instructions associated with the decoder 314 in FIG.4). Additional details of the particular steps of the processing workflow 700 are described above in the present disclosure.

[0072] FIG. 8 is a diagram illustrating an apparatus 800 according to an embodiment of the present disclosure. The apparatus 800 can be used to implement embodiments of the present disclosure. For example, the apparatus 800 may be configured to perform the functions of a sender or sending device and a receiver or receiving device according to any of the embodiments of the present disclosure. The apparatus 800 includes receiver units (RX) 820 or receiving means for receiving data via ingress ports 810. The apparatus 800 also includes transmitter units (TX) 840 or transmitting means for transmitting via data egress ports 850. For example, on the sender side, the sending device may use the RX 820 or receiving means to obtain an original image and then use the TX 840 or transmitting means for transmitting a compact weight map ^^and K sparse codebook- based latent features ^^, ⋯ ,^^^^^^to the receiver sider as described in FIG.4. On the receiver side, the decoder may use the RX 820 or receiving means to receive the compact weight map ^^and K sparse codebook-based latent features ^^^^, ⋯ , ^^^^, and then use the TX 840 or transmitting means for transmitting the decoded imageimage (e.g., the reconstructed output image ^^) to a display device or to another computing device.

[0073] The apparatus 800 includes a memory 860 or data storing means for storing the instructions and various data. The memory 860 can be any type of, or combination of, memory components capable of storing data and / or instructions. For example, the memory 860 can include volatile and / or non-volatile memory such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 860 can also include one or more disks, tape drives, and solid-state drives. In some embodiments, the memory 860 can be used as an over-flow data storage device to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. In some embodiments, the memory 860 can be memory that is integrated with the processor 830.Atty. Docket No.4502-81701 (6000635PCT02)

[0074] The apparatus 800 has one or more processors 830 or other processing means (e.g., central processing unit (CPU)) to process instructions. The one or more processors 830 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 830 are communicatively coupled via a system bus with the ingress ports 810, RX 820, TX 840, egress ports 850, and memory 860. The one or more processors 830 can be configured to execute instructions stored in the memory 860. As an example, in one embodiment, the memory 860 stores a scalable LIC with adaptive LSLR module 870. The scalable LIC with adaptive LSLR module 870 includes data, executable instructions, and / or one more sub- modules for implementing the disclosed embodiments. Thus, the one or more processors 830 provide a means for performing any computational, comparison, determination, initiation, configuration, or any other action corresponding to the claims when the appropriate instruction is executed by the processor 830. Thus, the inclusion of the scalable LIC with adaptive LSLR module 870 substantially improves the functionality of the apparatus 800.

[0075] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the disclosure is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0076] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, andAtty. Docket No.4502-81701 (6000635PCT02) alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

Atty. Docket No.4502-81701 (6000635PCT02) CLAIMS What is claimed is:

1. A method implemented by a sending device, the method comprising: encoding an input image x by K input encoders of the sending device to generate K image feature tensors ^^^, ⋯ , ^^^, wherein K is an integer greater than 1; computing K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^respectively based on the K image feature tensor ^^^, ⋯ , ^^^and corresponding basis codebook ^^, ⋯ , ^^; computing a weight map W that provides information for weighted combining K decoded image embedding features ^̂^^, ⋯ , ^̂^^, wherein the K decoded image embedding features ^̂^^, ⋯ , ^̂^^is respectively based on the K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^and the corresponding basis codebook ^^, ⋯ , ^^; computing a compact weight map ^^based on the weight map W and a target compression rate R; and transmitting the compact weight map ^^and the K sparse codebook-based latent features ^^^^, ⋯ , ^^^^to a receiving device.

2. The method according to claim 1, wherein computing the compact weight map ^^based on the weight map W and the target compression rate R comprises: computing a masking rate ^ based on the target compression rate R; generating a weight mask M based on the masking rate ^; and computing the compact weight map ^^based on the weight map W and the weight mask M.

3. The method according to claim 2, wherein the masking rate τ is a ratio of masked-out weights against all weights.

4. The method according to any of claims 2-3, further comprising generating the weight maskAtty. Docket No.4502-81701 (6000635PCT02) M based on the masking rate ^ and a masking strategy.

5. The method according to any of claims 2-4, further comprising: computing a target bit count ^^^^^based on the target compression rate R; and determining the masking strategy based on the target bit count ^^^^^.

6. The method according to any of claims 2-5, further comprising transmitting the weight mask M to the receiver.

7. The method according to any of claims 1-6, further comprising: computing a distortion loss D measuring a distortion between the input image x and a reconstructed output image ^^, and / or between a target combined latent feature ^̂^∗and a combined latent feature ^̂^, and modifying weights corresponding to non-masked-out pixels in the compact weight map ^^to reduce the distortion loss D.

8. The method according to any of claims 1-7, further comprising: computing, based on a plurality of candidate masking strategies, a distortion loss D measuring a distortion between the input image x and a reconstructed output image ^^ , and / or between a target combined latent feature ^̂^∗and a combined latent feature ^̂^, and selecting a masking strategy from the plurality of candidate masking strategies having a minimum distortion loss D.

9. A method implemented by a receiving device, the method comprising: receiving K sparse codebook-based latent features ^^ ^^^, ⋯ , ^^^and a compact weight map ^^from a sending device, wherein K is an integer greater than 1; computing K decoded image embedding features ^̂^^, ⋯ , ^̂^^based on the K sparseAtty. Docket No.4502-81701 (6000635PCT02) codebook-based latent features ^^ ^^^, ⋯ , ^^^; computing a filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^; computing a combined latent feature ^̂^based on the filled weight map ^^ ^^^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^; and computing a reconstructed output image ^^ based on the combined latent feature ^̂^.

10. The method according to claim 9, wherein computing the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^comprises predicting weights for masked-out weights in the compact weight map ^^.

11. The method according to any of claims 9-10, wherein computing the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^comprises predicting weights for unused basis codebooks in computing the K decoded image embedding features ^̂^^, ⋯ , ^̂^^.

12. The method according to any of claims 9-10, wherein computing the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^computing a set of new weights for recovering a target combined latent feature ^̂^∗.

13. The method according to any of claims 9-12, further comprising computing, by a neural network model, the filled weight map ^^ ^^^^based on the compact weight map ^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^.

14. The method according to claim 13, wherein the neural network model is optimized for a masking strategy corresponding to the compact weight map ^^.Atty. Docket No.4502-81701 (6000635PCT02) 15. The method according to any of claims 9-14, wherein computing the combined latent feature ^̂^based on the filled weight map ^^ ^^^^and the K decoded image embedding features ^̂^^, ⋯ , ^̂^^comprises computing ^̂^,^(^ = 1, … , ^^× ℎ^), for each pixel in ^̂^as follows ^ ^̂^,^= @ ^"^^^^^,^∗ ^̂^^,^^#^where ^"^^^^^,^is a k-th weight corresponding to l-th pixel in the filled weight map ^^ ^^^^.

16. An apparatus comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the apparatus to perform a method according to any of claims 1-8.

17. An apparatus comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the apparatus to perform a method according to any of claims 9-15.

18. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processors of an apparatus, cause the apparatus to perform a method according to any of claims 1-15.

Images