Neural network image compression using representation adaptive VQ encoder and base encoder

Abstract

A vector quantization (VQ) neural network-based image compression method includes encoding, by a first encoder, a first image to obtain a first latent feature corresponding to the first image; generating, based on the first latent feature and using a leading codebook, a leading codebook indices map and a first codeword map corresponding to the leading codebook indices map, wherein the leading codebook comprises codebook indices that are assigned to identical vectors in multiple codebooks; encoding the leading codebook indices map to generate an encoded leading codebook indices map; and transmitting the encoded leading codebook indices map to a decoder.

Description

NEURAL NETWORK IMAGE COMPRESSION USING REPRESENTATION ADAPTIVE VQ ENCODER AND BASE ENCODER CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Provisional Application No. 63 / 574,332 filed on April 4, 2024 and Provisional Application No. 63 / 574,334 filed on April 4, 2024. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.TECHNICAL FIELD

[0002] This disclosure is related to neural network-based image or video compression, and more specifically to neural network image / video compression using a representation adaptive vector quantization (VQ) encoder and a base encoder.BACKGROUND

[0003] With the rapid progress of deep learning research, the development of deep neural network-based image compression (NNIC) using state-of-the-art deep learning techniques has shown promising results achieving efficient image encoding and decoding. Unlike traditional image compression algorithms like Joint Photographic Experts Group (JPEG), High Efficiency Video Coding (HEVC), or Versatile Video Coding (VVC), which rely on handcrafted transformations (e.g., discrete cosine transform in JPEG), NNIC methods learn efficient representations of images through training on large datasets.SUMMARY

[0004] A first aspect relates to a VQ neural network-based image compression method, implemented by an encoding device. The method includes encoding, using a first encoder of the encoding device, a first image to obtain a first latent feature corresponding to the first image; generating, based on the first latent feature and using a leading codebook, a leading codebook indices map and a first codeword map corresponding to the leading codebook indices map, wherein the leading codebook comprises codebook indices that are assigned to identical vectors in multiple codebooks; encoding the leading codebook indices map to generate an encoded leading codebook indices map; and transmitting the encoded leading codebook indices map to a decoder.

[0005] Optionally, in a first implementation according to the first aspect, the method further includes encoding, using a second encoder of the encoding device, a second image to obtain a second latent feature and a hyper tensor corresponding to the second image; and generating, based on the hyper tensor and using a base codebook, a base codebook indices map, wherein encoding the leading codebook indices map to generate the encoded leading codebook indices map comprises using the base codebook indices map as a reference.

[0006] Optionally, in a second implementation according to the first aspect or any implementation thereof, the base codebook is trained to create the base codebook indices map to resemble the leading codebook indices map.

[0007] Optionally, in a third implementation according to the first aspect or any implementation thereof, the base codebook is trained to create the base codebook indices map to resemble the leading codebook indices map.

[0008] Optionally, in a fourth implementation according to the first aspect or any implementation thereof, wherein generating, based on the hyper tensor and using the base codebook, the base codebook indices map comprises: quantizing and entropy encoding the hyper tensor to obtained an encoded hyper tensor; decoding the encoded hyper tensor to obtained a reconstructed hyper tensor; performing tensor prediction using the reconstructed hyper tensor to obtain a predicted tensor of a first tensor; quantizing and entropy encoding a difference between first tensor and the predicted tensor; performing entropy decoding and tensor prediction to generate a reconstructed first tensor; and computing, using the base codebook, the base codebook indices map based on the reconstructed first tensor.

[0009] Optionally, in a fifth implementation according to the first aspect or any implementation thereof, the second image is a lower-quality image of the first image.

[0010] Optionally, in a sixth implementation according to the first aspect or any implementation thereof, the second image is the same as the first image.

[0011] Optionally, in a seventh implementation according to the first aspect or any implementation thereof, the codewords in the leading codebook or the base codebook are reorderedin descending frequency order, so that less bits are assigned to more frequently used indices to reduce a total bitrate.

[0012] Optionally, in an eighth implementation according to the first aspect or any implementation thereof, the codewords in the leading codebook or the base codebook are generated by a channel-weighted codeword generation method that accounts for a difference of importance factor among channels.

[0013] Optionally, in a tenth implementation according to the first aspect or any implementation thereof, the codewords in the leading codebook or the base codebook are generated by a codeword generation method that accounts for rate factor.

[0014] Optionally, in a tenth implementation according to the first aspect or any implementation thereof, the method further includes performing a location matching method to minimize a difference between the leading codebook indices map and the base codebook indices map.

[0015] Optionally, in an eleventh implementation according to the first aspect or any implementation thereof, the method further includes training the base codebook using a contrastive classification training method.

[0016] Optionally, in a twelfth implementation according to the first aspect or any implementation thereof, the method further includes performing masking and reconstruction to further minimize information transmission of the encoded leading codebook indices map.

[0017] Optionally, in a thirteenth implementation according to the first aspect or any implementation thereof, the masking applies a mask ratio based on an image complexity.

[0018] A second aspect relates to VQ neural network-based image compression method, implemented by a decoding device. The method includes receiving an encoded leading codebook indices map, wherein the encoded leading codebook indices map comprises indices that are mapped to identical vectors in multiple codebooks; generating, based on the encoded leading codebook indices map, N latent feature map by retrieving codeword from N codebooks using respective codebook indices in the encoded leading codebook indices map; combining the N latentfeature map with anN-channel weight map to generate a reconstructed latent feature; and decoding the reconstructed latent feature to obtain a reconstructed image.

[0019] Optionally, in a first implementation according to the second aspect, the method further includes generating control signals to guide a conditioned generative reconstruction process in decoding the reconstructed latent feature to obtain the reconstructed image.

[0020] A third aspect relates to an apparatus comprising a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the apparatus to perform the method according to any of the preceding aspects or any implementation thereof.

[0021] Afourth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computerexecutable instructions when executed by one or more processors of an apparatus, cause the apparatus to perform a method according to any of the preceding aspects or any implementation thereof.

[0022] For clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0023] These and other features, and the advantages thereof, will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF DRAWINGS

[0024] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0025] FIG. 1 is a diagram illustrating a single codebook or multiple codebooks VQ-based encoder framework in accordance with an embodiment of the present disclosure.

[0026] FIGS. 2A-2C are diagrams illustrating a single codebook, a multiple codebook, and a leading codebook in accordance with an embodiment of the present disclosure.

[0027] FIG. 3 is a diagram illustrating a leading codebook VQ-based encoder framework in accordance with an embodiment of the present disclosure.

[0028] FIG. 4 is a diagram illustrating a base encoder framework in accordance with an embodiment of the present disclosure.

[0029] FIG. 5 is a diagram illustrating an NNIC system framework in accordance with an embodiment of the present disclosure.

[0030] FIG. 6 is a diagram illustrating a masked generative transformer (MAGE) Prediction in accordance with an embodiment of the present disclosure.

[0031] FIG. 7 is a flowchart diagram illustrating a VQ neural network-based image compression method implemented by an encoding device in accordance with an embodiment of the present disclosure.

[0032] FIG. 8 is a flowchart diagram illustrating a VQ neural network-based image compression method implemented by a decoding device in accordance with an embodiment of the present disclosure.

[0033] FIG. 9 is a diagram illustrating a system framework in accordance with an embodiment of the present disclosure.DESCRIPTION OF EMBODIMENTS

[0034] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0035] The present disclosure describes various embodiments of systems and methods to improve image / video coding efficiency by using a two-branch NNIC framework incorporating a representation adaptive (RED)-vector quantization (VQ) encoder using a masked generative encoder (MAGE) for masked codebook indices prediction and a base encoder. The disclosed embodiments are applicable to both neural network-based image compression and neural networkbased video compression. In an embodiment, the RED-VQ encoder utilizes a high-quality (HQ) visual representation targeting for generating an image with high perceptual quality, and the base encoder provides fidelity-preserving information to guide the conditioned image generation in the RED-VQ encoder. By exploiting the power of high-quality generative visual representation prior, the efficiency and robustness of transmitting integers, and the guided image generation with fidelity-preserving controls, the disclosed embodiments achieve high compression rate and high reconstruction fidelity and perceptual quality at the same time.

[0036] In the context of image compression, some autoregressive scale hyperprior NNIC frameworks use a Variational Autoencoder (VAE) as a main encoder to explore the latent information of input image, a hyperprior model as a hyper encoder to explore additional hyper latent information, an autoregressive model as a context modelling to explore the spatial relationship among neighbor latent coefficients, and a Gaussian mixture model (GMM) or Gaussian scale mixture (GSM) to generate the mean and scale associated with each latent coefficients. In general, on a sender side, an input image x is passed through a VAE encoder to generate an image embedding feature y, which is further compressed through quantization and entropy coding into a data string that is efficient for storage and transmission. Quantization or quantizing refers to the process of mapping a continuous range of values to a smaller, discrete set of values. Entropy coding is a lossless data compression technique that assigns variable-length codes to symbols based on their probability of occurrence. The goal of entropy coding is to represent data more efficiently by using shorter codes for more frequent symbols and longer codes for less frequent ones.

[0037] On the receiver side, a decoded image embedding feature y is recovered from the received string through entropy decoding and dequantization. Then a VAE decoder reconstructs an output image x based on the decoded image embedding feature y. The target is to minimize the restoration loss between the reconstructed output x and the original input x, and to minimize the bits to represent the image embedding feature y for storage and transmission.

[0038] The current NNIC framework relies on learning a general compact image representation, i.e., a latent space where the image embedding feature y can capture the gist of the input x to reconstruct x. This framework has several severe limitations. First, the compression performance is innately bounded by the model capacity (e g., the network structure and number of parameters of the input encoder and the decoder) in learning the general prior P(x|y) in the image domain. Due to the limited model capacity, limited training data, and limited computation resources in both training and test stage, it is difficult to further improve the compression performance beyond a good baseline. Second, the NNIC models are learned to balance the competing goals in the rate-distortion (RD) loss, where reducing reconstruction distortion and reducing bitrate are contradicts with each other. Thus, due to the difficulty in balancing different loss terms in end-to-end training, it is difficult to improve the compression performance and the perceptual quality at the same time.

[0039] In comparison, VQ-based compression methods learn one or multiple highly compressed visual codebooks Ci (1 < i < K) by a vector-quantized autoencoder that is trained in a way that each codebook targets at a semantic category of images like outdoor, portrait, etc. VQ is a process used to map vectors from a large set (such as high-dimensional data like images) to a finite set of representative vectors. In general, an input image or video frame can be encoded into the latent space as an embedding feature y that captures the content and characteristics of the input image. The embedding feature y is then mapped into a sequence of codeword integer indices zi (1 < i < K), using the learned codebooks Ci (1 < i < K), respectively. The codeword integer indices is then transferred to receiver for decoding. The decoder uses the same codebooks to recover a set of features yi (1 < i < K) by using codewords in codebook Ci (1 < i < K)corresponding to the codeword indices zi (1 < i < K). Then a reconstructed feature y is generated by applying weighted sum to the set of features yi (1 < i < K). Finally, an output x can be reconstructed based on the reconstructed feature y.

[0040] By encoding codeword indices instead of latent features, VQ-based compression does not suffer from the sensitivity of computation mismatch between the sender and receivers. Transferring indices allows for the expansion of a latent feature dimension (often associated with better representation power for better reconstruction) without increasing bitrate, in comparison to transferring latent features or residues. Additionally, generative VQ increases robustness to input degradations. Realistic and rich textures can be generated using high-quality (HQ) codebooks even for low-quality (LQ) inputs. However, one issue with VQ-based compression methods is the difficulty in efficient bit-rate reduction where both multiple codebook indices maps, and their corresponding weights used in weighted sum need to be encoded in the bitstream. Another issue of VQ-based compression methods is the difficulty in HQ reconstruction, although the restored images may look good perceptually, important fidelity and rich details are usually lost in the reconstruction result, especially when the codebooks are trained using generative adversarial network (GAN).

[0041] Accordingly, the present disclosure describes various embodiments of VQ-based compression systems and methods that improve image / video coding efficiency to address one or more of the above issues.

[0042] FIG. 1 is a diagram illustrating a single codebook or multiple codebooks VQ-based encoder framework 100 in accordance with an embodiment of the present disclosure. The VQ- based encoder framework 100 includes a VQ encoder 102, one or more representation (REP) adaptors 110, a feature compressor 112, a feature merger 114, and a decoder 116. In general, an input image x is passed through the VQ encoder 102, which converts the input image x to an embedding feature y. In an embodiment, the input image x is a general 3-dimensional (3D) tensor with shape w x h x c, where w, h, and c are the width, height, and number of channels of the image,respectively. A tensor is a multi-dimensional array greater than 2 dimensions. For example, c=3 for color images, c=l for spectral images, or c=4 for RGB-D (color and depth) images.

[0043] In an embodiment, the VQ encoder 102 is a VAE encoder or model that has been trained to encode the image x into a discrete latent representation using one or multiple highly compressed visual codebooks Ci (1 < i < K), where each codebook targets at a semantic category of images like outdoor, portrait, etc. The embedding feature y represents a discrete, structured representation of the input image x, where each feature vector corresponding to a region or characteristic of the image x is replaced by the closest learned embedding from a visual codebook.

[0044] The embedding feature y is passed to one or more REP adaptors 110. A REP adaptor 110 include may include (1 to N) before quant RED (BFR)s 104, (1 to N) RED codebooks 106, and (1 to N) afler_quant_RED (AFT)s 108, where N is the number of codebooks contained in the REP adaptor 110. In an embodiment, the before quant RED 104 is configured to convert the latent feature y to N new latent feature spaces or subspaces corresponding to the N codebooks. The RED codebook 106 is configured to process the latent feature y and generate N codeword index maps by retrieving the codeword from N REP adaptors 110 using the respective codebook indices. A codeword index map stores indices that point to the corresponding entries in a codebook. Whereas, a codeword map contains the actual codewords rather than their indices. The after quant RED 108 further converts the N codeword index maps to N new codebook-based latent feature vqs.

[0045] In an embodiment, when a single codebook is used, the RED codebook 106 computes a codebook indices map zqand a codebook-based feature yqbased on the latent feature y and a Codebook C. The codebook-based feature yqindicates the codebook features corresponding to embedding features y of the image x. The codebook indices map zqstores indices that point to the corresponding codebook feature entries in the Codebook C. Specifically, the Codebook C comprises of N codewords, each having d dimensions. Each VQ vector yq(i = 1, . . . , width (m) x height ( )) in the codebook-based feature yqcorresponds to a codeword cql(i = 1, . . . , w x A) in cq, where cqG C that is nearest to the corresponding feature vector j,:

[0047] where Dist(-) is a distance metric, such as LI or L2 norm. That is, the entire codebook indices map zqhas w x h integers corresponding to the indices of w x h codewords.

[0048] In an embodiment, because there is only one codebook indices map zqand one codebook-based latent feature yqin the system, the feature compressor 112 is not required because the no weight map W is needed.

[0049] In an embodiment, when multiple (N) codebooks are used, the RED codebook 106 is configured to compute N codebook indices maps zq,nand N codebook-based latent feature yq.n(vqs) of shape w x h x dx N based on the embedding feature y and N Codebooks Cn(n=l, . . ., N). In these embodiments, the feature compressor 112 is configured to process the latent feature y and output a weight map W. In an embodiment, the weight map ffl has shape w x h x N, where w and h are identical to that of latent feature y, and N is identical to the number of before quant RED 104 / after_quant_RED 108 in the REP adaptor 110.

[0050] The feature merger 114 receives the weight map W and the N codebook-based latent feature vqs. The weight map IE guides / indicates how the N codebook-based latent feature vqs can be weighted combined to generate a combined latent feature y of shape w x h x d, which is passed to the decoder 116. d is the depth or number of feature channels or filters. The combined latent feature y utilizes the rich information from all codebooks in the system through the dense weight map W, which the decoder 116 uses to generate an optimal reconstructed output image x. The feature merger 114 can use weighted sum operation to combine two input signals, or it can use any other network architecture to combine two input signals. As a non-limiting example, in an embodiment, for each vector in y^ (k = 1 , . . . , w x h), the decoder 116 is configured to compute as:

[0051] y^T= wkTX vqsk

[0052] where wkis the kthvector of shape (N, 1) in the weight map W, and vqskis the kthtensor of shape (N, d) in the N codebook-based latent feature vqs.

[0053] In another embodiment, ykcan also be computed as:

[0054] yk= vqskTX wk

[0055] where the role of ffl and vqs are swapped where vqs is served as a weight map and W is served as a highly compressed latent feature.

[0056] FIGS. 2A-2C are diagrams illustrating a single codebook, a multiple codebook, and a leading codebook in accordance with an embodiment of the present disclosure. FIG. 2A illustrates the mapping of a feature vector to a single codebook. As described above, a codebook is a predefined set of representative vectors (also called codewords) in a high-dimensional space that are intended to represent the embedding features y of the image x. Each vector in a codebook is associated with an index value. The index value is used to reduce storage or transmission bits. For instance, instead of storing or encoding the full input vector, the system assigns the index value of the closest codebook vector to input vector and stores / encodes the just the index value. This reduces storage and computational complexity while enabling approximate representation.

[0057] In accordance with the disclosed embodiments, because of the way a codebook is trained and used, the coverage of the indices in a single codebook are not overlapped so that for an input vector, the input vector will be assigned to one and only one corresponding codework or representative vector in the codebook and the index associated with that representative vector. For instance, when an input vector is received, the input vector is compared to all the codebook vectors, and the closest one (e.g., based on some distance metric, often Euclidean distance) is selected. For example, as shown in FIG. 2A, for an input vector and using only a single codebook X, the input vector can only be assigned to one and only one representative vector in the codebook X (e.g., either the representative vector associated with index A, the representative vector associated with index B, or the representative vector associated with index C). In the depicted example, the input vector is assigned to the representative vector associated with index A as being the closest representative vector to the input vector.

[0058] However, when multiple codebooks are used (e.g., codebook X, codebook Y, and codebook Z as shown in FIG. 2B), the same input vector will be assigned to multiple indices, one for each codebook. For example, as shown in FIG. 2B, the indices covered by each of thecodebook from the multiple codebooks may overlap with each other as indicated by the bolded area in FIG. 2B. Moreover, it can be observed that all vectors inside this overlapped space are assigned to identical indices in each respective codebook. Based on this observation, the present disclosure introduces a new leading codebook, which is defined as a codebook containing codebook indices that are assigned to some or all overlapping space of multiple codebooks. For example, FIG. 2C illustrates a representative vector that is in a leading codebook.

[0059] FIG. 3 is a diagram illustrating a leading codebook VQ-based encoder framework 300 in accordance with an embodiment of the present disclosure. Similar to the VQ-based encoder framework 100 in FIG. 1, the leading codebook VQ-based encoder framework 300 includes the VQ encoder 102, one or more REP adaptors 110, the feature compressor 112, the feature merger 114, and the decoder 116 as described in FIG. 1. In contrast to the VQ-based encoder framework 100 in FIG. 1, the leading codebook VQ-based encoder framework 300 may include a leading codebook 320.

[0060] As described above, an input image x is passed through the VQ encoder 102, which converts the input image x to an embedding feature y. In an embodiment, the VQ encoder 102 is an HQ encoder. In an embodiment, the embedding feature y has shape w x h x d. where w and h depend on the input width and height as well as the network structure of the VQ encoder 102, and d is the number of feature channels.

[0061] In this embodiment, the embedding feature y is passed to the leading codebook 320. The leading codebook 320 comprises a before_quant_leading 322, a leading codebook 324, and an after quant leading 326. The before quant leading 322 converts latent feature y to a new latent feature space. The leading codebook 320 then processes the latent feature y and generates a leading codebook indices map hq idx and a corresponding codeword map by retrieving the codeword from the leading codebook 324 using the generated codebook indices. As described above, the leading codebook 324 is a codebook containing codebook indices that are assigned to some or all overlapping space of multiple codebooks. The after quant leading 326 further converts / encodes the leading codebook indices map hq idx or, as further described below, areconstructed leading codebook indices map hq_idx, to a discrete latent space to obtain a new latent feature vqO, which is transmitted to a receiver or decoder side.

[0062] On a decoder or receiver side, the latent feature vqO is by provided as input to one or more REP adaptors 110. As described in FIG. 1, the REP adaptor 110 includes (1 to N) before_quant_RED 104, (1 to N) RED codebooks 106, and (1 to N) after_quant_RED 108, where N is the number of codebooks contained in the REP adaptor 110. In an embodiment, for a REP adaptor 110 that contains N codebooks, N before_quant_red 104 are used to convert the latent feature vqO to N discrete latent spaces. N RED codebooks 106 process this latent feature and generate N codeword indices maps using the respective codebook indices. Then, N after_quant_RED 108 further converts the N codeword indices maps to N new latent feature vqs.

[0063] In some embodiments, because the leading codebook 320 was used to generate the leading codebook indices map hq Jdx for the N codebooks, for a REP adaptor 110 that contains N codebooks, the N RED codebooks 106 can be removed as indicated by the dash lines in FIG. 3. For example, in an embodiment, the N codeword indices maps generation and lookup can be bypassed by assigning the respective input to each of the N RED codebooks 106 directly to the output of the respective RED codebook 106, which effectively removes the N RED codebooks 106 from the REP adaptor 110. In these embodiments, the before quant RED 104 and the after quant RED 108 can map the discrete latent leading space to N discrete latent subspaces where the characteristic of each discrete latent subspace is based on the distribution of each channel of an output of the feature compressor 112, which enables the before quant RED 104 and the after quant RED 108 to adapt to more diversified latent representation spaces. In these embodiments, the role of the REP adaptor 110 and the feature compressor 112 are swapped. For instance, the feature compressor 112 can be regarded as a highly efficient feature compression because the feature compressor 112 outputs a highly compressed latent feature with shape w x h x N that preserves N-channel of the most fundamental information of its input, and the REP adaptor 110 can be regarded as a feature compressor / weight prediction because the REP adaptor 110outputs a weight map with shape w x h x d x N for the highly compressed latent feature generated from the feature compressor 112.

[0064] In an embodiment, the leading codebook 320 can be considered as a step to convert input latent feature to a discrete latent space and the N REP adaptor 110 further maps the discrete latent leading space to N discrete latent subspaces. In an embodiment, if the N RED codebooks 106 are not removed from the N REP adaptor 110, the characteristic of each subspace is based on the category and distribution of datasets used to train individual RED codebooks 106. However, if the N RED codebooks 106 are removed from the N REP adaptor 110, the characteristic of each subspace is based on the distribution of each channel of an output of the feature compressor 112 as stated above.

[0065] In some embodiments, the remaining process in FIG. 3 for generating the reconstructed output image x is as described in FIG. 1. Alternatively, as shown in FIG. 3, in some embodiments, the latent feature vqO generated by the leading codebook 320 and the latent feature vqs can also be concatenated together, at concatenator 318, which increases the number of codeword indices maps from N to N+l . In these embodiments, the feature merger 114 outputs a weight map with shape w x h x (N+l). Still, in some embodiments, the latent feature vqO can be used as a residual skip connection and added to each of the N latent features to form the final latent feature vqs.

[0066] By utilizing a leading codebook as described in FIG. 3, embodiments of the present disclosure effectively converts a multi-codebook system, where N codebook indices maps need to be transmitted to a receiver, to one where only a leading codebook indices map needs to be transmitted to a receiver. Because the index values in the leading codebook indices map corresponds to multiple codebooks, the leading codebook VQ-based encoder framework 300 reduces the number of bits required to be transmitted to a receiver, thus increasing efficiency without affecting the quality of the reconstructed output image x.

[0067] In some embodiments, an optional base codebook is provided to assist the encoding of the leading codebook indices map hq idx by using the base codebook indices map Iq Jdx as areference. For example, as further described below, in an embodiment, the base codebook indices map Iq idx is used to generate a prediction p of the leading codebook indices map hq idx (e.g., let the base codebook indices map Iq idx pass a neural network (NN) and then train the NN so that the output p is as close to the leading codebook indices map hq idx as possible). In an embodiment, the residual / difference between the leading codebook indices map hq idx and the prediction p of the leading codebook indices map hq idx (i.e., residual=hq idx-p) is encoded in the bitstream. Compression is achieved when residual is close to 0. At decoding stage, the decoder decodes the residual and adds the base codebook indices map Iq idx to reconstruct the leading codebook indices map hq idx. The reconstructed leading codebook indices map hq idx is referred to below as hq_idx. In an embodiment, a base codebook is trained such that the base codebook indices map Iq idx is as close as possible to leading codebook indices map hq idx.

[0068] As an example, FIG. 4 is a diagram illustrating a base encoder framework 400 in accordance with an embodiment of the present disclosure. The base encoder framework 400 includes a base encoder 402, an AE / AD 406, a feature extractor 410, a before_quant_base 412, a base codebook 414, and a code former 416. In some embodiments, the before quant base 412, the base codebook 414, or the code former 416 are optional(as indicated by the dash lines in FIG. 4) and can be omitted from the base encoder framework 400.

[0069] In an embodiment, the base encoder 402 is an LQ encoder. The base encoder 402 is configured to convert an input image xb to a latent feature yb and a hyper tensor zb. In an embodiment, the input image xb is identical to input image x in FIG. 3. In other embodiments, the input image xb is a low-quality version of input image x in FIG. 3. In an embodiment, the latent feature yb has a shape of w x h x d, where the w and h depend on the input width and height as well as the network structure of the base encoder 402, and d is the number of feature channels. A hyper tensor is multi-dimensional array of 4 or more dimensions. In some embodiments, the hyper tensor zb is used to assist in the prediction and entropy coding process in the AE / AD 406.

[0070] In an embodiment, the AE / AD 406 first quantizes and entropy encodes the hyper tensor zb into the bitstream. The AE / AD 406 then decodes the bitstream to generate the reconstructedhyper tensor zb . In an embodiment, the AE / AD 406 is configured to generate a predicted version of tensor yb using the reconstructed hyper tensor zb. Various prediction methods can be used. For example, in one embodiment, an auto regression (AR) prediction method is used to predict elements in tensor yb one after another. In another embodiment, a checkerboard prediction method is used to predict half elements in tensor yb using the other half elements as anchor. In yet another embodiment, a mask-based prediction method is used to predict elements in tensor yb based on fixed or dynamic masking schedule. The AE / AD 406 quantizes and entropy encodes the difference between tensor yb and predicted yb (i.e., the quantized residual) into the bitstream. The AE / AD 406 then performs entropy decoding and tensor prediction to generate the reconstructed tensor yb. In some embodiments, a separate prediction may be used to generate the predicted version of tensor yb.

[0071] In some embodiments, when the input image xb is a reconstructed low-quality version of input image x that is encoded by either a traditional compression codec such as JPEG / HEVC / VVC or by a neural network-based compression codec, the AE / AD 406 can be skipped because compression has already been performed on the input image xb by the other codec and does not need to be compressed / decompressed again by the AE / AD 406.

[0072] In an embodiment, the feature extractor 410 is configured to enhance the reconstructed yb and generate tensor yb.

[0073] The before quant base 412 converts latent feature yb to a new latent feature space. The base codebook 414 is then configured to compute a base codebook indices map based on the tensor yb using a base codebook. As stated above, in an embodiment, the base codebook is a codebook that was trained to produce a base codebook indices map Iq idx that resembles (i.e., is as close as possible) to a leading codebook indices map hq idx.

[0074] The code former 416 receives the resulting base codebook indices map as input and generates a refined base indices map Iq idx that is even more close to the leading codebook indices map hq idx than was previously trained.

[0075] Various neural networks can be used in the leading codebook VQ-based encoder framework 300 and the base encoder framework 400. In one embodiment, a Visual Transformer (ViT) or Transformer block, such as Residual Swin Transformer Block (RSTB), is used, which divides the input image or latent feature into patches and processes them as a sequence. In another embodiment, a Convolutional Neural Network (CNN) structure is used and the entire image or latent feature is encoded in a parallel manner.

[0076] FIG. 5 is a diagram illustrating a NNIC system framework 500 in accordance with an embodiment of the present disclosure. In an embodiment, the NNIC system framework 100 is a two-branch NNIC framework incorporating the leading codebook VQ-based encoder framework 300 and the base encoder framework 400 along with an AE / AD 502, and a ControlNet 506. In a first / lead processing branch of FIG. 5, an input image x is passed through the VQ encoder 102, which converts the input image x to an embedding feature y. The embedding feature y is passed to the leading codebook 320. As described in FIG. 3, the leading codebook 320 processes the latent feature y and generates leading codebook indices map hq idx. In a second / base processing branch of FIG. 5, an input image xb is processed through the base encoder framework 400 to generate a refined base indices map Iq idx as described in FIG. 4.

[0077] As shown in FIG. 5, in the NNIC system framework 500, the AE / AD 502 is configured to receive the refined base indices map Iq idx from the base processing branch and the leading codebook indices map hq idx from the leading codebook 320. In an embodiment, the AE 502 is configured to perform a prediction operation to generate a predicted version of the leading codebook indices map hq idx using the refined base indices map Iq idx as an optional reference. As described above, various prediction methods can be used. In an embodiment, once the AE 502 generates the predicted hq idx, the AE 502 is configured to quantize and entropy encode the quantized residual difference between the leading codebook indices map hq idx and the predicted hq idx. The AE 502 then performs entropy decoding and tensor prediction to generate a reconstructed leading codebook indices map (hq_idx), which is returned to the leading codebook 320 in the lead processing branch. The leading codebook 320 then generates a correspondingcodeword indices map vqO based on the reconstructed leading codebook indices map (hq_idx). The Afier Quant Leading 326 converts the codeword indices map vqO to a discrete latent space to obtain a latent feature vqO.

[0078] As described in FIG. 3, in some embodiments, latent feature vqO is then passed to one or more REP adaptors 110. As described in FIG. 3, the REP adaptors 110 convert the latent feature vqO to N discrete latent spaces, generate N codeword indices maps, and converts the N codeword indices maps to N codebook-based latent features vqs.

[0079] As described in FIG. 4, in some embodiments, the feature extractor 410 in the base encoder framework 400 is configured to enhance the reconstructed yb and generate tensor yb. As shown in FIG. 5, in some embodiments, the tensor yb is provided as input to both the feature compressor 112 and the ControlNet 506. The feature compressor 112 is configured to generate an N-channel weight map W using the tensor yb. As described above, the feature merger 114 receives the weight map W and the N codebook-based latent features vqs. The weight map W guides / indicates how the N codebook-based latent features vqs can be weighted combined to generate a combined latent feature y.

[0080] In an embodiment, using the tensor yb as input, the ControlNet 506 generates control signals yc to guide the conditioned generative reconstruction process based on the combined latent feature y. The decoder 116 use both the combined latent feature y and the control signals yc to generate the reconstructed output image x. There are multiple ways the decoder 116 can use the control signals yc to guide the reconstruction process. In some embodiments, the decoder 116 has a network structure of multiple CNN layers like the decoding network of a VAE. Then the control signals yc tunes the combined latent feature y through an affine transformation and generates a new tuned feature y*: * = + aQSy + y), where a is a pre-set guidance strength and the affine parameters / , y are given by:P.Y = d con{y, yc

[0081] where conQ is the concatenation operation. This affine transformation is usually implemented by an NN with the set of parameters 9 that are determined through training.

[0082] As described above, by using a leading codebook, the NNIC system framework 500 can convert N codeword indices map into one codeword indices map and thus reduce the number of bits needed to be transmitted to a receiver without affecting the quality of the reconstructed output image x. In some embodiments, to further reduce the total bitrate, the codewords in a codebook can be reordered in descending order based on a frequency of the codeword usage in training data. For instance, for images depicting nature, codewords corresponding to natural scenes may be used more frequently than codewords corresponding to human faces.

[0083] Additionally, existing methods do not consider the difference of importance factor among channels, resulting in suboptimal performance. For instance, the present disclosure recognizes the following observations: (1) the correlation among latent channels is relatively constant for different images, (2) a few distinct channels are sensitive to degradation operation such as quantization, and (3) the correlation between these distinct channels and all other channels is low, making these channels special and important compared to other channels. Because the output tensor of the layer is generated by the multiplication of input weight tensor and input activation tensor of the layer, the correlation among latent channels is caused by the channels’ input weight tensor and is the inherent property of a pretrained model. However, in current codebook codeword and indices generation method, a codeword cql(i = 1, ..., w x h) is nearest to the corresponding latent feature yi and zkis the index of codeword cq: zkl= argminCkE CDist(ck,yj), ckl= C[zkl],

[0084] where Dist(y) is a distance metric, such as LI or L2 norm. Because the Dist(y) function treats distances among channels equally, existing methods do not consider the difference of importance factor among channels, resulting in suboptimal performance. Thus, in some embodiments, a channel-weighted codeword generation method is provided to take the differenceof importance factor among channels into consideration. As an example, in one embodiment, a vector is defined as a weighting factor wf and an element-wise multiplication is applied to the output of Dist( ) zkl= argminCke c(wf x Dist ck,yi))

[0085] In an embodiment, vector wf can be defined as a predefined vector where constant value is supplied as channel importance factor for each channel, or vector ir ' can be defined as a learnable vector so that vector wf can be optimized during a training process.

[0086] Further, existing methods do not take the rate factor into consideration, resulting in suboptimal rate distortion (RD) performance. Thus, in some embodiments, the following RD based codeword generation method is provided to account for the rate factor:

[0087] Where Rate(f) is an entropy metric to estimate the entropy of the codeword index k, ist(') is a distance metric, such as LI or L2 norm, and lambda is a hyperparameter to control the weight between rate and distortion.

[0088] In some embodiments, because a leading codebook and a base codebook as described herein are trained separately, and the input to the leading codebook is a high quality encoder output, and the input for the base codebook is a low quality base encoder output, through this training process the leading codebook and the base codebook are mismatching, meaning the same two corresponding vector in the leading codebook and the base codebook may be assigned different index values and the difference value between the two index values may be large, which is bad for compression because the difference value is compressed, so having the difference value small is desirable.

[0089] In an embodiment, the following location matching method is provided to minimize the difference between the high quality index and the low quality index. First, a base codebook is trained independently. Then, the latent feature of training data is processed using VQ encoder102 and a leading codebook. The location matching method loops through HQ indices, for codeword index n, and records the vector location to a collection R(n) for any HQ latent feature vector who is assigned with same codeword index n. Then, the same latent feature of training data is processed using the base encoder 402 and a base codebook. The location matching method records all LQ indices for all LQ latent feature vectors. The location matching method then loops through HQ indices, for HQ codeword index n, records all LQ indices for the LQ vectors in location collection R(n), calculates the histogram of the LQ indices and selects the most frequent LQ index as the matching index for HQ index n. The location matching method then reorders the base codebook based on the generated LQ / HQ indices matching lookup table.

[0090] Using independently trained leading codebook and base codebook, the location matching method minimizes the difference between the high quality index and the low quality index by assigning identical index for matching location in HQ / LQ latent feature maps as much as possible.

[0091] In some embodiments, the location matching method can be viewed as a classification problem where the HQ codeword indices can be treated as the classification label for LQ latent feature.

[0092] In one embodiment, a contrastive classification training method is provided to train the base codebook where for HQ codeword index n. All LQ feature vectors are treated as positive samples if the HQ codeword index of their corresponding HQ feature vectors equals n. All LQ feature vectors are treated as negative samples if the HQ codeword index of their corresponding HQ feature vectors does not equal n.

[0093] Since the codeword index is decided by argmin() which is not differentiable, it can be replaced by differentiable measurement for training.

[0094] For instance, in one embodiment, softmax() can be used as soft classification label:

[0095] zkl= softmaxCkE c(-Dist(ck, yd)

[0096] In another embodiment, ist(x), Dist -x) or their normalized version can be used as the ground truth for training.

[0097] Assuming z is the LQ latent feature, C is the base codebook, gt dist is the output of ist(') from leading codebook, codebook loss can be defined by following example pseudo code:

[0098] Distance calculation:

[0099] q dist = Dist( z.detachQ, C )

[0100] e dist = Dist( z, C.detach() )

[0101] Distance normalization using softmax():

[0102] q norm = torch.nn.functional.softmax( -q dist / temp, dim=l )

[0103] e_norm = torchmn. functional. softmax( -e_dist / temp, dim=l )

[0104] gt norm = torch. nn. functional. softmax( -gt_dist.detach() / temp, dim=l )

[0105] Codebook loss using 12 loss:

[0106] q_dist_loss = self.mse( q norm, gt norm )

[0107] e_dist_loss = self.mse( e_norm, gt_norm )

[0108] Total codebook loss:

[0109] codebook_loss = q_dist_loss + e_dist_loss * beta

[0110] In some embodiments, a masking and reconstruction method is provided to further minimize information transmission of leading indices map hq idx. As an example, FIG. 6 is a diagram illustrating a masked generative transformer (MAGE) Prediction method 600 in accordance with an embodiment of the present disclosure.

[0111] Utilizing a masking schedule 602 equipped with predefined, structured, and compression-friendly schedules, and with the help of an optional reference Iq idx, only the unmasked proportion of hq idx (hq midx) is encoded and included into the bit stream. The chosen schedules indicate the proportion of the remaining information, contributing to a more efficient compression strategy.

[0112] In one embodiment, the masking schedule is a fixed schedule so that it does not need to be included into the bitstream, in another embodiment, the masking schedule is content adaptive so that the masking / unmasking map need to be included into the bitstream, in yet another embodiment, the masking schedule is content adaptive but provides only limited number of fixschedule based on the distribution of the content so that only the schedule index need to be included into the bitstream.

[0113] The AE / AD 604 first performs prediction operation to generate a predicted version of hq midx using Iq idx as an optional reference, then the quantized residual (difference between hq nidx and predicted hqjnidx) is encoded and included into the bitstream. The AE / AD 604 decodes the bitstream to reconstruct the residual and generate the reconstructed hqjnidx using the predicted hqjnidx. In an embodiment, an encoder-to-decoder transformer structure is used to generate reconstructed hq dx from hqjnidx where a pre-trained MAGE is provided as base structure.

[0114] The Indices Tokenizer 606 converts hq nidx to its embedding version hq mt. Then, the MAGE Transformer Encoder 608 convert hqjnt to an encoded version hq emt and transits the encoded version hq emt to a Modified MAGE Transformer Decoder 610. The Modified MAGE Transformer Decoder 610 is configured to predict the masked tokens using conditioning guided control information yb. A Class Linear Layer 612 generates predicted logits hq logits. Finally, an Argmax() operation 614 uses the predicted logits hq logits to generates the reconstructed hq dx.

[0115] In an embodiment, to incorporate conditioning control information from the base encoder output yb, inspired by common techniques in audio-to-text translation tasks where audio information is used to guide text token generation through cross-attention, a cross-attention block is inserted in each decoder block of the transformer decoder. This allows tokens to draw relevant information from yb, facilitating guided token generation.

[0116] In one embodiment for training, as the inserted cross-attention block in the Modified MAGE Transformer Decoder 610 only influences the decoder block, while maintaining the rest of the pre-trained parameters unchanged and frozen. Only the newly inserted cross-attention block, a Multi-Layer Perceptron (MLP) block for each decoder block, and the outermost Class Linear Layer are unfrozen. Following the training design of MAGE, the decoder is trained to accurately predict the masked tokens.

[0117] In one embodiment, none, some, or all residual signal (the difference between the reconstructed hq_idx and original hq idx) is encoded and included into the bitstream.

[0118] In some embodiments, because image regions vary in detail and complexity, a statistical procedure is provided for discerning complexity in low-bit compression where image entropy, contrast, color diversity (histogram entropy), and spatial frequency (Fourier transform) are utilized to build the complexity evaluation function. The normalized complexity score synthesizes these aspects, providing insights into image complexity. The complexity equation has been formulated as:

[0119] F (X)=w 1 * entropy(x)+w2 * contrast(X)+w3 * color_entropy(X)+w4 * s_freq(X),

[0120] where entropyQ, contrastQ, color entropyQ and s_freq() indicate image entropy, image contrast, color diversity (histogram entropy), and spatial frequency (Fourier transform).

[0121] Leveraging a variety of images, the image complexity distribution with the formula has been acquired and different thresholds for dynamic mask schedules have been set accordingly. For intricate image patches, a low mask ratio is applied, while straightforward patches receive an exceptionally high mask ratio.

[0122] Image is partitioned to multiple blocks first, then different mask schedules are applied to blocks with different complexities. This ensures that, while maintaining image quality, more transmission bits are allocated to high-complexity regions, and lower bit rates are used for low- complexity areas. The complexity-aware masking procedure offers a straightforward yet effective approach to further reduce transmission bit rates dynamically, enabling our proposed framework to better serve extreme low-bit image compression scenarios.

[0123] FIG. 7 is a flowchart diagram illustrating a VQ neural network-based image compression method 700 implemented by an encoding device in accordance with an embodiment of the present disclosure. The encoding device, at step 702 encodes, using a first encoder of the encoding device, a first image to obtain a first latent feature corresponding to the first image. At step 704, the encoding device generates, based on the first latent feature and using a leading codebook, a leading codebook indices map and a corresponding first codeword map. The leadingcodebook comprises codebook indices that are assigned to identical vectors in multiple codebooks. At step 706, the encoding device encodes the leading codebook indices map to generate an encoded leading codebook indices map. The encoding device, at step 708, transmits the encoded leading codebook indices map to a decoder.

[0124] FIG. 8 is a flowchart diagram illustrating a VQ neural network-based image compression method 800 implemented by a decoding device in accordance with an embodiment of the present disclosure. The decoding device, at step 802, receives an encoded leading codebook indices map, wherein the encoded leading codebook indices map comprises indices that are mapped to identical vectors in multiple codebooks. At step 804, the decoding device generates, based on the encoded leading codebook indices map, N latent feature map by retrieving codeword from N codebooks using respective codebook indices in the encoded leading codebook indices map. The decoding device, at step 806, combines the N latent feature map with an N- channel weight map to generate a reconstructed latent feature. At step 808, the decoding device decodes the reconstructed latent feature to obtain a reconstructed image.

[0125] FIG. 9 is a diagram illustrating an apparatus 900 according to an embodiment of the present disclosure. The apparatus 900 can be used to implement embodiments of the present disclosure. For example, the apparatus 900 may be configured to perform the functions of a sending / transmitting device or a receiving device according to any of the embodiments of the present disclosure. The apparatus 900 includes receiver units (RX) 920 or receiving means for receiving data via ingress ports 910. The apparatus 900 also includes transmitter units (TX) 940 or transmitting means for transmitting data via data egress ports 950.

[0126] The apparatus 900 includes a memory 960 or data storing means for storing the instructions and various data. The memory 960 can be any type of, or combination of, memory components capable of storing data and / or instructions. For example, the memory 960 can include volatile and / or non-volatile memory such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 960 can also include one or more disks, tape drives, and solid-state drives. In some embodiments, the memory 960 can be used as an over-flow data storage device to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. In some embodiments, the memory 960 can be memory that is integrated with the processor 930.

[0127] The apparatus 900 has one or more processors 930 or other processing means (e.g., central processing unit (CPU)) to process instructions. The one or more processors 930 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 930 are communicatively coupled via a system bus with the ingress ports 910, RX 920, TX 940, egress ports 950, and memory 960. The one or more processors 930 can be configured to execute instructions stored in the memory 960. As an example, in one embodiment, the memory 960 stores a neural network image compression unit 970. The neural network image compression unit 970 includes data, executable instructions, and / or one more sub-routines for implementing the disclosed embodiments. Thus, the one or more processors 930 provide a means for performing any computational, comparison, determination, initiation, configuration, or any other action corresponding to the claims when the appropriate instruction is executed by the processor 930. Thus, the inclusion of the neural network image compression unit 970 substantially improves the functionality of the apparatus 900.

[0128] While several embodiments have been provided in the present disclosure, a person of ordinary skill in the art would understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the disclosure is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0129] In summary, the disclosed embodiments includes, but is not limited to, the following novel features:

[0130] A two-branch framework incorporating a RED-VQ encoder using a masked generative encoder (MAGE) for masked codebook indices prediction and a base encoder. In some embodiments, high mask ratio (50%, 75%, 89%, 94%) indices prediction by MAGE provides better indices prediction and better bitrate reduction. A complexity-aware masking technique as described above further reduces transmission bit rates while maintaining image reconstruction quality.

[0131] The introduction of a leading codebook effectively converts a multi-codebook system to a leading-codebook pipelined with one or more REP-Adaptors. Thus, providing further bitrate reduction.

[0132] Additionally, some embodiments include codebook reorder, channel -weight indices generation and RD based indices generation, which enables better codeword matching. Further, in some embodiments, location matching, contrastive classification based training provides better reference indices map for indices prediction and compression.

[0133] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A vector quantization (VQ) neural network-based image compression method implemented by an encoding device, the method comprising: encoding, using a first encoder of the an encoding device, a first image to obtain a first latent feature corresponding to the first image; generating, based on the first latent feature and using a leading codebook, a leading codebook indices map and a first codeword map corresponding to the leading codebook indices map, wherein the leading codebook comprises codebook indices that are assigned to identical vectors in multiple codebooks; encoding the leading codebook indices map to generate an encoded leading codebook indices map; and transmitting the encoded leading codebook indices map toward a decoder.

2. The VQ neural network-based image compression method of claim 1, encoding, using a second encoder of the an encoding device, a second image to obtain a second latent feature and a hyper tensor corresponding to the second image; and generating, based on the hyper tensor and using a base codebook, a base codebook indices map, wherein encoding the leading codebook indices map to generate the encoded leading codebook indices map comprises using the base codebook indices map as a reference.

3. The VQ neural network-based image compression method of claim 2, wherein the base codebook is trained to create the base codebook indices map to resemble to the leading codebook indices map.

4. The VQ neural network-based image compression method according to any of claims 2-3, wherein the base codebook is trained to create the base codebook indices map to be as close as possible to the leading codebook indices map.

5. The VQ neural network-based image compression method according to any of claims 2-4, wherein generating, based on the hyper tensor and using the base codebook, the base codebook indices map comprises: quantizing and entropy encoding the hyper tensor to obtained an encoded hyper tensor; decoding the encoded hyper tensor to obtained a reconstructed hyper tensor; performing tensor prediction using the reconstructed hyper tensor to obtain a predicted tensor of a first tensor; quantizing and entropy encoding a difference between first tensor and the predicted tensor; performing entropy decoding and tensor prediction to generate a reconstructed first tensor; and computing, using the base codebook, the base codebook indices map based on the reconstructed first tensor.

6. The VQ neural network-based image compression method according to any of claims 2-5, wherein the second image is a lower-quality image of the first image.

7. The VQ neural network-based image compression method according to any of claims 2-5, wherein the second image is the same as the first image.

8. The VQ neural network-based image compression method according to any of claims 2-7, wherein codewords in the leading codebook or the base codebook are reordered in descending frequency order so that less bits are assigned to more frequently used indices to reduce a total bitrate.

9. The VQ neural network-based image compression method according to any of claims 2-8, wherein codewords in the leading codebook or the base codebook are generated by a channel- weighted codeword generation method that accounts for a difference of importance factor among channels.

10. The VQ neural network-based image compression method according to any of claims 2-9, wherein codewords in the leading codebook or the base codebook are generated by a codeword generation method that accounts for rate factor.

11. The VQ neural network-based image compression method according to any of claims 2-10, further comprising performing a location matching method to minimize a difference between the leading codebook indices map and the base codebook indices map.

12. The VQ neural network-based image compression method according to any of claims 2-11, further comprising training the base codebook using a contrastive classification training method.

13. The VQ neural network-based image compression method according to any of claims 1-12, further comprising performing masking and reconstruction to further minimize information transmission of the encoded leading codebook indices map.

14. The VQ neural network-based image compression method according to claim 13, wherein the masking applies a mask ratio based on an image complexity.

15. A vector quantization (VQ) neural network-based image compression method, implemented by a decoding device, the method comprising: receiving an encoded leading codebook indices map, wherein the encoded leading codebook indices map comprises indices that are mapped to identical vectors in multiple codebooks;generating, based on the encoded leading codebook indices map, N latent feature map by retrieving codeword from N codebooks using respective codebook indices in the encoded leading codebook indices map; combining the N latent feature map with an N-channel weight map to generate a reconstructed latent feature; and decoding the reconstructed latent feature to obtain a reconstructed image.

16. The VQ neural network-based image compression method of claim 15, wherein decoding the reconstructed latent feature to obtain the reconstructed image comprises: generating control signals to guide a conditioned generative reconstruction process in decoding the reconstructed latent feature to obtain the reconstructed image.

17. An apparatus comprising: a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the apparatus to: encode, using a first encoder of the apparatus, a first image to obtain a first latent feature corresponding to the first image; generate, based on the first latent feature and using a leading codebook, a leading codebook indices map and a first codeword map corresponding to the leading codebook indices map, wherein the leading codebook comprises codebook indices that are assigned to identical vectors in multiple codebooks; encode the leading codebook indices map to generate an encoded leading codebook indices map; and transmit the encoded leading codebook indices map to a decoder.

18. The apparatus of claim 17, wherein the one or more processors execute the instructions to further cause the apparatus to: encode, using a second encoder of the apparatus, a second image to obtain a second latent feature and a hyper tensor corresponding to the second image; and generate, based on the hyper tensor and using a base codebook, a base codebook indices map, wherein encoding the leading codebook indices map to generate the encoded leading codebook indices map comprises using the base codebook indices map as a reference.

19. The apparatus of claim 18, wherein the base codebook is trained to create the base codebook indices map to resemble the leading codebook indices map.

20. The apparatus according to any of claims 18-19, wherein the base codebook is trained to create the base codebook indices map to be as close as possible to the leading codebook indices map.

21. The apparatus according to any of claims 18-20, wherein the one or more processors execute the instructions to further cause the apparatus to: quantize and entropy encoding the hyper tensor to obtained an encoded hyper tensor; decode the encoded hyper tensor to obtained a reconstructed hyper tensor; perform tensor prediction using the reconstructed hyper tensor to obtain a predicted tensor of a first tensor; quantize and entropy encoding a difference between first tensor and the predicted tensor; perform entropy decoding and tensor prediction to generate a reconstructed first tensor; and compute, using the base codebook, the base codebook indices map based on the reconstructed first tensor.

22. The apparatus according to any of claims 18-21, wherein the second image is a lower- quality image of the first image.

23. The apparatus according to any of claims 18-22, wherein codewords in the leading codebook or the base codebook are reordered in descending frequency order, so that less bits are assigned to more frequently used indices to reduce a total bitrate.

24. The apparatus according to any of claims 18-23, wherein codewords in the leading codebook or the base codebook are generated by a channel-weighted codeword generation method that accounts for a difference of importance factor among channels.

25. The apparatus according to any of claims 18-24, wherein codewords in the leading codebook or the base codebook are generated by a codeword generation method that accounts for rate factor.

26. The apparatus according to any of claims 18-25, wherein the one or more processors execute the instructions to further cause the apparatus to perform a location matching method to minimize a difference between the leading codebook indices map and the base codebook indices map.

27. The apparatus according to any of claims 18-26, wherein the one or more processors execute the instructions to further cause the apparatus to configured to train the base codebook using a contrastive classification training method.

28. The apparatus according to any of claims 18-21, wherein the second image is the same as the first image.

29. The apparatus according to any of claims 17-28, wherein the one or more processors execute the instructions to further cause the apparatus to configured to perform masking and reconstruction to further minimize information transmission of the encoded leading codebook indices map.

30. The apparatus of claim 29, wherein the masking applies a mask ratio based on an image complexity.

31. An apparatus comprising: a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the apparatus to: receive an encoded leading codebook indices map, wherein the encoded leading codebook indices map comprises indices that are mapped to identical vectors in multiple codebooks; generate, based on the encoded leading codebook indices map, N latent feature map by retrieving codeword from N codebooks using respective codebook indices in the encoded leading codebook indices map; combine the N latent feature map with an N-channel weight map to generate a reconstructed latent feature; and decode the reconstructed latent feature to obtain a reconstructed image.

32. The apparatus of claim 31, wherein the one or more processors execute the instructions to further cause the apparatus to generate control signals to guide a conditioned generative reconstruction process in decoding the reconstructed latent feature to obtain the reconstructed image.

33. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processors of an apparatus, cause the apparatus to perform a method according to any of claims 1-17.

34. An apparatus comprising: a first encoder or first encoding means for encoding a first image to obtain a first latent feature corresponding to the first image; a first generating means or means for generating , based on the first latent feature and using a leading codebook, a leading codebook indices map and a first codeword map corresponding to the leading codebook indices map, wherein the leading codebook comprises codebook indices that are assigned to identical vectors in multiple codebooks; a second encoding means for encoding the leading codebook indices map to generate an encoded leading codebook indices map; and a transmitting means or means for transmitting the encoded leading codebook indices map to a decoder.

35. The apparatus of claim 34, further comprising: a second encoder or encoding means for encoding a second image to obtain a second latent feature and a hyper tensor corresponding to the second image; and a second generating means or means for generating, based on the hyper tensor and using a base codebook, a base codebook indices map, wherein encoding the leading codebook indices map to generate the encoded leading codebook indices map comprises using the base codebook indices map as a reference.

36. The apparatus of claim 35, wherein the base codebook is trained to create the base codebook indices map to resemble the leading codebook indices map.

37. The apparatus according to any of claims 35-36, wherein the base codebook is trained to create the base codebook indices map to be as close as possible to the leading codebook indices map.

38. The apparatus according to any of claims 35-37, further comprising: a first quantize and entropy encoding means for quantizing and entropy encoding the hyper tensor to obtained an encoded hyper tensor; a decoding means for decoding the encoded hyper tensor to obtained a reconstructed hyper tensor; a tensor prediction means for performing tensor prediction using the reconstructed hyper tensor to obtain a predicted tensor of a first tensor; a second quantize and entropy encoding means for quantizing and entropy encoding a difference between first tensor and the predicted tensor; an entropy decoding and tensor prediction means for performing entropy decoding and tensor prediction to generate a reconstructed first tensor; and a computing means for computing , using the base codebook, the base codebook indices map based on the reconstructed first tensor.

39. The apparatus according to any of claims 35-38, wherein the second image is a lower- quality image of the first image.

40. The apparatus according to any of claims 35-39, wherein codewords in the leading codebook or the base codebook are reordered in descending frequency order, so that less bits are assigned to more frequently used indices to reduce a total bitrate.

41. The apparatus according to any of claims 35-40, wherein codewords in the leading codebook or the base codebook are generated by a channel-weighted codeword generation method that accounts for a difference of importance factor among channels.

42. The apparatus according to any of claims 35-41, wherein codewords in the leading codebook or the base codebook are generated by a codeword generation method that accounts for rate factor.

43. The apparatus according to any of claims 35-42, further comprising a location matching means for minimizing a difference between the leading codebook indices map and the base codebook indices map.

44. The apparatus according to any of claims 35-43, further comprising a contrastive classification training means to train the base codebook.

45. The apparatus according to any of claims 35-38, wherein the second image is the same as the first image.

46. The apparatus according to any of claims 34-45, further comprising a masking and reconstruction means to further minimize information transmission of the encoded leading codebook indices map.

47. The apparatus of claim 46, wherein the masking and reconstruction means applies a mask ratio based on an image complexity.

48. An apparatus comprising: a receiving means for receiving an encoded leading codebook indices map, wherein the encoded leading codebook indices map comprises indices that are mapped to identical vectors in multiple codebooks; a generating means for generating, based on the encoded leading codebook indices map, N latent feature map by retrieving codeword from N codebooks using respective codebook indices in the encoded leading codebook indices map; a combining means for combining the N latent feature map with an N-channel weight map to generate a reconstructed latent feature; and a decoding means for decoding the reconstructed latent feature to obtain a reconstructed image.

49. The apparatus of claim 48, further comprising a control signals generating means for generating control signals to guide a conditioned generative reconstruction process in decoding the reconstructed latent feature to obtain the reconstructed image.

Images